Data Noise | Introduction to Data Mining | Part 7

Key Takeaways

so those of you who have a scientific or signal-processing background are probably familiar with the term noise noise in the data science context is when we have an invalid signal of some sort that overlaps valid data this obscures our our actual attribute values and fundamentally what it means is that some of our data objects have invalid values in some of the attributes they don't have real the inaccurate values there so examples of this in real life we have the distortion of a person's voice over the phone snow on old television screens peeled CRT television screens noise can appear because of human inconsistency in labeling you see this a lot in sports for instance that require human judging there's a lot of inconsistency in how people get labeled there and just in general if you're trying to say ranked websites for instance human constancy and lays in labeling can be a real problem so as sort of a practical example of what noise can do when there's a lot of it this is a pretty straightforward signal we've got two sine waves here with different with different frequencies but the same amplitude there's a blue one and a green one and if we in so we could generate the sine wave it looks very clean very pretty we can even sort of distinguish the two different sine waves if we add those two waves together and then throw noise at it just sort of basic white noise like you might see in any kind of randomization thing and you end up with something that looks like this so the noise has completely obscured our actual signal so noise is again fundamentally invalid data points that are that are obscuring our signals we have to be there's always some noise in any system it's just the nature of the universe sadly but understanding where your noise is at its worst and how you can deal with it is very important but even recognizing that it's there is the first step recognizing which of your attributes are noisy versus which are not are more noisy which is that what versus which of them are less noisy sort of the the complementary problem complimentary problem to noise is the problem of outliers so outliers often look like noise at first their data objects that have characteristics that are considerably different from most of the other objects in the data set so if we look at sort of the the visual here we've got some sort of two-dimensional graphing of our data and most of our to each dot each pixel point represents a data object that's been plotted on a graph so we've got you know four clusters very Nut kind of nicely defined clusters and then we've got these three other points just kind of hanging out in the middle of nowhere far away from all of the other data so the big distinction between outliers are that between outliers and noise is that outliers are actually valid values the data was collected properly it's clean but it's outside of the normal range the data object for some reason doesn't look like a normal object all right so so that's outliers and noise those are sort of the first category of data quality problems that get encountered a lot