Intro to Statistics

Key Takeaways

I bring you the light of Statistics hello world it's Suraj and understanding statistics is a really important part of data science there are entire textbooks and graduate level programs dedicated to mastering this branch of mathematics so to give you a brief but relevant overview we're gonna learn about just three major concepts in statistics that all data scientists should understand for context we'll apply them to a data set of loans that were issued to people between 2007 and 2015 to figure out the type of people that received loans whether or not they're given credit score was appropriate and whether or not we can create a model to better predict their credit score using machine learning it's hard to understate how crucial statistics is in data science in fact in 96 the term data science was used for the first time in the title of a statistical conference call ifcs the title was data signs classification and related methods data signs started with statistics and has evolved to include concepts like machine learning and artificial intelligence statistics is a collection of procedures and principles for gaining information in order to make decisions when faced with uncertainty I should have used it in my last relationship it's an extremely valuable skill to understand so much so that statisticians can work in virtually any field from business to social science to medicine in the context of data science think of a data scientist as a person who's better at statistics than any programmer and better at programming than any statistician we can use statistics for so many problems like identifying the risk factors for a type of cancer or customizing a spam detection system or establishing the relationship between salary and demographic variables in population survey data so let's focus on a very specific problem in 2008 there was a subprime mortgage crisis in the United States which was terrible for many companies like when Lehman Brothers went down down down down down but it did create opportunities for new players in the retail credit field following the credit scarcity that took place briefly during those times peer-to-peer lending companies thrived according to PwC u.s. peer-to-peer lending platform volumes have grown an average of 84 percent per quarter since 2007 Lending Club is one example of a popular p2p lending marketplace and we can use readily available data from them to help us figure out if credit scores for a given set of people are accurate and if we can create more optimized ones ourselves the Lending Club data set contains 887 K loan applications over a period of eight years and we can download it directly from their websites as you can see there are a lot of columns meaning features in this data set for every person we have the amount they were loaned various details about their background like whether or not they own a home where they live and of course their credit score using statistics we can gain deeper insight into how this data is structured and then based on this structure we can optimally apply other data science techniques to get even more information so let's start with the first statistics concept we can use here statistical features this is probably the most used statistics concept in data science it's usually the first stats technique we would apply when exploring a data set and includes concepts like bias variance mean median percentiles and many others all of them are easy to understand and implement in code but it'll take a while to define every single one of them and you can't download them into your brain yet so I've linked to a detailed cheat sheet in the video description to visualize some of these statistical features let's create what's called a box plot to examine the relationship between income and loan amount box plots are a standard way of displaying the distribution of data based on a few statistical features in order to do that we'll look at a subset from our data where income is less than 120 K per year the reason being that applications with income above this threshold are not statistically representative of our population out of 800 ATK loans only 10% have annual incomes higher than that if we didn't cap our annual income we would have a lot of outliers in our box plot wouldn't look as insightful using our box plot we can easily visualize some statistical features the line in the middle is the median value of the data the median is used over the mean since it's more robust to outliers the first quartile is a 25th percentile meaning 25% of the points in the data fall below that value the third quartile is a 75th percentile meaning 75% of the points in the data fall below that value the Max and min values represent the lower and upper ends of our data range box plots are awesome because they demonstrate how we can utilize basic statistical features instantly if a box plot is short it means that our data points are generally similar many values are in a small range if it's tall it implies that our data points are different since the values are more spread out if the median value is closer to the bottom we know that most of the data has lower values if the median value is closer to the top we know that most of the data has higher values if the median line is not in the middle of the box it means we have skewed data we could keep going here if the whiskers are super long it means our data has a high standard deviation and variance which means our values are spread out in highly varying as you can see we can get a lot of information from just a few simple statistical features that are all easy to calculate my friend said he made it up to tackle top spot and Ola made was a brand new box Lotte will notice that the quartile distribution of fully paid is very different from the quartile distribution of charged-off but it's similar to current in grade period and issued this means that Lending Club has been more selective with its newer loans also charged off and default statuses hold similarities in terms of the quartile distribution deferring from all the others this lets us know that the income variables are important for predicting loan grades if we add another dimension to the analysis by generating the box plots for income versus loan grade will find that a graded loans have a median income that's superior to other grades but we can't say the same about the other quartiles notice though that f.g and be graded loans hold a similar income quartile distribution lacking consistency hmm maybe income actually isn't that critical when determining lending clubs loan grades will have to keep investigating here to learn more the second important concept from statistics to know is the probability distribution we can define probability as the percent chance that some event will occur usually this is quantified in the range of 0 to 1 where 0 means we are sure that it won't occur and 1 means we're totally sure it will occur we can think of a probability distribution as a function that represents the probabilities of all possible values in an experiment there are many different types of distributions so much interesting theory here for example a uniform distribution has a single value which only occurs in a certain range while anything outside of that range is solely 0 think of it as an on or off distribution a normal distribution is specifically defined by its mean and standard deviation with this we know the average value of our dataset as well as how spread out it is the Poisson distribution is like the normal but with the Edit factor of skewness when skewness is high then the spread of the data will be different in different directions one direction could be very spread while the other could be very concentrated there are more distributions I just wanted to give you an overview of three important ones if we generate a distribution plot for annual incomes from single applications we'll find that it's heavily skewed heavily peaked and has a long right tail these points are regularly observed in distributions that are fit by a power law a power law is a functional relationship between two quantities or a relative change in one quantity results in a proportional relative change in the other quantity independent of the initial size of these quantities basically one quantity varies as a power of another we can informally say that we have a power law candidate distribution here most applications are coming from the lower end of the income spectrum an interesting observation the third concept to know about is Bayesian statistics to understand Bayesian stats we have to first understand frequency stats everyone everyone stay calm no gang wars please frequency statistics is the type of stats that involves applying math to analyze the probability of some event occurring where specifically the only data we compute on is prior data if we had a die and were asked what the chance of rolling a 6 was most people would say it's 1 and 6 but what if someone were to tell us that the specified die given to us was loaded to always land on 6 frequency stats only takes into account prior data that new evidence that was given to us about the die being loaded is not being taken into account Bayesian statistics however does take into account this evidence we can illustrate it by taking a look at Bayes theorem where e is the evidence and H is the hypothesis the probability of the hypothesis given the evidence is equal to the prior probability multiplied by the likelihood of the evidence e if the hypothesis is true divided by the priori probability the evidence itself is true Bayesian stats takes everything into account we use it whenever we feel that our prior data will not be a good representation of our future data and results Bayes theorem simplifies complex concepts it explains a lot using a few simple variables it supports the concept of conditional probability meaning if a occurred if played a role in the occurrence of B Bayes theorem can help us predict the probability of someone having a specific disease knowing their age it can let us know if an email is spam based on the number of words it's used to remove uncertainty it was even used to help predict the configurations of the Enigma machine to translate the German codes in World War two so how do we utilize Bayesian statistics here one way is to build a Bayesian classifier algorithm one that will predict credit scores given other features in the data set naive Bayes is a family of algorithms that takes advantage of Bayes theorem to predict a target variable this type of classifier assumes all features are unrelated to each other using a few features we select we can predict whether a loan applicant should be accepted or rejected the accuracy looks all right I'm sure we can try several other classifier models here to find a better predictor but this is a great first step as you can see sistex is supremely useful in data science and we only covered three key concepts from statistics there are so many more like up sampling and down sampling and dimensionality reduction there's more but there are just three things to remember from this video statistical features like bias variance and many others help us explore a data set to gain valuable insights probability distributions define the percent chance that some event will occur and we can use them to understand the spread of data and Bayesian statistics expresses probability as a degree of belief in an event which can change as new information is gathered rather a fixed value based on frequency the coding challenge for this week is to use statistics to perform exploratory data analysis on this Lending Club data set in the form of a Jupiter notebook the top two most detailed reports win I'll give them a shout out next week post your github links in the comment section details will be in their github readme in the video description what's one thing you like about statistics let me know in the comments section and please subscribe for more programming videos for now I've got to predict the target so thanks for watching