Data Science Interview Questions | Data Science Tutorial | Data Science Interviews | Edureka Rewind

Key Takeaways

hey I all uh to the session on preparing for data science interview questions I'm kic so I've been like working in this data science domain for now close to 78 years so in this session we are uh going to focus on certain fundamental questions and the thought processes you need to sit for an data science interview and then from there on we'll make the questions a bit more complex as we move on and will try to answer uh which kind of skill sets people normally look at when you appear for a data science interview in any company so there are like various roles depending on that uh the complexity and the variety in the questions might change uh but this session will try to cover things in a fundamental level so that It prepares in a very common ground irrespective of which role you are applying for you you might be able to at least get a perspective of what data science requires as such right so with that note let's see uh how we're going to structure the various questions that's going to come up in this session today majorly focusing on statistics data analytics machine learning and probability so these are the four headers under which the questions will come and I'll discuss the question and how to approach a particular question in a structured approach right in a structured Manner and once we understand the problem well we will see how does that thought process of coming out with the solution really works right so all of you you know that uh data science is in booming State many openings are coming up in uh across the globe in various companies irrespective of which company it is which particular business it deals with there is a potential of some applications from data science right they have lot of data different systems of their businesses generating the data with uh an enormous amount of volume velocity and variety so with that sort of data asset that every company has now got they want to now leverage it towards growing the business to the next level so you have some of the stats here like this is more than sufficient motivation that getting into data science would definitely land you in some really good professional carer okay so let's start with the questions here directly we'll in the beginning start to focus on some of the fundamental uh questions which is more for you to understand what is data science than like uh a particular interviewer asking you that question for instance uh many people wonder what data science is all about right though there are many online sources and blogs which describes data science in a very nutshell this is what it boils down to right a person who is very good in understanding the computer algorithms a person who understands the statistics and mathematical ideas and applying these two knowledges from the Computer Science and Mathematics into a particular application right a business application where somebody sees a value coming out from the data so how do you apply that so that's how kind of data science approaches so when you combine these two powerful Concepts coming in from computer science and mathem matics on a real world application the sort of outcome that comes from particular data science project goes in a direction where people see a return on investment right so the people you bring in the technology you bring in the ideas you are working on all of it should kind of give you some return on the investment you have put in so that's where Industries are started to looking at uh data science so various subjects which are very important for you to know statistics computer science Applied Mathematics and and then subjects like linear algebra calculus and a few more right so fundamentally from computer science algorithms and data structures might might be very useful from the mathematics or statistics and things like calculus linear algebra Matrix factorization and Concepts like that and from the application it is more from your experiences from the industry if you have worked for retail you know how the business process in retail works right so people often also ask from the technology end do we need any sort of experiences in Python language right python or like for example our programming as well uh so python is one of the most uh looked out for kind of programming skills particularly when you want to build Solutions in data science domain and with availability of libraries like numai pandas python has now established its ground very strongly in giving a very robust framework for Designing data science Solutions right and in particular things which it has like L dictionaries tuples and stats are one of its own sort of one of its capabilities which sets python in its own League of programming languages suitable for up coming out with data SCI Solutions so there are many more other libraries as well apart from this for building machine learning algorithms but these are some common uh libraries that you would normally find uh people using it and also with distributions like Anaconda python has shown its capabilities even for a production grade solutioning right wherein you make sure that all the dependencies that one particular library has for building a data science solution is all in one place so quite a popular programming language uh in the recent kind of happenings Although our programming is also equally good uh in terms of producing a quick prototype for most of your modeling tasks but python is moving itself into a production grade where things can be deployed after the sort of prototype into a production environment and it can phas the customers from day one so that sort of capabilities are coming up with python right so let's talk something a bit more specific with the data right so when people are doing any sort of data analysis they normally face something that we know by the name selection bias right so what is actually a selection bias the fundamental place where you start doing a data analysis is by selecting a representative sample right so that's where we like normally start doing any analysis so when you like working for a company which has let's say 1 billion records in their databases like 1 billion is like very very large number which represents various customer uh customers data or it might also be depending on which feature you are working on and so on so if you collect all of those it might very easily come to 1 billion records which is nothing but in a structured form form number of rows so with that an enormous amount of volume in the data any analysis that you take up might have to have lot of filters like saying I only want to analyze a particular feature in my product let's say right and I want only want my customers from XYZ region which is let's say top four region or top five regions so you might like put many features or filters like this but later on U if you would like to do a kind of an analysis which covers most of your customer Custer base right there comes the tricky situation of not being able to use the entire volume of 1 billion records at the same time you want to do a really good study uh or analysis based on what data you have so in statistics we normally use this idea of doing a kind of a randomized selection right so with this randomized selection we make sure that out of these 1 billion records we are choosing a small seet let's say of 1 million which is a true representation of the entire population right but what happens with this is while we do this 1 million record selection in a randomized way there are chances that you might have certain bias in the analysis obviously because you are not using the entire population so selection bias is this particular sort of a characteristics while you are doing a sampling on a large population of data very common example for this is uh if you want to do a exit pole analysis of particular election even before the election results are coming up and you have not chosen a repres resentative sample for doing that exit pole analysis right by that you I mean you only have asked some questions for a selective few from the particular constituency and they have a opinion towards one candidate but that doesn't represent the entire opinion of the population in that constituency so selection buys is like very important to handle and most of the time people employ things like randomized selection or selection sampling techniques like uh stratified sampling and so on so by that you can minimize this election buyers so these are some very generic questions so let's start with some sort of statistical questions and also how to deal with different types of data okay so doing any sort of data with a structured information so structured information I mean there are many rows and like uh many columns and it looks more like a tabular data right so with such a data in in place there are like these two different formats one saying the long and wide so let me like show you an example here you have a record of two customers right and you store just two values which is the height and the weight so these are like in columns so with the these two customers these height and weight being a separate column is one format like a long format let's say transform this by having only one particular column which will say attribute right by that I will bring in these two columns as one column by calling that as an attribute and put the values in one column so this sort of format is called the long one so what really happens is instead of having two separate columns for two of your attributes you put all both of those into one column and by doing that there are a lot of benefits with respect to the task you have in your hand particularly in data visualization certain data visualizations would need you to not put your attributes as a separate column but rather as the one column which can have the attribute names so which might then go into building your Legends right so these kind of techniques are kind of very common formats between like the long and the wide and very frequently will people like deal with both these data formats depending on what task they are doing particularly when we are building visualization dashboards okay talking a bit more uh on the data analysis perspective uh people like uh know that in stats normal distribution uh kind of is that one like the Godfather of the distributions you have many distributions which normally people try to see if is present in my data or not but the moment people kind of see normal distribution coming up in any data things are kind of bit easy to understand and in typical case any distribution any data distribution that you would like to find out given data for analysis it gives us a lot of characteristics around what the data is about if I'm let's say analyzing the salaries of the employees in my company right I might see that there are some employees who are in that like that thick crust in uh the center where majority of the people are sitting with a moderate level of salary ranges and then there are these sort of extremes on the left and the rights so you are like very commonly people refer whenever you talk about salaries to a bell curve right and then they start talking about top 25% of the performance in my company the bottom 25 and the middle one which are like are sort of the normal performance so this sort of bell curved distribution is very commonly understood and as well as used in doing many data analysis so is the other distributions as well but normal distribution has its own significance so when somebody asks you around uh anything around normal distribution the first thing you should visualize is the symmetrical bell-shaped curve right and the moment you get that bell shaped curved in your imagination start thinking of certain properties like what is the mean of a normal distribution what is the standard deviation of a normal distribution and in particular a special case of normal distribution which we call the standard normal distribution so in that standard normal distribution we know the mean is exactly zero all the time and the standard deviation is exactly one right so there are different places where normal distributions are kind of used and if you are comfortable with ideas like Central limit theorem or uh the law of large numbers you might want to relate that as well to normal distribution particularly the centrer limit the right but the idea is a distribution which is symmetrical around the mean so that's what a normal distribution is so depending on which variable you are trying to analyze even a given data set whether it it could be like an employee salary or your sales in a for a business of yours right or your uh number of uh let's say interact C of the customers on your product so any variable that you define can have a symmetrical bell-shaped distribution which we normally refer by normal distribution and the moment we understand that something follows a normal distribution all these properties of that distribution is like revealed so that's sort of the importance of doing any analysis around the distribution of data and normal distribution like very common one and in many statistical techniques in even model building exercises if you have anything in a normal distribution um many other possibilities of applying certain modeling techniques comes out evidently there and also there are many other modeling techniques in stats and machine learning where there is an fundamental assumption that things should follow a normal distribution if it is not following then the model is wrong so there are many use cases of knowing what normal distribution is but in simple terms it's a symmetrical bell-shaped curve right AB testing so quite a popular approach people particularly who are working with product right and what happens is U when you are as a company having lot many features inside a product for instance if LinkedIn is a company which has a web page right it has lot of features inside that you have some jobs of portal in LinkedIn you have places where you can connect to your professionals uh in the similar industry you can also read about uh the post people are doing in the portal and so on so there are different places in the website with many features so what happens is if LinkedIn has a company is looking for some changes for instance changing the entire website's design and Aesthetics design and Aesthetics or changing one particular feature inside their website right so these changes are normally accompanied by sort of a process called AB testing so what happens as an analyst you might be working with LinkedIn and say on a fine day they might come out with a new feature new design or new sort of changes in their website so you as a person would tell okay this is my framework for testing this new change by defining a metric so my metric would be in simple terms saying if I change this website from A to B is my number of footprints on the website going to go down or not so this is my metric and if I successfully establish the fact that after rolling out this new website my number of customers who are going to visit my website is not going to go down I can be confident that okay fine this works now roll out this new feature so in this framework we normally have two set of users to identify the particular uh risk associated with uh getting this new feature into the platform and in which we in a randomized way put one user group and expose them to an older website and another user group and expose them to the newer new features right or the new website and when we compare the results of a particular metric like the number of clicks or the number of purchases and so on we should be able to see that these two groups uh are either exactly the same or quite different and if it is on the negative side of the difference we say the feature is not good and even if when the difference is not at all there we say even if we bring this new fature nothing is going to happen so this AB testing framework is quite Rob robust in its own um way right and a very common question if you have like worked as data analyst or if you are expecting to be like sitting for this data analyst kind of a role knowing AB testing framework is very important important okay so sometimes also when you do these kind of AB testing sort of analysis people normally come out with sort of saying uh what should be the sample size of my uh users whom I should be getting to participate in my AB testing framework or also when you are building some models you might uh see that there are certain statistical measures which has to be evaluated by the end of model building exercise like if if you're building a machine learning model let's say and you want to see if the those metrics on which I'm evaluating are really good or not right so in that sensitivity is one of those uh methods or like metrics which we normally evaluate and I'm going to show you some uh something that we normally refer by the name confusion Matrix so I'll spend some time explaining this and then come to what do we mean by like sensitivity so let's say you are building a model right a model for predicting whether a particular customer is going to purchase from my platform within one month or not right so very simple problem statement which might include many variables that we would bring in and then finally build this model saying okay this is my final model which says with 90% accuracy whether my customer is going to buy from my platform let's say an online e-commerce platform within next month or not so this is my model so now as without going into the details of the model let's assume that after you build the model you have got the results so while we analyze and evaluate what the result is all about we might come out with a confusion Matrix so what it says so obviously when you are building a model which follows a supervised way of learning you will see from the historical data after people have purchased in the platform within next one month whether they are buying or not so I can obviously create u a really good training data set right which will contain the information after people let's do the first transaction with me in next one month whether they are buying the next product or not so with that data set I train my model and uh the wave confusion Matrix puts that is by saying my actual data says something and you have predicted something right so let's kind of get into the details of that so this particular box which says my actual data says the customer will buy and you are also predicting the same so this we call the true positive the prediction is actually true right in a positive Direction saying uh the purchase happened now like move diagonally opposite to this TP which is this TN which is true negative which means the prediction of uh your model saying the customer will not buy is actually matching with the the actual data as well right so both of these values the true positive and the true negatives for the two cases is the right predictions from your model but consider the off diagonal elements the false negative and the false positive and these two cases this is sort of an error why because your actual data says the customer is not going to buy in this case let's say but your prediction says he's going to buy so the prediction is actually positive whereas the actual data is on the negative side so this is a false prediction right so you have your uh in this case your FP which is uh the false positive cases and on the off diagonal element if you look at this one which is FN which is your false negative for the cases where you are predicting the customer is not going to buy but in the actual data it says that customer actually has brought the product so in in this case the model is wrong so the type one and type two errors needs to be taken care of when you are building any machine learning model if these errors are low then your model is going to move towards that 100% accuracy Mark but normally any machine learning model has its own limitations so particularly there is one metric which we prer by calling sensitivity so what happens is U these true positives and true negatives needs to be controlled right if my model is very good in true positive ones like the positive cases of when the customer is buying but it is doing very bad job when the customer is not buying the cases in which the customer is not buying then the model has some sort of issues right it is doing only good in one place but doing very bad on the other case so I need to find out that by some trick so sensitivity help us to find that out which is in simple terms is the ratio between the true positive in the denominator we have all the cases of positive predictions so imagine now if the type 1 error is going to grow my sensitivity is going to come down so if my true positives are like very high the sensitivity will also be high so this is sort of what we call uh the statistical power if this sensitivity is really good uh I would say that my positive cases are predicted well and the exact opposite of this sensitivity is what we know know by specificity uh so we need to make sure that in a very good machine learning model sensitivity and specificity both are balanced right so in very simple terms this is what we like mean here the ratio of true positive by the total positive events there and as I mentioned both of these uh sensitivity and specificity play an good role when you want to evaluate a model's output right and one more common problem so these questions might be immediately following one another when people ask you about sensitivity and specificity as you know it's about the machine learning model's output right when the model is done you understand whether the model is good or not so in those cases we also come across some uh kind of issues like overfitting and underfitting given machine learning model right so these words are very common and the idea is depending on the complexity of your model you might see that you want to adapt very sort of exactly to your data points or you might want to do a generalization so for instance here if I have these uh red and uh blue dots here right and if I draw a curve like this which separates the red from the blue and when this separation happens I'm building actually a classifier using some sort of modeling technique but now Imagine by drawing a smooth curve like the one which is uh given in Black you might be overgeneralizing it right by which I'm mean there might be some red dots on the other side of this boundary you can obviously see that but the moment I kind of a bit more flexible by drawing this Green Boundary which actually covers all that issues which is coming out with these red dots on the other side of the boundary so this Green Boundary has like taken care of that but the problem when we are building any model right the idea is you need to generalize to the pattern found in the data so if you don't do that generalization well you are underfitting but if you do that generalization like too specific you are overfitting so this curve might be represented by some polinomial right but that zigzag kind of a polinomial might be bit more complex than a smooth curve like the one which is shown in the black so you need to be very careful when you are building a model particularly in the cases of uh regression models where it is represented by a line and a polinomial you need to make sure that the polinomial is not so complex at the same time not so simple then you will either end up in an overfitting situation versus an underfitting situation so we need to have a good balance between these two right so in summary when kind of statistical questions comes in it would mostly covering things like uh some basic statistical properties as you would be like very aware of like things like standard deviation averages how to interpret median how to interpret quartiles right the first quartile second cortile and so and so on and how what do you mean by percentiles right these are some basic questions a bit more complex in nature might be discussions around sensitivity overfitting underfitting these are like statistical ideas so you want to prepare maybe from a Basics level using the properties like standard deviation mean and so on till things like overfitting uh underfitting statistical the sensitivity and specificity kind of ideas so that will like make your ground a bit more stronger when you are going for the interviews and these are at least the bare minimum for you to understand in the statistical Concepts right if anything less than that you might like face some difficulties in the interview okay but let's also talk about now uh questions which are related a bit more on data analysis so let's see how what kind of data analysis questions which might pop up in the interview okay so some generally questions like this people normally do analysis on structured data which is in rows and columns but there might be cases when the data is not so well structured and those places the data might be textual for for instance uh in Twitter right if you're doing any sort of algorithms like sentiment analysis quite commonly known algorithms so in that case the sentiment analysis could be for a brand for a election campaign or maybe something else around your product features and so on uh so text analytics in its own a really large domain and in python as well as R there are number of libraries so in particular R has libraries like TM right the text mining package python as well we have packages like pandas packages like the numi ones right and also packages like nltk which is built only for natural language processing so it can deal with many different sort of text mining approaches or text analytics approaches so in comparison if you talk about as I said the the robustness in Python is much more than in R but in terms of features both are powerful enough with the libraries and uh packages that it offers right one of the fundamental sort of starting place when you do any analysis so when you are given a data set and you are asked to do some sort of basic analysis of what that data tells you maybe typically of questions like I am in a retail business and my sales in a particular region is going down so this is an analysis that is expected out of you and you need to dig through in understanding what really is the problem in the sales going down so if this is sort of the data this which is given to you you might want to First Look at the transactional data which is present in the system then you might also want to go to outside of your network maybe you might get the uh Sentiments of your customers from social media platforms and so on so there will be different sources of data that you will collect but often times uh collecting the data is not only the task right and not like only building a model or doing statistical analysis might like come very later in the stage but what comes before that after you have collected the data is to make sure that the Integrity of the data is maintained you get rid of all the unwanted noise from the data and then finally prepare the data for doing the sort of modeling exercise or doing descriptive analytics on top of it so this cleaning and understanding the data doing lot of Explorations with plot in essence takes close to 70 to 80% % of your time in any data analysis task so if your company maintains the data in a very well structured way uh this kind of heavy time which we spend on data analysis might be red the data cleaning might be reduced otherwise you need to like take this up for any new project that you take take up which for which the data is not available in in a prior or you don't have uh like any pipeline which do this cleaning you have to write it down of your own so very very important if you don't do the cleaning part and understand the data well the analysis or the models that you build might end up giving you a very very bad uh performance right so very important as I said 80% of the time people normally spend on this uh task right and often times uh when you are analyzing things like the example I told my sales are going down what do I do it is not possible to come out with such answers to complex problems like this with just one variable right so you might also want sometime stimes to move Beyond one variable and talk about let's say how to do multivariate or by variate kind of analysis so often times this question comes up uh where you like ask to distinguish between this univariate bivariate and multivariate Analysis and the idea is very simple in any sort of analysis uh it is not only one variable which kind of decides uh the end output of your analysis but there are multiple factors involved so when there are multiple factors involved you you might also want to look at things like correlation there are multiple variables you want to see if there is any correlation between these things sales are going down but because of what is it because my sales representatives are not going to the market or is it my products are bad or is there some other reasons so with all the variables in one place you might want to go and dig deeper to see if there is any relationships coming in the variables or not and when we collectively get all these variables together and do some sort of coherent analysis around the problem you come out with a really crisp answers to what you are trying to analyze right and moving on there is also times when people uh do some sort of grouping right with the data you do a sampling right you get a data set in your system or in in whichever servers you're doing the analysis but in that there might be lot many number of times when uh even the randomized sampling of getting the true representative from the population U like the might not work well right so in those cases you might want to do some sort of systematic sampling or maybe a cluster based sampling as well wherein you might decide to say I want to analyze the issue with only five regions in my mind and with the five regions I'm going to form different clusters or in the systematic sampling you might also want to say uh that with the five regions that I have got I might want to analyze only for one product right which is not doing that good in the sales so these kind of sampling techniques like the cluster based one or the systematic sampling techniques and there are different names for this people might be able to give a very good uh interpretation of what really went wrong in whichever sort of analysis you're doing so one example is like sales going down but you can adapt this to other analysis as well but the idea is instead of doing a randomized sampling by which we are not very sure which kind of data is coming in uh the the data set which we want to use for analysis but if you do it in a cluster basis a cluster or a systematic sampling you know exactly which clusters or which sort of uh regions in in this example you are like analyzing and in your end of the analysis you'll be very able to say uh this is not like a randomized sample that I have taken but from these five regions so there are many different ways of uh doing clustering cluster or or sort of the systematic sampling which kind of helps in this particular final end results of your anal to put your end results in the right perspectives right instead of doing a randomized sampling okay one more uh quite a useful sort of an idea kind of widely borrowed from the field of linear algebra and this is a bit related to what we earlier saw between moving from one variable to multiple variables right and Egan value and Egan vectors kind of a concept borrowed from linear algebra helps us to bring in some some in some way a linear combination of different variables together for instance in some complex analysis it might happen so that given data set it might have many columns right let's assume you have a data set with 1 million rows and uh let's assume 10,000 columns so and these 10,000 columns are some features there are complex problems like that but in most often like most of the time not all the 10,000 variables are useful right the input variables so what we can do is we might want to transform this data set in a lower dimensional Space by which we mean this 10,000 columns can be reduced to let's say only 100 columns right so EG value and Egan vectors are these ideas which helps us in this transformation and the idea is can this 100 variables be represented as some sort of linear combinations of the 10,000 variables and if I'm able to do that my Di dimensionality is reduced the time I take to do the analysis is kind of also reduced and uh the representability which will come with only 100 variables will go up right so quite a powerful idea Egan values and the Egan vectors and as I said the Egan vectors is kind of that linear combinations of many uh uh kind of variables there and uh this calculations around Egan vectors normally happens for a correlation or a Co variance kind of a matrix which as you know the measure correlation is also about how two two variables are related or how strongly two variables are correlated right so that's why we are also saying uh this Egan vectors can help us to compress the sort of data that we have right and that's because of one Ean Vector can be representing a 100 column 100 variables together right so that's sort of how it works Works quite a powerful idea and commonly used methods for reducing the dimensions of an large data set like the PCA principal component analysis is actually based On LAN value and igen vectors so if somebody asks you about igen value and igen vectors in an interview also talk about the PCA principal component analysis which is actually based on these two concepts so that gives them a good idea to the interview view like you know about Egan values and Egan vectors and you are also able to think of it application like in PCA right so we talked about this false positive and the false negative cases in our confusion Matrix example so this is exactly the same we also talked about the type one and type two error okay so but let's now also drill further and say examples or kind of scenarios when the false positives are important and scenarios where the false negatives are sort of important and the by the term importance we mean are we like allowed to do this mistake if we are building a machine learning model are we even allowed to do a mistake on either of the cases the positive or the negatives so for instance here if I take an example in a medical uh domain where we have let's say a process called chemotherapy which is uh normally given to cancer patients which is a radioactive kind of a therapy which kills the cancerous cells right so it is very focused sort of a therapy of on the cancerous cells so what will happen if you like predict let's say you are building a model for detecting cancers right given a CT image and this model would obviously not be 100% correct all machine learning model has its limitations but you are here required to predict this whether a patient has cancer or not and based on that radiologist might decide that whether the chemotherapy is right for this patient or not but imagine now if you have predicted somebody to be positive for cancer but the patient is actually not having the cancer cells there right so in those cases you might end up saying let's go ahead with the chemotherapy but the side effects of chemotherapy are like very very adverse right because you're giving these therapies on the healthy cells if the patient is not having the cancers so in these cases uh sort of the false positives becomes a bit more important so it it will be absolutely fine if if your model says the patient doesn't have cancer if even if there is like this slight possibility of cancer present in the cells of the patients but in that case you are not like exposing the patient with a chemotherapy there right which is like more harmful than saying that the patient doesn't have the cancer right so in these cases the false negative though the false negative itself is not so good in this case but at least with this particular example the false positive gets the importance than the false negative but both are bad as you know right from the confusion Matrix discussions so in simple terms it is better to not expose the patient with a false positive with a treatment like this chemotherapy like treatment then it is like much better than saying you don't have cancer okay and a very similar example uh in some other context might also come up so if you would like to think of some other examples in the same context okay so where is the other case now which is the false negative right so we talked about the false positives importance but uh there might be also cases where false negative might become a bit more important there and for examples like this if you are let's say building a model where you want to convict a particular criminal basis all the records and the arguments which has happened in the code right and let's see what would happen if you make a criminal Go free right because your model says it's false negative the though the person is actually a criminal but because your model predicted based on all the evidences you had the person is not a criminal so you're letting a criminal like walk free in society so that kind of is more harmful than convicting that person and maybe for a prolonged period you might also want to get more evidences and build a stronger case so it is fine to keep a suspect in behind the uh bars for a longer period than letting the suspect Go free there when we like know that that it might be a case of a criminal going free from the Judicial Systems there right so in these cases the other case becomes more important so keep in mind it is very easy to get yourself confused between this false negative and false positive but if you keep an example in mind always and like don't give any room for confusion there though it is a confusion Matrix based on which these two ideas comes in you will be able at every time put these examples in front and talk from that so if you start to explain what false negative is in terms of the formula you might get confused but if you take an example and then explain things are much more clear for you as well as the person who is hearing that in the interview right and in cases when both are important typically the one which uh relates to banking industry you are building a model which will decide whether to give a loan to a person or not bases many of the input attributes that around which you have like collected the application from the customer but here you say if the customer is really good and you are missing the opportunity there of not giving the loan the customer is really bad in terms of its his or her credit history and you are giving the loan in both the cases in one case you're losing the business in the other cases you're taking a risk in which you will lose your money right so in this case and in this example both kind of has an equal role so if your positive or false negatives are in either of this is high you you will end up losing some chunk of your money there keep these three examples in mind and every time you get to like hear false positive false negative things should not be like confusing at all okay so now let's also talk about building a machine learning model so so far we have discussed about what happens after building the model right but let's step one uh uh like get one step back and see how do we normally build a machine learning model and what kind of processes we normally follow so when we are like building a machine learning model we know that we need to given a data set we need to divide it into different buckets or different parts so the commonly known uh divisions that we like have is your training data right then we also divide the data into something called test data and sometimes people also divide uh or keep one portion of your large data set which is called the validation data so often times people confuse between the test data and the validation data right so what happens is in the training process there are certain models where in while you are training you will use the training data obviously but in the process of training you can also involve something like a validation step right which will make sure that during the process there is one part dedicatedly given for the validation of the model and when the model is done you might see that the final model is well trained on the data at the same time validated but when the model is completely done then only you get into a process that we call testing right so you can imagine like this you have a data of thousand uh Records you keep some 700 records for training 100 record for uh validation and the remaining 200 records for testing so there are three splits right if I would like to explain this with an example this is how it will happen so there is a process called kfold cross validation in which K can be any number like between five and 10 mostly there are like standards saying fivefold cross validation or 10-fold cross validation and the idea is when you are building your model you will work with a training set right and a validation set in which a small portion of the data you keep for validation and the rest you use for training so what you see here test set can be like replaced with a validation set right and you can see that this is a rolling uh sort of subset and you keep changing it in each fold so you go for the first fold of the iteration you keep one validation set and the rest of it is the training set and the next fold you move this window to another subset and the rest is training data and so on and when this model is done using this kfold cross validation approach in the end you will get a model which you can then use on the testing data to see if the accuracies are good or not so this kind of brings in lot of performance improvements people have also found validation said to be an really good uh way of tuning the parameters in many machine learning models as you might know there are something called parameters typically in the neural network models and these parameters need to be tuned as the model is kind of proceeding so we cannot use the testing data set for tuning this parameters so validation set kind of comes very handy in those cases right so we just talked about the cross validation so when you keep moving this validation set in each of the fold first fold second fold third fold and the validation set is keep changing you kind of do this process of cross validation right and the idea behind doing cross validation is uh is to see how well is your final model generalizes to the data that you have so independent of which data you use for training your model should generalize uh because often times it happens when you train your machine learning model it works very good in the training part but when it comes to the testing the model does very bad the same problem with the overfitting and the uh underfitting cases so in this approach of cross validation you have made sure that your TR model has trained on various subsets of the data data right and in the process we also have this small validation set so that every stage of the Cross validation process you're able to use a different subset so that means you have trained your model very well so irrespective of which data you use your model is going to do well in the testing cases so that's how cross validation brings in uh the capabilities okay so with the two pillars like the statistical analysis and the basic uh data analysis I hope you are getting some sense of how people ask a particular question coming from either the model building perspective or from normal data analysis perspective right so we're going to know now go a bit more deeper into questions which might relate directly to machine learning so these are all the most questions that you have seen so far are either saying how do we do analysis after the model is built or how do we normally perform simple data analysis like AB testing Frameworks and so on but what if you are asked something very particular particularly from a machine learning domain right so the next set of questions will cover those part like people might also start with the basics like what do you mean by Machine learning so the idea I think must be very clear you are given a set of data points particular to a given uh domain right and you would like to build a learning algorithm which will take the historical data and predict something for the future so as we talked about in many examples finding whether a convict is actually convict or not like predicting if given the evidences if person is is a convict there or not or predicting whether we should be giving a loan to a customer or not right predicting the onset of cancer in a given patient by using the given patients historical records and so on and now this algorithms are now even becoming more complex like it is starting to work on speech data phas data which are mostly used for biometric authentication systems right so many many use cases coming up from various Industries right so in machine learning the most uh commonly used two types of learning the supervised and the unsupervised learning there are like two other types as well the semi-supervised and the reinforcement learning and the idea is around if you are given a set of input attributes do you have a label which can help to learn the input attributes around any data points that you have or you don't have right so if you have then the approach could be a supervisor learning approach versus if you don't have then the approach is more an unsupervised so some examples of the algorithms are like support Vector machine regression a Bas decision trees all of these are the supervised learning algorithms so in a very simple terms if I say you are given an input attributes for identifying let's assume a sort of given an image of different fruits based on the characteristics of the fruit can you like identify whether it is Apple bananas or the oranges so if I'm given that label with me the model will run keeping in mind that given these characteristics this is an apple this is an orange and this is a banana but in the other case if you go for a clustering approach where such a label of saying that it is a apple banana and orange is not available then we might just simply segregate the data points with the input features maybe with color with texture or with uh the shape right with that we can maybe say that but particular fruit with this shape is like uh you know into a bucket which we can like call Banana or another kind of a spherical shape it might be apple or an orange so depending on the presence of a label uh we can say either to use supervised or an unsupervised learning and both of these approaches are quite common and sometimes there are certain algorithms which can have the both ways it can also learn in unsupervised Manner and the supervised manner so depending on how you model the problem the fundamental difference comes from the fact on whether we have the label or not okay so when we talk about the supervised learning algorithms other name for superv kind of one of the types in the supervised learning algorithms is classification right and the classification is around given a set of input attributes and the label is like categories right for instance if it is fruit bananas apple and oranges like if it is a customer whom we want to predict into either uh kind of a defaulter type or a good customer so the classes are two like one which is saying the customer is is going to be a defaulter the other says the customer is going to be a good customer same is true for when you want to build an classification algorithm for detecting cancer whether the patient has a cancer or not has a cancer right or if you want to detect let's say a malicious content or a malicious file which might be a virus Frozen or uh warm or something else right so in that case the classes are now many so more than one class can also be there but the fundamental idea is We are following a supervised learning algorithm but the type of problem we are solving is a classification problem so instead of saying classification algorithm we can also say it's a classification problem using supervised learning algorithm so these are the various types of classification algorithms like the linear regression decision tree then you have the support Vector machines and so on right so now let's talk about one of the types of classification algorithm called the logistic regression right so very commonly used algorithms and Banks or companies as big as like American Express sort of have leveraged the logistic regression algorithms to quite a extent and they have like built a really really robust implementations of these algorithms particularly in banking sector cases like predicting whether a customer is going to be a defaulter or not given if I issue the customer a credit card or give a loan right these kind of decisions can very robustly be taken from a logistic regression algorithm and these algorithms are best suited for two class problems or a binary problem right where you have either yes or no quite a common technique as I mentioned uh and in all the possible cases wherever you have these binary classes of problems you might use a logistic regression a political leader winning a election or not somebody getting a success in an examination or not and as I mentioned uh whether to give a loan to a customer basis whether he or she or is going to become a defaulter or not and many of these like binar