Complete Life Cycle of a Data Science Project

hello all today we'll be discussing about the complete lifecycle of a data science project in my previous videos I have actually uploaded many videos on machine learning and deep learning so basically every use cases that I begin with was with respect to a specific use case I had some kind of data and based on that data in order to solve that particular use case I was applying some machine learning algorithm and deep learning algorithms so for doing the predictions for the future data itself so that is how I was actually solving the use cases in this particular section or in this particular tutorial we'll be discussing all the life cycle that a data scientist follows for executing a data sense project so the first thing to begin with is basically of you skills that we will be having so this is the whole diagram and this is the approximate diagram that I've drawn over here so that you understand each and every steps as we go ahead and till the deployment stage that we try to do for the model deployment itself so to begin with we have the start step and over here we have something like business understandings so first of all we have a use case suppose over here we have a use case so I am picking up this use case and with respect to some domain expert II okay I am trying to understand what this use case requires so first of all we need to have some business understanding regarding this use case after we have some business understanding suppose I want to check whether the patient is having diabetes or okay so close this is my use case that I am thinking about whether the person is having a diabetes on now for this obviously I should have some data right and the and the first step actually begins with is basically like my data gathering I have to prepare my diet I have to collect the data entropy from various data sources some of the data sources that I will be taking an example is basically hospitals right then we'll be having private clinics of doctors right apart from that other shared data of the shared data that are available for most of the diseases right suppose if I am taking for in the case of diabetes so different different data sources will I'll be actually referring to order to collect this particular table the next thing is that after I collect this whole data right after I collect this whole data the next thing is that whether the data is in the right format whether the data is in the right format see the data is being collected from various sources right so most of the data will not be in the right format at all machine learning algorithm will be X acceptable to take that particular input data for the prediction so for that particular step what we do is do we actually do lot of cleaning wrangling and exploration we basically do exploratory data analysis what we call it as EDA in exploratory data analysis what we do is that we try to handle missing values we try to handle missing values suppose if I have some missing values in my data and apart from that suppose if I have many features in my data set what I do is that I apply something called a feature engineering I apply something called a feature engineer now let me just discuss about what is featuring linear feature engineering is a step wherein we will try to dig examine each and every feature you know and try to convert that feature into the right format so that it is acceptable for a machine learning algorithm to adjust so that basically means that suppose in some of my features I have something called as categorical variables so I'll try to handle the scattering variables by using something called as dummy variables concept that I have already discussed in my previous machine learning algorithms right so in the previous machine learning algorithms you can see that you have used something called as one hot encoding and label encoding to handle tactical variables apart from this we will also we will we will also try to replace the non values with some suitable values if it is possible and most of the techniques that is basically used is something called as me medium and more right so those techniques what it helps us to do replace the land values with the average values of that particular fault so that is what is the example of each engineering you know we'll try to claim with each and every feature try to put it in the right format so the machine learning algorithm will be able to easily understand what a particular input is now after feature engineering right we will also be having a step which is called as feature selection a feature selection is also a very important step guys feature selection basically says that we should when we are going to apply any machine learning algorithm in it will first check whether each and every feature is very important for that particular data solution data problem or that particularly useless whether each and every feature is actually correlated to the output that we are trying to find out if it is not correlated we should remove those features right so in regression problems some of the feature selection techniques are something called as back elimination methods so back elimination methods what we do is that we try to remove the features from the particular data set we try to find out a p-value of a value so suppose if we say that the p-value is less than point zero five we will keep that feature otherwise we will not keep that thing sorry we will keep that feature otherwise we will not keep that feature that basically means that that feature is not playing a very good role in actually determining your output so what this is one of the technique that is called is back elimination method the other ways are basically by finding correlations between the that particular and the output right whichever feature we are trying to check whether that feature is actually important or not so correlation and covariance is another good technique to find out whether that feature is playing a better role so after the data is actually created data is actually collected and after that it is processed right so we can also see this particular stage as data analysis as unusual as usual data analysis or data pre-processing and always remember in data pre-processing we have two steps one is feature engineering and asked of doing feature engineering we have something called as feature selection now the next thing is that after this particular step is complete and now see this we are going in this particular step then we have completed all these things right we have understood what the data is and we have made the data into the right format we have done the pre-processing techniques we have applied feature engineering we have applied feature selection we are handled on values and that is basically called as exploratory data analysis right now after that we should also check whether our date in this teacher engineering we should also check whether the our out whether in our data set is basically having a balanced data set or it is they're having an imbalanced data set so balanced an imbalance data set basically says that suppose if you are trying to find out whether the person is having diabetes or not and suppose in your data set you have around like you have around thousand records now suppose in this particular thousand requires you are having like nine hundred feel records us basically saying that the person is not having diabetes and suppose only 100 records are basically saying that the person is having diabetes this is a complete imbalance data set you can see because the data set range is basically between nine is to one which is very very bad right now if you have an imbalance data set what the problem usually happens is that some of the machine learning algorithms that you will be applying will get completely biased to the maximum number of outputs right suppose I have 900 record saying as no so what my machine learning algorithm will happen what will happen to that it will become biased to this kind of values you can go into my clear list and see that how we can handle imbalanced imbalance data set right I haven't actually explained about that whole techniques right now apart from that once we complete this whole section of data acquisition and understanding you know and we are completing the data analysis and data pre-processing stage we will go to the next stage that is modeling now once my data is ready my right I have the right format of my data the next thing is that I will try to apply a machine learning on deep learning algorithm now the next thing applies that what kind of machine learning algorithm I will apply they are various kind of machine learning algorithms and they have various ways to select how much machine learning algorithm and which machine learning algorithm means to get applied the one way is basically is with respect to accuracy then we have something called as confusion matrix indefinitely if I if I have a data set which is imbalanced I can't only depend on accuracy I have to go and see my confusion matrix where I am where I'll be trying to see my to positive values through negative values false positive values and false negative values right if I have my an imbalance data set I have to consider the accuracy I have to consider confusion matrix then I also have to consider something called as cross validation cross validation is a very important technique wherein once we divide our data set into training and test different type of train and test data set will be created and the accuracy will be the average mean of all the predictions all the all the predictions of the test and train data so cross validation is also a very important technique for doing that then based on this balanced and imbalanced data set we have some of the algorithms like decision tree random forests which will not be impacted by the using the imbalance data so modeling will basically be applied with feature engineering model training and model evaluation the model evaluation stage we will be checking with respect to accuracy confusion matrix and cross validation you can consider feature engineering here also you can consider feature engineering here right so that can be a part of data pre-processing step also and it can also be the part of modeling step once my model is ready the next step is basically deploying that model now before I make you understand why should we deploy a model you should basically understand that whenever we create a model right a model is just like a black box it is just like a black box we give some input to the model it will be predicting some output that's it that is the work of the model right so because of this what we do is that we deploy this model in the form of services in the form of api's right we web services you can call it as web services now this web services works like this kind of black box model we need to deploy this model into some web server wherein we will be creating an API on our web services from it and what we have to do is that then from a front and we just need to hit this particular web service give our input data over here and get the output data and based on this output data we should apply some changes in our website and try to show the information that we want to show to their end user and that is the step that we are actually doing in the deployment step in the deployment they are various means of doing well deploying the web service one is basically flask right and now EWS that is Amazon Web Services as your is providing a lot of features where you can deploy the machine learning model into them and you can actually create a Web API from that and then that particular web API can be integrated with any any front-end application with and it is of interoperable basically can you can connect it to any any programming language whether it is dotnet or whether it is Java whether it is - whether it is C++ right so that actually provides an interoperability feature where you can just fit to that particular API with the input that is required and you get particular and you get the output from that right now the next thing is that after the deployment there is always there is something called as you need to observe the accuracy of this particular model that it is giving for the real-time data right so usually after the deployment one month of time is usually taken for testing you know one month of time is usually taken protested wherever we provide a new kind of data what is actually the output if the output is good the output is good right the output is good in terms of accuracy then we say that the model is good right if the output is not good then what happens again we need to follow the same step that is start from feature engineering go to the model training and go to the model analysis again but before doing feature engineering we should also consider this step of collecting the data sources suppose if you require more data source wherein suppose if our accuracy is not good then they may be also probabilities that I need more amount of data to train my model so I may also go back to this particular step and try to collect more data more data from different data sources once I collected I follow all the steps that is clearing wrangling and exploration then provide my new data to my modeling along with the previous data you know and do the feature engineering part model training part and model evaluation after I do this again I deploy and this is again a continuous process you know as as time goes on you write new data gets created again and again get people sentiment may change people lifestyle may change the health way so basically the health of the people may change and again new data is actually getting generated right so this is a continuous process this is a continuous process so suppose after three months till three months suppose my model that is deployed right is giving me good results but after three months suppose it is not giving us good results then what is this next step we will again follow this whole lifecycle variables go and collect our new data do the cleaning landing and exploration process send it to my modeling section do my feature engineering do my model training gear and then this model training we may also change our machine learning algorithm we may also change our machine learning algorithm this is very important to understand the some of the machine learning algorithm may not work with the new kind of data that we have okay so we may change our machine learning algorithm and then again we do the evaluation and we do the deployment process now you may be understanding that now you may also be you should also understand that the reason why I am explaining you this is that till now till now I have actually uploaded videos on various kind of use cases for machine learning and deep learning right so in my future videos I will be showing you how to deploy your model how to deploy a model and over here the type of environment that I will be taking is basically AWS assured web services and when I take AWS into consideration I will be showing you how you can use few Burnett's how you can use Dockers you know to actually follow that continuous process of deployment right I'll also be showing you how to drain the model internally so that you you can actually do this process continuously now along with the web services I'll also show you how to create the web services how you can integrate it with your front-end application with a particular example so this was a whole lifecycle of a data center project I hope you like this particular video oh please do subscribe the channel if you have not subscribed I try to upload many as many as four videos and I also have planche to upload around 100 hours of content and this in this coming two to three months so please don't miss it please do subscribe the channel and God bless you all happy learning thank you one and all [Music]