Implementing Machine Learninng Pipelines USsing Sklearn And Python

Key Takeaways

hello guys so in this video we are going to discuss about pipeline and machine learning with the help of a scalar and python now uh till now we have solved so many different kind of machine learning problem statements you know we have seen what are the different different modules we really need to solve you know like feature engineering and in feature engineering it can be handling missing values doing some kind of column transformation um you know probably uh trying to apply a dimensional trade reduction techniques like pc and all and then finally applying a model and then deploying a model now this particular process you can obviously do in jupyter notebook also line by line but when we go ahead in implementing an end to end solution right the initial eda analysis can definitely be done with the help of jupyter notebook but when we actually convert this into a modular coding it is better that we start to start using pipelines now pipelines sequentially apply a list of transforms and a final estimator final estimator basically means any kind of machine learning algorithms so in this video i'm going to cover about pipelines i'm going to do a detailed explanation about it how we can basically implement how we can apply different different transformation techniques with the help of pipeline and we'll also see that how we can apply clustering algorithms how we can apply any classification or regression algorithms also please make sure that you watch this video till the end because this will actually help you to understand in the later stages how to implement an end to end a complete machine learning problem statement okay so let me go over here and let me start discussing about pipelines now first of all as i said over here from the definition you can see that it sequentially apply a list of transformer and a final estimator so list of transformers can be anything like they can we can basically apply feature scaling technique we can basically apply imputing technique we can basically apply a dimensionality reduction technique and at the end of the day we will be applying a machine learning algorithm okay so let's start with the basic thing how do we go ahead with applying uh this pipeline technique so first of all i am going to import from sklearn.pipeline import pipeline right so uh i will just go ahead and import this and let's say i want to apply one simple technique of feature scaling let's say for feature scaling the most common technique that we can definitely use is something called a standard scalar now let's say if i really want to apply the feature scaling technique obviously you know that which library i'll be using let's say from sklearn i'm going to basically use something called as preprocessing okay pre-processing the okay i am missing the spelling okay import come on import standard scala right so once i do this right and probably let's say that i want to apply a simple linear regression model so for that i will be using from a scalar dot linear underscore model i'm going to just import logistic regression or any kind of algorithm that you basically want let's say in this particular case i will just use logistic regression and this is my algorithm i am specifically going to apply okay now i want to combine this standard scala which is a transformation technique okay we have just taken one later on i'll show you multiple examples how we can combine many okay so that is the reason i'm telling you try to see this entire video this is super important video and this will be super helpful when you're implementing end-to-end project okay so here i'm going to combine this technique this is one is transformation technique and this is basically my estimator estimator basically means the kind of model that i really want to apply so in order to apply this first of all i will just go ahead and create my steps okay what all steps i really want to follow and whenever we really want to create the steps i have to basically give this in the form of like a key value pair let's say in the tuple right so here whenever i'm giving with the help of key value pairs inside a tuple it becomes a name tuple okay so the first step that i want to basically talk about is this is my standard scalar so this is the name that i'm actually giving okay and this will basically get initialized to my standard scalar okay so this is the first first step that i'm going to apply after this after this i will go and go to my next step in my next step i'm just going to apply my logistic regression okay or logistic regression i can just give a simple name let's say i'll say this is a as a classifier problem okay and then i'm going to basically initialize this into logistic regression right now this is perfect right so once i execute this so this is my steps or list of name tuples and here you can see first one is standard scalar and second one is classifier okay now if i really want to convert this into pipeline all i have to do is that whatever pipeline i have actually imported so i'll just write it down and i'll just going to give the steps okay steps now if i press shift app over here you just need to give the steps pipeline of transformers with the final estimator here we have a final estimator we have a uh uh transformer right transformer basically means i'm not saying that an lp transformer but a list of transform which i am actually trying to perform okay now once i execute this here you will be able to see that this is how my pipeline looks like so whatever data i'm actually going to go give it will go first to this particular step and then it will go to this next step okay when i do the fit okay and then automatically it gets trained let me just save this in over here inside my pipe variable now the next thing that i'm actually going to do is that i'm trying to visualize this particular thing okay and we have techniques to visualize this uh entire pipeline also in an efficient way so for visualization technique the technique that we are basically going to use is something called a set underscore config so in order to visualize this pipeline and this is also super important okay obviously we can see from here firstly applying standard scale and then classify so here i'm just going to see visualize a pipeline okay and for visualizing the pipeline what i'm actually going to do is that i'm going to import from sk learn i'm going to import set underscore config okay so once i do this and then uh if i just set it to something like set underscore config with display as diagram i'll give you all these materials you can definitely try it out but i really want you all to see this video till the end and please make sure that you hit like also so once i do set underscore config with display as diagram now if i probably try to display the pipe okay here you'll be able to see how easily this entire thing is basically we can basically see that what all steps are involved right here you have standard scalar here you have logistic regression this is also perfect okay now what i'm actually going to do with respect to taking a simple data set i'll just try to pass through this particular pipeline okay and this will again be very very super easy step all i have to do is that take up any data set that you want and for this i'm just going to create my own data set okay so for creating my data set what i'm going to do and you can apply this to any data set let's create a data set over here now creating this specific data set i'll say from sklearn dot data sets i'm going to import something called as make classification since this is a classification problem and you can basically create different different kind of data sets like make moons make multi-label classification make regression you know so that you don't have to be dependent much on the data set initially just to practice i'll just go ahead and make right make classification now inside this what i'm actually going to perform is that i'll just initialize this with x comma y and i'm going to say okay i'm going to call make classification okay and uh let me just initialize by default if you go and see this okay with respect to mix classification it is going to create two features right sorry it is going to create number of features as 20 and this is number of samples is 100 uh number of classes in that specific feature right this y value will be two okay so here you will be able to see all these features and how many data points it is creating 100 samples if you want to change the number of samples you can basically write it down okay it is up to you let's say that i just want to use n underscore samples as thousand so that you'll also be able to see some good examples over here so if i probably go and write x dot shape here you will be able to see thousand columns as thousand rows and 20 columns okay now the next step is that i will first of all do train test split so i'll say from scalar from scalar dot pre-processing import train test split okay so let's see sk learn train test split i think it is it should definitely be in the okay it is not in processing it is in model selection so model selection import train test split okay and obviously i will just go ahead and copy and paste this particular code uh let's be little bit smart instead of writing each and every line of code so i'm just going to copy this let me hide this and here we go perfect i'm just going to copy this and i'm going to paste it over here to create my xtrain and my train data perfect now after the train test split here what i'm actually going to use i'm going to use my pipeline and i'm going to do the fit on my extra and white rain data okay so all i'll be doing pipe dot fit on x underscore train comma y underscore train but now here you can see that we have not done standard scaling separately all the processes inside the pipeline so once i do dot fit on x twin and y train i it'll basically going to pass through this particular pipeline and finally you'll be able to see that it will get trained for this logistic regression in short when it when my entire data goes right in the standard scala it is going to take that train data internally it is going to perform fit underscore transform and then it is then going to train it with the help of logistic regression so once i execute this here it is see that my fit has actually happened okay and probably if i really want to do the prediction now dot predict right with my x underscore test here you will be able to see that my prediction will also happen okay so now in the prediction now you should understand in the prediction this x underscore test when it passes through the standard scalar pipeline it is not going to perform fit underscore transfer is just going to perform transform and finally you'll be able to see this entire predictions okay so this is how easily you can basically use pipelines like you can combine transformation technique feature scaling technique and all and you can do this okay so this was one of the examples that we specifically performed over here now let's go to some more complex examples and let's see that how we can combine multiple processing steps and classifiers and how we can do this now what i'm actually going to do over here is that i'll first of all uh again uh use different different techniques now one technique is that i can apply standard scaling i can apply dimensional reduction technique and then i can probably apply a classifier okay so let's go ahead and let's do that and let's try to see one more example so this is my example second okay example second and in this example second what i'm actually going to do i'm going to display a pipeline with dimensionality first of all i will try to do with the standard scalar then dimensionality reduction and then estimator okay just trying to make more complex things so that you'll be able to understand okay so i'll make this as a markdown okay so as usual uh what all things i actually want i will just try to import it first of all in order to use pca i'll say from sklearn dot decomposition import pca okay so pci has been imported then from scalar dot let's say i want to apply an other algorithm which is called as svm and here i'm just going to apply the svc okay so these are the two algorithms that i'm going to do it uh let's see what is the problem over here okay some some warnings it's okay we can actually handle this so it's not a big issue with respect to warnings standard scalar i've already done that now let me go ahead and write all my steps what all steps i'm going to follow so this is my first one now in the first one what i'm actually going to do here obviously we need to do feature scaling i just write scaling for the scaling purpose i'm just going to use uh my standard scalar again which i have actually imported long back and then the next one the name tuple will basically be happening with pca okay and here i'm just going to basically write a pc i'm going to initialize it to pca and number of components also i can basically give right now that 20 components whenever we initialize with respect to this particular data there are 20 features i want to reduce it to let's say two features or three features let's say i want to reduce it to three features okay so this is my second one okay so the third step after applying the pca what i'm actually going to do i'm going to apply my svc model okay since it is a classification problem so here i'm just going to apply it to svc okay so these are my total steps and all i have to do is that just go ahead and write pipeline and then probably convert this into this and here is my entire pipeline so this basically shows that in the first step i'm going to apply the standard scala pca and then svc okay so i'll just say pipe two pipe two okay so this will basically be my second pipeline okay and now all i have to do is that just go ahead pipe two and probably just write dot fit if you really want to directly use this uh with respect to any fitting method with respect to my extreme data so that it will pass one by one but there are also some questions there where which people say that okay chris i just want to apply let's say that i just want to take up this scaling technique and probably see that how we can basically apply just standard scaling that that can also be your scenario right you don't want to apply all of the thing let's say i just want to verify with one of them and i can basically just check whether everything is working fine or not so how do i do that so all you have to do is that just write this particular name that is the reason we are giving this name right so if i write pipe two dot scaling dot transform or dot fit underscore transform and that this step let's say if i write fit underscore transform let's see whether this will work or not and once i do the fit underscore transform this i will do it in my x strain let's say so once i execute here you can see that yes it is working so independently also with the just help of the key name you can take it up and you can apply that specific function okay but right now i don't want to apply this i want to apply one by one all the steps right so all i'll do i'll write pipe to dot fit on my x twin comma y train data right so once i execute this here it is perfect everything is there and if i want to do the execution type two dot predict and here i will just try to predict all my extras data so with respect to this also you can basically do the prediction right so this was the second example how i combined three over here two feature engineering techniques like one is standard scaling one is reducing dimension and then finally applying an estimator now uh this is not it guys we are also going to apply grits or cv so please make sure that you watch this video till the end now i'm going to display a more complex pipeline oh you know and here i'm going to use something called as column transformer now see there are techniques which through which we can actually do feature transformation okay so here let's take up a complex example of column transformer and this is super super important okay and we'll try to see how we can basically do column transformer also right now i think this steps are used a lot you know with respect to implementing anything you will be able to use this now first of all what i'm actually going to do is that let's say whenever we get a data set the first step is that we basically try to handle the missing value now for handling the missing values we in sql we have something called as from sklearn dot impute i'm going to just import the simple computer okay this is specifically for numerical variables so category variables we can use some other techniques uh but for numerical variables we can use simple computer okay uh so let's go ahead and import this and let's start creating the pipeline with different different steps okay so though and what we can also do is that we can create multiple pipeline and we can combine those pipeline together also right we can we can let's say that for uh processing numerical data we can create a separate pipeline for processing categorical data we can create a separate pipeline of and then we can combine both of them so that it can be applied for the entire data set now this is super important so please make sure that you watch this very closely okay so let's say i'm just going to name it like numeric processor okay numeric processor and here i'm just going to create my pipeline pipeline okay so this pipeline are there and obviously inside this i'm just going to define all the steps that i'm going to use so steps along with my uh name tuple okay so here first is that i'm just going to see what kind of imputation i will try to do it let's say if i have a numerical variable i want to just perform the imputation with replace the missing values with mean so here i'll just initialize to simple computer and then this will basically be my missing values which is treated as np.nan so for this i'll also use numpy okay import numpy as np okay so wherever there is a np.nan uh the type of strategy that i'm actually going to use for the imputing i will say it as mean right strategy okay and again you can use different different strategy strategy here i'm specifically going to apply it as me okay so perfect uh till here we have actually done it uh you can see over here this is my entire one step okay and after this bracket i'll just open up another step okay so another step here i'm specifically going to apply standard scaling let's say scalar and here i'm just going to apply my standard scanner okay perfect so here uh i've done this step so this is basically my numerical processing pipeline that basically means whenever i have my numerical variable this is how my pipeline will look like okay first of all it has to pass through a simple computer then standard scalar and all now similarly this is for numerical right we can also do it for categorical so for numerical pipeline processing pipeline i'll just write it down processing pipeline okay and then i will be doing it for categorical processing pipeline now for categorical what all things we can basically use again uh we'll use simple imputer only but instead of replacing the missing values with something else we can replace it with a constant we can create a new category okay so that is what we are going to use in the first section so let me just copy this entire thing and paste it over here okay i'll just change the name to categorical in processor imputation over here instead of writing mean i will say dot underscore constant okay and then i'm going to apply the simple imputer instead of writing np dot nan uh i will just say that uh over here i will say fill there is also another parameter which we can specify over here and that is called as fill value i'll say just say that okay wherever you find out the missing value just replace it with the constant word the constant word can be anything like missing okay and obviously the strategy that should be used is a constant okay over there we saw called strategy that we used in numerical one was mean right here we are specifically going to use constant okay so this is the step that we are going to basically apply and usually after applying this right we for categorical variables what all things we can do we will we can we can basically apply one hot encoding also so for one hot encoding i will just import another library so for one not encoding what i'm actually going to do i'm just going to say from sk learn dot pre processing import one hot encoder okay so one hot encoder is done uh so here you can see that uh uh instead of writing scalar now i'll just write one hot okay so here will be my one hot one hot encoder with the initialize technique and whenever i find out any uh unknown things i'm just going to ignore them okay perfect so this is my categorical processor this is my numerical process now see i will try to combine this this will be quite amazing because now i'm getting my categorical processor now if i really want to combine both this pipeline and then probably uh try to create this in a way wherein i want to combine this both now okay but obviously i have to define my estimator also first of all let's go ahead and combine this now in order to combine this what we are going to do is that we are going to import another library which is called as from sklearn dot compose import column transformer now we are going to i am going to show you the importance of column transformer because at the end of the day you are combining processors or combining processing techniques right in this right so from sklearn dot uh compose you are just going to import the column transformer now in column transformer one thing that you just have to do is that just initialize this column transformer once you initialize this here you are just going to mention all the steps that you have right in the form of new numerical uh sorry name tuples right so here first step is basically your let's see what is the name over here computation mean okay here you can basically give your own name so categorical this will be your categorical and with respect to this you're just going to initialize with respect to this category processor okay so uh wherever let's say if you have any feature in your data set right so you can just initialize whatever feature name is there let's say if there is gender if there is age right or i'll not say gender but here you can also have city right so this let's what is this this is basically your column names okay gender and city where you really want to apply the categorical processor when category processor what you have first of all you try to impute the constant value with some constant that is missing value and then you apply the one hot encoding right now once you do this the next step what happens is that in the next step again uh after applying the categorical you just need to apply the numerical so numerical here you can basically define and this will basically get assigned to numerical processor and in numerical process of the third parameter in the column transformer that it can go is basically a column name let's say the column name is something like age or comma height right i really want to apply on this particular numerical processor okay so here is what you are basically doing with the help of column transformer and this will be my final under not final underscore preprocessor so i will just write it as preprocessor because still there are many things to develop okay so once i do this uh now if i probably go and see my preprocessor here you'll be seeing an amazing diagram okay so see this now this looks amazing right the column transformer has two things one is categorical and one is numerical and it'll go step by step right here simple computer will get applied then one hot encoder then standard scalar then simple computer right so this steps you'll be able to see that how quickly we have actually created this kind of pipelines and then when you are right creating an end to end project right like this only you have to actually go ahead and create okay now finally still we really need to add an estimator now in order to add an estimator i will also be using one more thing from sql on dot pipeline uh i'm going to import make pipeline because we have to basically create a custom pipeline over here now in make pipeline what i'm actually going to do is that i'm going to combine this entire thing along with that simple estimator okay along with a simple estimator so first thing first i'm just going to apply all the pre-processing techniques comma i finally i'll just apply logistic regression or whatever algorithm i want and this is done and this is my final pipe okay this is my final pipe okay and now if i probably go and see my pipe this is how it looks like you know so nicely column transformer categorical numerical simple imputer logistic so once this gets over it is going to go over here and this is just going to do the transformation technique now all you need is that you need a data set which has this columns like genericity age and height and probably you will be able to do this probably i can also create an excel sheet over here manually create all this particular data set and then you can basically combine it so in short what i have actually done is a combined multiple processing right processing techniques for categorical i can create a different pipeline for numerical i can create a different pipeline then i can combine this both with the help of column transformer and then finally i made a make pipeline where i'm combining it with an estimator estimator is nothing but it is basically logistic regression so here is how it looks like see this construct a pipeline for the given estimator okay now in the next video i will try to show you with respect to grid search cv also how we can do pipelining concept with gracer cv so yes i hope you have liked this particular video uh again try it from your site try different different approaches try to create this particular pipeline like this try to apply in a data set and then you will be able to get more and more perfection as you go ahead and don't forget to use google because i also use google a lot to in order to see all these kind of examples so in the first instance you'll not be able to get it exactly right but definitely see the combination that is possible okay so yes this was it from my side i'll see you all in the next video thank you bye-bye take care