Constructing Machine Learning Pipelines using Scikit-learn | DataHour by Anuj Dhoundiyal

Key Takeaways

uh so hello all hello folks uh today uh we'll be having an overview or uh we'll be having a complete session on basically constructing machine learning pipeline using scikit-learn so I hope you guys have some idea on machine learning and uh scikit-learn or SK learn as we uh use in data science uh but if you don't even have an ID of that I will try to explain it from scratch like how to build a machine learning Pipeline on what are the ml pipelines and if you have some experience then I will help you to improvise your Concepts and tell you more use cases where uh these pipelines are built and how it helps you during a development of your production level course right so a brief about me uh my name is I'm a data scientist at IBM I hold total uh 6.7 years of work experience uh my area of expertise lies in machine learning NLP natural language processing and deep learning uh my hobbies are traveling I love to travel and have short acquaintances with people where I meet and try to learn something new from uh from them and then I love to go to gym to keep myself physically and mentally fit and uh that's all about me uh during the session you can ask the questions and I think uh at the last we'll be accumulating all the questions and try to answer you uh feel free to ask as many questions as you want so that your concepts are clear and uh you basically have some fruitful knowledge after attending this session so uh this is the table of contents we'll be going to a machine learning pipeline what exactly is a machine learning pipeline then how uh psychic learn helps to build a ml pipeline now the flow diagram of a pipeline how ml pipeline looks like like what are the various steps or how to how you need to build up that and then we'll be going up to the pipeline concept and a column Transformer that is a key concept while you are building any machine learning pipeline uh column Transformer is something you need to be aware of how to use that and when to use that then some of the use cases where machine learning pipeline can be used and then some examples going forward and then uh the most important thing to the custom machine learning Transformer right so uh this is the main thing just because uh you try to deal with the real world data set where uh there are many uh pre-processing or feature engineering techniques you need to apply in them then you need to build a custom machine learning Transformers there in the pipeline you can have that Transformer and that can do the required task for you well uh when I will go to this topic I'll try to explain that and then we'll have some Hands-On session we will have a basically a python level code that can that basically explained how to construct a ml Pipeline and what are the various steps involved and then what are the libraries we are using and then and the last we will have a short q a uh where you guys can pitch in your questions and I will try to answer them right so going forward with the session uh just a minute if I can minimize this yes a machine learning python now exactly what is uh machine learning pipelines uh it's it's a basically when you talk about a pipeline uh the first thing that comes up to your mind is something uh a series of steps right uh it can be parallely executed can be serially executed but there will be series of step one step two step three and finally attaining the last objective of your uh complete workflow so it's it's exactly the same it's a workflow where you have the data and then you have to create a model out of that right a machine learning model now there will be various steps involved in that right whenever you have a data source uh first that data source will be helped to ingest the data then you need to perform some data cleaning so if you have some background in machine learning then you must be aware what is data cleaning and data pre-processing if not then I will try to explain it so suppose you have a data now uh the data in the raw form uh can never be used while building up any ml model like you have to perform various data processing right now what I mean by data processing data cleaning is uh in a data where uh some some kind of data you are recording right you are recording temperature of each and every day right so now there can be some values that are missing right uh so suppose a tool is recording the temperature of every day and sometimes the tool is not working right so uh some data points will be missing now how to fill that record because while building up any model you need the complete data you can't have okay I'm having a three years data but one year data is missing in that or some data points inside of some months are missing you need to fill that data right so uh this data I'm missing is a is a kind of technique you follow for data preprocessing State then data cleaning data cleaning you need to clean the data right so now uh data can be in any form you need you have to actually transform the data so that you can use in your ml pipeline so that you can actually use it in your model so these are the various steps you need to perform on the data so the data becomes suitable so for your input to the model right so the first is data injection then we talk about data cleaning the data pre-processing and then modeling and then deployment so today we what we will be discussing is uh accept the deployment the complete ml pipeline that is used to construct the complete workflow so as to create a pipeline or you can say an estimator or even a model right so people term it as an estimator too so estimator is uh only uh like you can term it as a ml model that you use and you can further use it for according to your use cases right so today we'll be discussing about that how to build that right now each stage of the pipeline as it is clear like pipeline is a series of steps right so the output from its previous stage gets other input to this next it's as easy as that right you have step one step two so in step two the input will be the output from step one right so whatever processing or whatever step you have performed on the data on step one the further uh process data will be flowing into the step two so you need to be aware how how to perform that and what are the key requirements when you perform these steps right then uh if you if if in short term you need to know like what is ml pipeline then it's all it can also be considered a way to codify or automate the workflow right now uh if you start building any uh ml model or if you if you have the data source then uh there are many ways to do that it's not only that uh from the first step only you need to create an ml pipeline so it's a way exactly to do that and that is the best suitable way whenever you are productionizing your code and whenever you need to code in a proper manner I would say right uh not not every component is uh not in a in a in a particular way which can be uh which your code can be readable right so then you use the ml pipeline but if your code components are scattered in a way and there are further changes then it becomes quite difficult that's why we use ml Pipeline and we say that it's a way to codify and automate the workflow because you are automating right you need to perform three steps you are automating it in a sense that in a less lines of code in the less number of lines and more readable code you are trying to achieve that right and in this you don't build and maintain a model right you rather you build and maintain a complete pipeline right so now if you are doing various pre-processing steps or using any uh pre-built packages right so if I can take an example of a min max scalar right what it does is a min max scalar is a kind of scaling technique so uh if you have a data right any data you can take and uh that data values are not in a particular range right you need to convert them so that it's basically in the range of a to B right now while you do that that's basically a free concept to ml right you need to convert those values in a particular range from so that it becomes suitable for your data otherwise it will require more computing power or basically that level of data is suitable for your further machine learning workflow right so if you're trying to uh apply a min max scalar on a particular column then you can easily do that using a ml pipeline also right the one way is directly applying that on a column but if there are further number of steps right so when you build that you build that in the form of Pipeline and now the final end product will be an estimator or a pipeline you can say or a model that you are trying to save it so that you can use it in the future for the prediction for the prediction case right whenever you build a model in the future you use it for prediction so that how activate your model is right so this was all about the machine learning pipeline now uh we'll be more focusing upon how to build a ml pipeline using scikitlan so uh people here if understand like what is scikit-learn like the people also term it as a scalar just because in Python you have that package by the name of SQL so people tell me that sk1 or Cyclone that is the same thing so for you I've in the session I'll be using scikit-learn terminology but that is same as the scale if you want to further uh apply this knowledge on the pragmatic approach then you can use that right so sqln basically provides An Elegant way to create a machine learning training workflow why because it has various functions its enriched library in Python and it provides various uh functions for you so that you can use them and build your pipeline right and what cycle pipeline will be it's a link of all the steps that we talk about a series of steps it what it will do it will shorten your code and like make is easier to read and address now uh readability of the code is a very important factor that we don't consider nowadays but yes it is very important just because in the future if any change needs to be applied in that python or in that code so whether whenever you have a ml proper ml pipeline it becomes very easy because it's a kind of plug and play right uh you need to apply a more uh like more feature processing step you can easily insert into a pipeline or you you can try to remove that or you need to modify that it becomes quite easy rather than having a code in which all the components are scattered it becomes very difficult so that is why ml pipelines are being considered in various workflows and now the more interest packages you get just like uh if I'll be talking about a hugging phase package like that is a package in NLP Community open source Community which can use it for various NLP tasks right so they also have built a certain pipeline mechanism right because it becomes very easy to use that and it becomes very easy for other people to use right they are open source that and you can exactly download it and easily use that by terming it as which pipeline to use the name of the pipeline and what model to use right so that's why these pipelines are now being considered at a high level and whenever you are having your code and you want to certainly deploy it in the production in your organization or you are creating your own r d course that it becomes very important to have those Concepts in your mind while building up that code right now uh definition of pipeline if you go to scikit-learn official Library it terms at sequentially apply a list of transforms and a final estimator now uh data of it as quite technical terms but if you try to break it sequence actually apply a list of transform that means uh see in a sequence you need to apply a list of steps Now list of steps means uh first you need to transform your data or all of the feature processing steps you need to apply now in that they term it as transform so because you are building a Transformer you are building a custom Transformer or you are applying transformation on your data that's why the term is the list of transfer you need to certainly apply a list of steps right and a final estimator now what is the estimator estimator is a model so uh if you talk about any data you want to do classification on that so the estimator you can use is a random Forest a logistic regression these are the certain classifiers I'm talking about right so these are offered term it has an estimator and estimator in itself is a model you you certainly have you have already libraries that support you you can build it from this class too but there are python English libraries that helps you to import those models and directly use it right so sequentially apply a list of transforms and a final list better now the intermediate steps of pipeline must Implement fit and transform methods and estimator must Implement fit now what does what does that mean this in particular I am talking about because people try to certainly look at the official library and don't know it does not understand the technicalities of the statements that are put uh put on that side so I'm trying to explain that intermediate steps of pipeline must Implement fit and transfer now every Transformer every transformation steps you are trying to build must have certain kind of two functions right one is fit one is transformed now why they are important because every transformation Steps either you are using it pre-built just like I talked about min max uh there is standardization standard scalar so these are the pre-built and now if you are trying to build up your own you need to build up your custom transform because there will be certain requirements where you will be working with real world data sets and you need to apply a custom Transformer just like if I be talking about any NLP data set right and in uh any NLP use case and you have a certain kind of data where you have the sentences and you want to classify them now in this certainly you need to perform custom transformation according to your domain use cases right so there is a domain use case of healthcare where you need to certainly uh do some some kind of custom transformation according to your use case and the domain of the data then you need to build a custom because the pre-build will not allow you to do that or they do not have the required functionality as per your use case then you like try to build a custom Transformer that's why it becomes very important in the real life data set that you know about how to build that we'll be discussing that but here what I'm trying to make you understand is the Transformers must have two kind of method fit and transform because whenever you try to fit your pipeline the first method that is called is fit another second method that is called is transform because first is try to fit your data and then transform the data according to whatever function functionality you have used in that and the final estimator that is the model must implemented a fit method so it must have a fit method just because the whole pipeline or the whenever you actually apply a DOT fit method on a pipeline first the transformation happens where a fit and transform occurs and then you are fitting like your model is at your data actually your transform data gets fit inside your model and your pipeline is ready now further you can use it for prediction as your use case is and you can do all all kind of steps right so that that is the beauty of the pipeline it tries to manages all of the workload that is required by you if if a new data point is there rather than performing all of those steps by you it tries to formulate what what is the pipeline what was the pipeline saved it tries to pick up those models apply those transformation and then give you the final output value so this becomes very easy for you right now uh this is the diagram of a pipeline a basic diagram I have put up put upon so that you guys are not confused much input is the data Final is the prediction now let's see what's inside is that right Transformers one two k why what this signifies is the various transformation steps so if you have a date time you apply three kind of transformation so basically Transformers one Transformer to Transformer three three kind of Transformations you are trying to build and then the final is the estimator so estimator can be a regression a regressor or a classify it terms it at whether the use case is a regression problem statement of a or a classification problem statements right it tries to apply that on your transform data and then it try to build up the prediction so the basic the if you if you try to understand the input of the data source the whole thing uh actually indicated in the green box is the pipeline and then you can use pipeline to actually further predict the new kind of values that will be coming right so this was all about what is the pipeline and uh what is what are the various steps now uh further deep diving into it we'll try to understand more because uh whenever you see a pipeline yes we are amazed that okay this is some kind of functionality that is uh readily available for us to use but how to use that and how to use it various functionality because every functionality comes with various Concepts right you need to understand what are the various Concepts okay so pipeline are custom uh column Transformer I would say uh custom will be uh having it at the last slide deck so first try to understand column transfer now uh when there was a pipeline what was the use of column transform I will try to explain it to any shot so pipeline whenever you apply you apply a list of Transformations so what happens uh whenever you have a pipeline and you apply a list of transfers like I want to apply Transformer 1 and Transformer to two kind of transformation like two kind of actually processing your data or you can say feature engineering so whenever you apply in a pipeline uh if you have a data set it applies the transformation on all of your columns right whenever you're using a pipeline it applies those transformation on the same column that that you need to do that so suppose uh there is a transformation and there is a data set a b c d I'm taking four features a b c d now there is a use case where you need to apply a different transformation on a a different transformation on b and a different transformation in C and no transformation on column D now this is a typical use case which you will find in a real world data it's not that all kind of like a signal transformation can be applied to whole uh data set no so suppose you have a column and that is categorical now you need to convert that categorical column into numeric so as to process your data in an ml model right now there is a b that is in numerical state but it is not actually in the particular rate like the values are spread out it's not a even distribution right so you need to conver convert that so that it it basically takes the values all of the values take some a value between a to B so it becomes easy for the further processing and there is a column see that about that in itself contains various null values it does not have many recorded values and you need to provide a particular value so that all of the uh attribute container significant value it contains the value so that it can be further given up to your model now these kind of use cases are very much you can say very much like very much in use in organizations right uh it it's all happened like whenever you have a real world data sets you need to apply different kinds of transformation on different kind of columns right so then comes the concept of column Transformer what what the beauty of volume Transformer is it helps you to transform each column set separately so uh in a pipeline what was that whenever you apply uh whenever you apply in a step it applies on the whole set of columns and then it tries to accumulate that right all of the columns are transformed but in a column Transformer you can apply okay I want to apply a transformation on a column this transformation this transformation of B this transformation C and then cumulated because at the end you need a whole transform data or a data set with a transform The Columns of a b c d so that it can be an input to your pipeline so that's why the column Transformer concept came uh came on and it it was a very easy to use that right I will explain you through the visual interpretations to how to exactly do that but let's try to as of not a register to understand why what was the use case of a column Transformer right now uh column Transformer as we know that it's a Transformer so you need to implement a fit and a transform method right as as we discussed right in the previous Slide the scientist learn official uh official page also states that you need you need to apply a sequence of steps to transform an add a estimator now here we are talking about a Transformer that's why you need to implement a fixed and a fit Transformer or transform method you can see two kind of method you need to implement so that whenever you are trying to fit the data first the fit a function will be called and then the transform function will be called right so here a short example is given so uh the same kind of exams I'm talking about like you need to compute the mean in column one and you need to basically one or 10 code now what is one not encode a brief overview like whenever you have a categorical columns right now uh a column with three kind of values such as a column of size right your if you are taking a e-commerce data set right now sure it says s small medium large like you need to convert them into numerical values you can't just provide a ml model with a categorical column right so OneNote encoding is a particular technique there are many more techniques but here I'm only focusing upon a kind of transformation so why not encode is the kind of transformation you can apply on your column and make it convertible into a numerical value so that the further your model can take take down it right now it uh it returns only column on which it has performed a transform now the one concept then in column Transformer that can be uh difficult for you is whenever you apply a column transformation it only returns a column where you are applying a transformation so it tries to forget all of the columns and take so even if you are giving the column Transformer your ABCD data setting you're trying to apply a Transformer on a column it will just apply the transformation and give you the a column it will just try to forget the b c and d volume now this is the negative but there is also a way to deal with that so it has an argument like what you want to do with the remainder columns right just part the just uh pass that parameter as pass through that means apply my transformation on a given column name and just bypass all of the column as it is to the second transformation because second will handle those columns right so there is a argument I will uh like we will see how to pass this uh basically parameter and where to pass this right now the default value is drop that's why I was calling whenever you apply a column Transformer and you give down a column name it tries to forget it it tries to drop all of the columns so that's why the default value is dropped but you can change the default value and make it as pass through now all of the columns the various other columns will be uh passing it through and the particular transformation will be getting applied already so this was about the column Transformer that you can use now uh let's try to understand uh what are the what are the uses of like what what what kind of uses uh like the way the Machinery pipeline that we built but what exactly is the use of that like why we need to build that right the first question that comes up to your mind that I can also have a code where it performs all of those steps now what is the what is the exactly the usefulness of doing that right now first is is try to improve your ml model see speed up the development and operationalization now speed up the development yes uh rather than a scattered code you have all of the code in a single Pipeline and uh like a beautifully synced in the number of steps so it will speed up the development and operationalization it will help you like in the further course of action whether whenever you'll be deploying your models uh in production rather than having different kind of model for uh Transformations for every other activity you will have a single pipeline it is it will be easy to maintain the complete Pipeline and for further operationalizing it in number of versions right in production there are use cases there you further need to retain your model so retraining of the pipeline also becomes very easy because then you try then you is it's easy to maintain the version okay this is my version when Pipeline and now uh this is my version two pipeline so that's why speed up the development operationalization uh create a convenient workflow now that is uh that is all we understood out of pipeline that it's very easy to create a typical workflow for that so for the uh for the other persons to whenever someone looks at your code it's very easy to interpret that and it is basically in a workflow so workflow what I mean a workflow is a series of steps right so it becomes very easy like okay uh you have created ml pipeline for this use case and what are the kind of steps you have used so it becomes easy readability easy in a workflow kind of step and whenever you're writing your code if it's in a convenient workflow manner then it becomes very easy for further to replicate that work too and uh whenever you have any changes so you can use it so I think that let's example of the four point flexibility flexibility is that only you you have various like you have the kind of flexibility where you can actually uh remove a certain step add a certain steps and if you want to modify it a bit it becomes very easy in a pipeline code readability we have talked about and scalability right scalability is also very uh important aspect of ml pipeline just because if you want to further scale it so this kind of things uh you get in organization like uh Whenever there is a real world data set and you want to perform a particular kind of transformation and it requires a heavy workload right a heavy computational power so it becomes quite easy whenever you're using a machine learning pipeline to scale your to scale that particular part right so that's why scalability is also also Factor it has various factors I've talked about few just to make you understand what are the uses it had various users now whenever you're using that they're kind of there can be many uses for it so I've discussed few of them right uh now just jumping on to the particular example because uh just we know that all the theoretical content is important and it is important for you to understand now whenever it comes to pragmatic approach of applying them uh certain people gets confused right how to apply them so this is a this is a very uh common example uh if you have a data set and you want to apply a standard scalar on all of your columns and then there is a support Vector classifier now this is a classifier support Vector machines uh this is a particular algorithm that is built up and it is readily available in SQL or cycle and packages where you can exactly use them so what I'm trying to do is I am applying I'm trying to apply a transformation called standard scalar it's basically a it's basically a DOT a kind of transformation that converts your column in a defined range of minus one to one so that's that's a particular defined range and it basically uh it's a kind of technique that hold together applies on your columns and they try to standardize all of your columns and then having the transform data try to give the data further into the classifier OR regression here I use a classifier right so it's as easy right steps uh steps will be you know that it's a series of steps so steps will be a list object and then there are tuples so in Python we have tuples uh tuples are just uh you you can use the the curly curly brackets and you can specify what kind of value so you can create a couple of two values three values so here we create a couple of two values first is a terminology now the scalar is not something that if I have written scalar so you have to also write a scalar no it's according to your use that if you want to term it as a scalar you can put as a scalar if you want to term it a transformation this is the basically user defined name that I have given to a scaling component you can give it as any in any name and the second is exactly the function the second thing is important when you ever you have these standard scalar is a pre-built package that is available in cycle learn but now as I have talked about custom yes I'll be discussing how to apply a custom transformation tool so instead of standard scalar you can have your custom Transformer too also and write a write a given name and scalar you can always change it right and now the second uh like first is this step and then now second step will again be your Tuple with first value indicating a name given to a estimator you can Atomic an estimator with the model and second is the function now SVC is a support Vector classifier that is available in sqln and it tries to like it's basically a classification algorithm so whether you're whenever you have a classroom kind of data set you can easily apply them on a data to get the result and if your data set is a regression problem statement then you can apply various regression techniques like uh like the linear regression you can apply so that uh that's a kind of regression algorithm that applies on your data and try to give try to keep the value according to your use skills okay so now this is the SK lamp this is the only package that allows you to have a have a functionality of importing a pipeline now pipeline accepts an argument called steps that is a list of Tuple values and Tuple indicates all of your steps so it's as easy right pipeline it will have an argument now uh you'd play up with that you can create a one two three kind of various transformation but it's a list value list of tuples that that that's particularly defined by the package that it accepts a list argument and where the list will be couple of values like values of Tuple basically the the various double values you can indicate and now it will be serially executed right first the scaling happens then the classification building happens all of that so it it's as easy to understand that pipeline is a pre-built package that is available for you you can import it and then you can pass them pass down the list of steps so uh in the below I've written all of that that I've talked about right so the only difference is whenever you have the pipeline that you directly call the fit method and whenever you call this pipeline dot fit so what's the beauty of this is as soon as you call pipeline.fit first the scalar will actually that this scalar component will work upon and internally it will call fit and it turned it will call fit underscore transform right firstly fitting your data and then calling the transformation and then your SVC classifier internally the fit method gets invoked right because your pipeline needs to be built so the whole pipeline is ready as soon as this two steps are uh like the first scaling happens then your estimator if it happens and as soon as this is early your pipeline is ready to use you can directly use our pipeline you can save it on the disk uh save it as a model that you can use it uh in the future and then you can directly call the predict now uh one more thing is if you guys are aware about grade search and all of those sites so now these are the kind of approaches where uh we go to an extent where we kind of uh like to find the hyper parameter so hyper parameters in a short term is the parameters of an algorithm so the actually the changeable parameter which you can tune it like the tunable you whenever you search it on a web uh people term it a tunable parameter tunable parameters it you can certainly tune it right uh these those are the hyper but before even doing that you need to understand the significance of that so A short example I will take it for you if uh you guys understand what is random process so Random Forest is a kind of uh classification algorithm okay now it has various various tunable parameters right now one of the parameter is what suppose random for the address name so is try to create trees like for us is a collection of trees right random part is also same a collection of trees right now collection of trees are collection of models so a tunable parameter called the depth of the tree depth of the tree indicates how long will your tree will be how long will your uh the decision three or you can say the in inside of forest how how long your trees can be right so this is a tutable parameter that is called a hyper parameter but in order to find the C there are default values given by the packages but every now and then those default values won't work for you in real life use cases because those values are there for you to use but according to your use credit can differ right so somebody have different kind of data somebody would have different kind of data but all of those parameters are tunable that's why they are kept tunable you can actually tune them so particular kind of approach is a grid grid search is a kind of approach where you can exactly find the most appropriate value according to your data set in simple terms according to your data set finding the most appropriate value of that hyper parameter grid search allows you to do that so I will not go deep into research that is a different topic all together but yes in order for you to understand that is that is the grid sum now uh the beauty of the pipeline is in even if you have functions like grid search you can pass down your pipeline into that what it will try to do do that take your pipeline apply the series of steps and try to find the best possible kind of parameter so those kind of parameters you can also you can also give the ranges okay my depth of the tree would be somewhere between 2 to 20 find the best value okay it can also do that because then it will take less computational power the more wide range you give 1 200 one to five hundred the more computational power because it tries to find uh I try to build the model for each and every possible value and then see that okay what is the kind of metric that is actually going to suit your processor so if you have given your accuracy as a certain parameter then you try to see that okay for bit for which value for which hyper parameter value I am getting the best set of accuracy that is the final hyper parameter value that you'll be trying to use so great search the algorithm that gives you the best kind of hyper parameters and here I discuss discuss about that is because grid sets is something that uh guys like you have to use it right whenever you are working with the data sets and in your organization and you want to actually uh fine tune your model you want to actually build up a model that is more suitable to your use case you have to apply those techniques so that's why this pipeline is an argument to your grid search packages too like gets our first functions too and it try to compute the best set of parameters so that's why the pipeline is important and is exactly usable in various kind of functions available right so now uh we have discussed about how to build a pipeline now let's try to understand uh the concept we understood in previous slide deck that what is a column Transformer we had a look okay what is a column Transformer now I guess everyone here present should know that I've easily explained you what are the kind of transformation but now uh you have a column Transformer and you need to understand how to build that path practically you have the theoretical knowledge now this is the this is the more uh of a practical way of representing how to build build that now column Transformer is easily available as a function in sqln packages again now how you applied that so there is a there is a price column in your data set so your data set has various kind of columns out of that one column is the price and you want to actually impute the value so I was talking about the basic values right so suppose you have a data set and there are regular prices that are being recorded for each and every day now someday the machine does not work and the price column is not even recorded what will happen right you need to take down that data in order to build the model uh you can't just uh give any and or zero values even if you can provide it as zero values but that will not be a more accurate model for you because that will be something uh that will be more uh relatable to the on the error side that's why you build a simple tutor is a package that tries to build up a strategy and strategy you can Define strategies mean that means mean of the prices it will try to impute it will try to impute whatever values do not have it right to compute the mean of the column and it will go so I am not saying this is the best strategy but it's a kind of strategy there are various other mean median mode and even if you you can apply a custom model to impute those values that is the beauty of machine learning so there are various kind of stuff but here uh on a particular column applying a transformation so we have column transformer for again a list of steps list of tuples here the Tuple is first his name second is the function and now in the square bracket I have given the column name now it can accept multiple because it can be right uh you have two columns and you you want to include the same kind of strategy you can pass it price comma a comma B on three of The Columns if you try to do the same value right now the key argument you can see here is remainder is also pass through that means I am applying a transformation on my price column but what happens to my other column just make it just pass through the entire technique I do not want to transform those columns those are the best available state for my model right so that's why remainder is also password is a key argument and that I have uh where you understand the default value is dropped so that's why if if your transformation occur on a price column all the other columns will be drop but you do not want that right because this will be in itself is a kind of Step you will be applying in the pipeline right so you do not want to play with that okay my all other columns will be drop you do not want that right so that's why a column Transformer is applied in this state and you can do that a modification with various columns we will having a handsome session in order to understand that uh I have written same right uh whenever you call them Transformer when you find that you you uh you know that in the behind that working you should have two functions fit and fit underscore transform right fit in order to fit your data fit underscore transform in order to apply that transformation so it's a kind of data pre-processed steps too right uh custom machine learning pipeline now this is an uh very interesting topic guys uh that's why I talked about uh in the last because before that you need to know what is ml pipeline what is the column Transformers what is Pipeline and uh there are function Transformers too I'll be talking about in the code session because that are quite easy to use as it name such as function Transformer just directly applying a function right now custom ml pipeline now this is the this is the kind of pipeline where uh you guys will be more into it because uh these kind of pipelines in the real life use cases you will have to build that right it's not always that you want to use a computer or you want to use the standard scaler on more kind of data set that you have an organization you have to build up custom techniques in order to clean that in order to pre-process that right that's why you have custom L pipeline now here uh in scikit-learn we have a privilege uh basically it have already built functions that you can use that and there are two kind of classes that are already available that you can inherit in your now inheritance all of you guys should know that if you do not know it's a very easy thing like uh you have class A Class B you want to inherit the properties of classes so you you can have your class B and you can use the functionality of classmate just easily by inheriting classes right that that's the beauty of inheritance so here also there are pre-built classes the pre-built methods you can try to automatically inherit them in your class that you are trying to build it in a custom way right as easy as that so there it it is very useful and we will try to see that rather than uh having a theory uh I will try to explain it through the visual interpretation right class custom Transformer now this is the custom class that I have created right now custom Transformer I can easily write down anything what whatever suits my need right uh in the brackets this is the kind of inheritance you are trying to inherit two classes base estimator and Transformer mix in from the SQL so from the SQL and only these two classes are there you inherit those classes why because uh I know that there are Prevail Transformations but I need to build a custom one how to build a custom one you are in editing two classes and there are three function in it we all know that that's a default Constructor okay you can write down anything so that uh this is a transformation that has been called or in the in the cases where you need to actually modify the feature or you want to declare certain arguments you can pass all of that knowledge in your init method fit fit is a method that you have to uh actually inherit it and just return self that the returning the instance of that uh class only just because fate is a method that you have to write down without this your custom transformation won't work you have to do that but the beam function in this is transform now here the parameters is x y y because Y is the target variable right uh you you want to do a transformation on your all of the features set so here the transform functions you can exactly apply any kind of transformation now in this uh you have a you have a column x x will have all of your features right you want to perform whatever kind of transformation you have to you can write down the custom code so this is the functionality will where inside the transform method we can write down your custom interpretation okay I have to square the column 1 and basically uh in column 2 I have to do this and then I have to return so return your transformed uh data and as soon as you have your pipeline this transformation will automatically call these method you don't have to call it automatically they are called that that is the beauty of using an ml pipeline or a custom Transformer pipeline that these are automatically all but the standard way of doing this is this you have to implement it in a way that you have to inherit base estimated and transform a mix in and then do that so that is the whole Theory I have written I'll uh I know that we are also be running out of time so I will try to go on to the code section too so that you guys have exactly the knowledge how to do that that whatever I have talked about here right so just a minute before I go yes so uh try to enlarge for you yes so see uh first is importing all of the libraries so if you try to have a look at this code guys uh I've used certain packages but what what the kind of packages I will try to highlight here is SK learn write SK learn psychic learn whatever you can say uh these are the kind of packages compose pipeline impute pre-processing so SQL and that's why I say it's a very interest library in Python so whosoever is working in machine learning domain or data science domain you have to be aware about what kind of functionalities are there and the main thing is how to use that because people know that this is available but how to use that that is certainly a missing part that is a majority of the people and you need to have that so sqln provides all of that now this is also uh highlighting the custom ml pipeline classes that I try to say sklearn.base uh these are the Transformations that are uh sorry the classes that are available so as to build up custom Transformer right now just I have created default data sets uh default data set the this is exactly I'm preparing a data set which has few of the NN values so that we can understand how transformation happens right so price column a column B column this is easy I am trying to build a custom data set so this is your data right it has price column it has a it has B now uh my price column also has some Nan values that means non-tributed values or null values a also has B also now uh just try to have a look at this right there is a column Transformer I have built on an input price column and an imputer is mean so I have said okay on my price column now see only price column I am imputing rather other columns saying that okay just pass through that A and B will not be changed only price column will be button now these Nan values from for which value they will be muted is the mean of this price column I've put down the strategy as mean there are other arguments to mode median anything you can put down so this is a builder and see the column transform is as as soon as I try to build that fit underscore Transformer I'm trying why I'm trying this is I'm trying I want you to actually visualize and see what happens as soon as you do that and the remainder is passed through see what happens it gives your data set a list of lists with First Column as imputed right because my first value is N A and C this is the mean and this mean is repeated wherever there was null values right so that's why I say it's not the best kind of strategy but here we are not discussing upon what will be the best imputing strategy here we are trying to understand how to do this right so see this values and see the second column values n a n and in the third column also you will find any n right so A and B are unchanged you have applied a column Transformer But A and B are unchanged and only the change column is price just because you gave that you can pass down a also a column will also be imputed you could pass on price A and B all of the columns will be muted so but if you want to apply okay I want to apply a computer copy the strategy B in second so what you can do is just try to uh just try to this copy this yeah just try to copy this and build down a strategy as median for a column now median value will be imputed in the uh a column values wherever there was null values right this is the first kind of column transfer now uh this is the whole data set I have print now let's try to understand the variation what was I was talking about a column I want a median now on price okay I was good that median value was there and I'm happy manual values are removed but by all my other column I want to build the median so do this and just pass on the column here now here one additional thing that has been done here is remainder you understood that here uh we have passed down it has passed through pass through means passing all of the columns as it is here what I have done is in the remainder also I've applied a computer I have said that let's try to build a constant value with value minus 1 right now as soon as apply the fit underscore transform see this first value median uh mean second null values are being reported by median and third whatever null values were there that are replaced by minus one so as easy as that right if you try to do that you try to impute a call log differently B and then combine that but in our column Transformer that is all in automatically where other workflow steps are automatically done by that then by the library so that that's how easy is uh for you to build a column Transformer now we'll try to understand column transfer with the python because as of now you understood what is column Transformer now I have tried to load a data set that is bind data set now that that's all of the different properties of a bind and now this is the pipeline see it appears very big to you but I'll try to explain you in steps no see understand uh this is the kind of series of steps versus the feature engineering teacher engineering all of the pre-processing steps right first is column Transformer here on uh certainly you have tried to pick I've tried to build names in the form of column so on the Alcohol column you are trying to apply a drop on second balance uh the somalic acid you are trying to apply a Minimax scalar and you're passing down the column okay because this is a terminology this doesn't accept a column name this we have given so that you can understand or this is a basically column name this is also a column name and then on this uh on this particular uh value of the column attribute here you have applied a pipeline and in the pipeline there is a series of steps that are given log transformation in Cabin so even if you apply a column transformation inside a column Transformer you can easily have a pipeline because you have a use case where in particular column you have to apply a series of steps whenever you talk about series of steps there is a python in place that's one of magnesium column I want to apply two steps that's why log transformation and cable and this was talking