Tutorial 5-End To End ML Project-Data Transformation Implementation Using Pipelines

Key Takeaways

hello guys my name is krishnaik and welcome to my YouTube channel so guys we are going to continue the seven day series for the end-to-end machine learning projects with cicd pipelines and deployment obviously we'll be using mlops as we go ahead but here uh in this particular video like we will be talking about data transformation already in our previous video we had implemented data ingestion again what was the main purpose of the data ingestion was that that we probably read our data set from kind some kind of source and then we did our train test split so here was the entire code we did our Trend test split and we saved this in inside one artifact folder right with respect to all the data set so this was with respect to data ingestion now if I probably go and see my train dot csvn touch.csv I have all these particular data points but one thing that you can probably observe over here is that right I have so many different kind of features like categorical features numerical features and all so with respect to categorical numerical features what kind of data transformation you need to to do and that is what we are going to discuss in this specific video right so here I have my data transformation now I am going to write my entire code over here the main purpose of the data transformation is basically to do feature engineering Plus data cleaning if I really want to change some of my data set convert my categorical features into a numerical features I can definitely do it from here now let's go ahead and let's write this code again guys if you have not seen this particular playlist the entire playlist link will be given in the description all I want from you all side is that now less number of people are basically seeing it so please make sure that you watch this entire series it is quite important um you know you will learn any kind of projects once we do this implementation it will become very easy for you okay so let me go ahead and first of all I will go and import sys so first I will be requiring this and then from data class as I am going to import data class the reason why I'm doing this I'll let you know okay along with this I will also be requiring numpy as NP so see I'm focusing on basically doing data transformation that basically means any data set that I have I will probably be using all the transformation technique if there is a categorical feature if there is numerical feature if I want to handle missing values everything will be handled over here so here I have import numpy as NP and then we go ahead with the next step that I'm going to import pandas SPD so pandas as PD um PD okay okay so this is the problem you know sometime when you start using too much of help from the vs code itself right then along with this I'm going to import from sklearn dot compose I'm going to import column Transformer now see why I'm importing this because if you remember in my notebook file for the model training purpose what all things we specifically did right if we go over here we used color Transformer you used one hot encoder we use standard scalar so all these things we are also going to use again in this data transformation technique one by one so I'm going to import column Transformer this column Transformer is basically used to create the pipeline right so first of all I may be doing one hot encoding I may be doing standard scaling so if I really want to create in the form of pipeline I can basically use this column Transformer Library okay and then let's say if I have some missing values so for that I will be using sklearn.imputer okay so here you'll be able to see dot impute uh sorry dot impute I will be importing something called a simple imputer and again different different amputation techniques you can basically do again the main thing is that I am just trying to show you some ways all the other ways you can definitely explore in that okay then uh what I will do I will also make sure that I implement the pipeline so for that I will be importing the pipeline also okay so importing the pipeline and finally from sklearn dot preprocessing since these two libraries will be super super important okay which libraries that is import one hot encoder and along with this we'll also be using standard scalar okay standard scalar now just let me know why one hot encoder and standard scalar is used try to write in the description of this particular video now these are some of the libraries that I will be using in this specific file now coming to the next one the first thing that I will be requiring is exception right so from exception because I need to do the exception handling over here in data transformation also right so from Source from exception I'm going to import my custom exception along with this I'm also going to import my loggers source.logger okay and I'm going to import my logging over here and then um yeah I think this two will be fine if there is anything most to be imported I will probably go ahead and write the code later on okay now as usual uh in the data ingestion right we had also created a data ignition config now this data injection config uh is just like providing all the input things that is required for this data ingestion component okay now similarly for data transformation also I will try to create that so over here I will write class data transformation config okay now this config what it will give me it will give me any path that I will probably be requiring any inputs I may probably require for my data and transformation ah component itself right so here what I'm going to do I will create my preprocessor I will talk about this why I am creating this so this will be my preprocessor object file path okay so let's say that I want to probably create any any models right and I want to probably save that into a pickle file so for that I require any one kind of path right so I will write OS dot path okay I did not import OS so let me go ahead and import OS okay import OS I will be requiring this okay perfect now here I will write os.path dot join and again I will try to get my artifact folder okay and let's say if I'm creating any pipeline pickle file I will name that as preprocessor reprocessor dot pickle file right now see I have just created preprocessor pickle file you can also create model file but this is just an example to show you in data transformation config what exactly you want okay you basically just want to give the input to the data transformation uh component okay and now here I will again use data class I hope everybody knows the feature of data class please do let me know in the comment section why data class is basically required okay already I told in my previous video but just this is just to check your knowledge because in an interview they may ask you now let me go ahead and write my data transformation data transformation okay now with respect to this data transformation class now this is the config or this is the input that I will probably be giving over here now first obviously as usual I will be writing def on underscore underscore init uh underscore underscore and this will be with self and here I'll be initializing self dot okay and I will write data transformation underscore config all right this will be initialized to what this will be initialized to this particular class over here right so in short what will happen inside is data transformation config I will be having this particular variable okay so that is the first parameter I will be probably requiring away in the data transformation config okay now what we will do is that we'll write a new function inside this particular class which is basically called as gate the Transformer get to a data Transformer object okay so get data Transformer underscore object okay and here I'm going to use the self okay so these are there uh over here what I will be doing is that the reason I'm creating this function okay this is just to create all my pickle files uh which will basically be responsible in doing converting your categorical features into numerical or sorry categorical into numerical if you want to probably perform standard scalar and all and all okay so here I will write try and this will basically be inside this try I will okay so here I will go ahead and write try and this will basically be pass and again X accept and here also what we'll do we'll just write it as pass so I'm just going to write the code uh with respect to a try and catch block over here now as usual if I probably try to open thetrain.csv you will be able to see what are the numerical features and categorical features that I have right and obviously that I specifically performed in the Eda also and I got the information about that right we understood that what all features were there again I'm again telling you guys if you did not see the previous part you really need to see it otherwise you will not be able to understand okay so over here first of all I will take all my numerical features I have already made the distinguation between the numerical features so here will be my numerical features over here okay so this will basically be my numerical feature in the numerical feature I have write score and reading score and probably in the categorical features I have all the remaining features so here let me go and use this categorical features over here so this is nothing but gender race certificity parental level of Education lunch and test preparation course now what we will do is that we will do two important things okay one is first of all create a pipeline with respect to the pipeline we also saw guys there are some missing values okay so we need to handle those missing values so for that if I probably try to create my numerical pipeline okay and this numerical pipeline I will initialize to pipeline okay now see this steps is super super important okay and this is the most important thing with respect to this so here I will be writing two steps okay the first step over here will be my imputer imputer basically means this will be responsible in handling my missing values so imputer and here I will be initializing my sample imputer and the strategy strategy okay will be used as median because there are some outliers which we also saw in the Eda right so if in the EDF we also saw in some of the numerical features there were definitely lot of outlier so we are basically using median okay so this is done with respect to the handling the missing values now after this what we'll do we'll also try to handle standard scalar so for this I will create another variable like scalar okay so just a second so this will basically be scalar and inside the scalar I will just try to initialize standard scalar now what exactly is happening over here we have created a pipeline which is doing two important things one is handling the missing values with this particular strategy like median and the second one is basically doing the standard scaling okay and this pipeline needs to run on the training data set okay like fit underscore transform on the training data set and we just do transform on the test data set now similarly I will go and create my categorical pipeline right so this will be my categorical pipeline and let's say if I get a numerical feature where sorry in categorical pipeline obviously what I will get I will basically get all my categorical variables so here I will just Define it so let's go ahead and probably Define our categorical pipeline now inside this also we need to handle multiple steps like let's say if they are missing values in categorical features then how do we handle that and for handling the category feature converting into a numerical value how should we basically handle that you know all those things will probably try to do so inside this what we are going to do again uh this was a pipeline so let's see why this pipeline is not coming Pi per line okay okay no worries so here it is okay this was the function over here okay now fine with respect to the pipeline here I'm going to basically create my steps and again my steps will basically be having multiple steps all together so here I will be writing steps okay the first step again to handle the numerical feature sorry to handle the missing values so here I will be creating imputer and usually what we do we basically use a simple imputer and the strategy that we can probably use over here strategy that we can start to start strategy please make sure that you write the spelling well otherwise strategy okay strategy now this looks fine okay strategy strategy it will be nothing but it will be most underscore frequent okay so that basically means I am just going to replace all the missing values with the help of mode okay now this was with respect to the steps the first step I am going to probably apply this in my pipeline the second thing obviously I'm going to use my one hot encoder and again here you can also use Target guided encoder or whatever encoder you do because in the Eda we have seen that there are very less number of categories in every categorical variable so here we are just going to use one hot encoder and we are going to initialize it okay and then coming to the third one if I probably talk about the third one over here I also have to do the standard scaling okay and I think if you don't do also it is fine but let's go ahead and do the standard scaling so that I show you many many things okay over here so done uh this is all the techniques that we are going to perform in the categorical uh pipeline okay the first thing first uh what we are going to do over here is that we are going to apply the imputer pipeline then one hot encoding then scalar okay so here missing values is getting handled for the categorical features then one hot encoding is basically handled for the one hot encoder and then you are basically doing it at scaling now I will also go ahead and write my login okay so login dot info and here I can go and basically write categorical categorical columns encoding okay encoding completed right so here I can write this logging info on simulator I can copy this and I can say login.info and here also I will talk about the numerical columns numerical columns and here I will not say encoding but at least standard scaling I will basically write okay completed now here you can basically see all both of these things we have basically done now all we need to do is that combine this numerical Pipeline with this categorical pipeline together so for that only we will be using column Transformer so for this we'll be using preprocessor preprocessor will be my variable and here I am going to use column Transformer and this column Transformer will be having the combination of two things okay one is the numerical pipeline okay numerical pipeline and the second one is something called as numerical pipeline comma numerical underscore columns okay so here see what we are doing this is what is the numerical pipeline that we are giving over here okay and for what variables we are specifically doing this we are doing it for the numerical columns okay so inside this we first of all give our pipeline name then what pipeline it is and then our column similarly over here the next step will basically be my categorical five line right so this will be like cat underscore pipelines this will basically be my name and here I'm going to basically give my categorical Pipeline and then this will basically be my categorical columns so a comma categorical columns so yes both of them have been given inside this column Transformer again understand what all things we read we created a numerical pipeline which is doing two tasks right that is computer and scalar okay that is handling the missing values and Performing standard scalar then we have categorical pipeline that is performing inputer one hot encoding and standard scaling and then we did the login.info then we finally created I'll not write this over here right now let me just go and give simple columns right like this I will just go and give this logging statement right just to display our categorical and numerical feature and then here I am creating a column Transformer which is the combination of numerical Pipeline and categorical pipeline okay so all these things is done and finally what we do after this we come outside and we return we return the preprocessor okay now once we return the preprocessor that basically means we are returning this entire thing with respect to whatever things we have done over here now in the accept block what I can do again it is very much simple I will write exception as e and I will try to raise my custom exception right and this we have done it multiple times so I will go ahead and write this custom exception with e-commerce done perfect so we have actually created the Gated get data Transformer object the main aim of this particular definition of this particular function is to I will also write the definition this is this function is responsible responsible for data transformation based on the different types of data okay so perfect uh I've done this and this is very much good uh we are we are we are going in a very good way we are able to complete all these things and all okay so uh finally uh you can see get data Transformer object now let's start our data transformation techniques okay so this is where I will start my function so this will be my another function which is like definition initiate I will just write it as initiate data transformation that basically means I'm starting my data transformation inside this function okay and here again I'll be using self and then I will give my train underscore path okay train underscore path and this will basically be my test underscore path so training part and test part and you know from where you are getting this you are getting this from the data ingestion right and done over here again I will be writing try uh pass then I will go ahead and write accept pass okay as usual okay now fun this is good now let's go and read our training data set so I'll write train underscore DF is nothing but uh uh since we have already imported pandas so I will write PD dot read underscore CSV with my training path okay then I can go ahead and write my test underscore DF which is basically PD dot read underscore CSV okay so this will basically be my test path now here I can basically write my logging logging dot info so that you'll be able to log all the details the the train and test read train and test data I will just write like retrain and touch data completed done so uh initially we'll go ahead and read this now the next step what we are going to basically do over here is that after this I will write our number logging logging dot info and I'll write that obtaining obtaining pre processing object okay so done this is done with respect to this I will be reading all my pre-processor object so for that I will just write preprocessor now what is this pre-processor object whatever object I created on top right this one this pre-processor I will try to call this so this will be nothing but here I will write self Dot get Transformer object right so here I will be getting this particular object over here perfect now I will go and create my target column just to make sure that um my target column name so this will be my target column name is nothing but math underscore score so if you see the data set over here so this is my training data set there is something called as matte underscore score so for this data set will be my target column okay numerical column you basically know what is my numerical column uh this is nothing but uh writing underscore and reading underscore so I will copy this same thing for my numerical column so this will be my target this will be my numerical columns and now what I will be doing is that uh you can basically check out with respect to continuous values numerical values again categorical features also you can make sure that you can basically write it in a way okay now this is fine perfect till here now what I am going to do I will say input input underscore feature underscore train underscore DF here I will write train underscore DF dot drop I'm just going to drop the columns which column that is that is nothing but my Target column name okay from the input feature and I will just write access is equal to 1. so guys one final thing that I am going to do over here over here you could see that we have created this pre-processing object right and you know that this can be converted into a pickle file it needs to be converted into a pickle file so for this we have not converted anywhere but we have taken the path right the path is basically given somewhere over here in the data transformation config right like pre-processor pickle so what we will do will go ahead and save this we'll save this pickle file in the same location so over here uh to save it I will just do it after this so here I will just create a function which is called as save underscore object okay and I will just give this file two parameters one is file object or file path okay this two variables I will give I will write it as self Dot or data transformation config so this will be the same thing data transformation config or this entire thing I will be giving this okay self dot data transformationconfig dot preprocessing object file this will be my first parameter with respect to the file path okay in short I am having that entire file path like artificial artifacts pre-processing pickle right so this is what is the file path over here right something like this so uh this will be my first path and this is first parameter and the second parameter will basically be my object okay so this will be pre-processing underscore obj okay so this is my object that I really want to save it now because this is the what I has a model with respect to all the transformation now where do we write this function save object now to make your work very much easier uh I told you right we are going to basically use utils right so utils is one place inside the source right so if you probably just go and minimize this Source inside this Source you have something called as utils and I told you that utils will have all the common things that we are going to probably trying to import or use right the common functionalities which the entire project can use so here also I will write import OS okay and here I will go ahead and write input CIS for my exception handling um import numpy as NP import Panda other than PD Okay so NP import pandas as PD okay oops I hate this now I'm going to stop this now uh import Panda's SPD okay and if I want to probably do train test split I can also write it over here see the train test played initial in data ingestion was written over here instead I could add written over here and called that particular function over here itself right so it is up to you okay uh if you want to make your Cloud very much cleaner then we will do this now what I will do I will just create this function called as ah save path okay so ah OK I will also make sure that from SRC dot exception okay import import custom exception okay now perfect uh now I have defined this save object function over here this is taking the file path and obj it will basically take the file path or it will make a directory like this if it already exists it is going to do it then we are going to open the file path in the right byte mode and we are going to do the Del dot dump okay so what is this Del this is also another Library which will actually help us to create the pickle file okay right now Delhi is not there so go ahead and import over here so like Del okay and I will just go into the terminal okay and uh probably install paper install minus r minus r requirement.txt just to make sure that that drill is getting imported or installed so yes successfully installed Dil this is good now if I go and see import Dil this will be available so this will be deal dot jump uh dump sorry dump when we dump this this will be saved in the specific file path okay so in short with the help of this code what we are doing is that in the data transformation we are saving this pickle name in the hard disk okay now in order to use this all I will be doing again where is I'll just import from source or I can basically import something like this let me just write I think it is inside the source itself right so okay no worries from Source Dot utils import from Source dot utils input save underscore object perfect now this save up another object we can call it over here and this is just used for saving the pickle file okay good we have done it almost data transformation and data so in data transformation we are handling the categorical values numerical values we are creating this entire pickle file doing everything okay now it's time we go into the data ingestion path I had already written this code okay now what I will do I will also import from here from a scalar sorry from Source I'm going to from Source Dot data transformation I think it is Source from Source Dot components Dot data transformation I'm going to import data transformation and along with this I will also import data transformation config okay I'll input data transformation config just to check everything is working fine or not okay and finally over here uh when I do object dot initiate data initiation then I will be returning two values one is train data and touch data so here let me go ahead and write train underscore data comma test underscore data and this will what I will be basically getting now after this all I will be doing is that I can create a again I can probably see in the data transformation right I can probably just call call this data transformation and with respect to data transformation I can probably call this all the functions that I have okay so here I will again go into my data ingestion and let me go ahead and write data underscore transformation this will be nothing but data transformation okay once I initialize this you'll be able to see that it will call this self dot transformation function then what I'll do I will just call this initiate data transformation so here if I write data underscore transformation dot initiate data transformation done most of the execution will happen inside is data transformation what we need to give our train data an odd test data okay and that is what we need to give I'm just trying to see whether it will be running or not I'll get some errors let's see how what all error is going to be getting fine we are good now let's execute this code okay so what is happening we have combined data ingestion and then after that we have combined data transformation okay oh hope so it works I have continuously written the code now now usually see when we are writing in the real world scenario right we will be facing some kind of issues so here I will go ahead and write python Source data injection.py um okay I have to basically go ahead and write python Source components slash data simulation dot py so the execution has started let's see whether this will get executed perfectly or not at the end of the day I should be able to see some pickle file over here so now if I go and see okay pickle file is not get created let me see why it has not got created so refresh great great execution has happened successfully now let's see Google something is there let's see the logs injection of the data is completed okay fine after the injection uh of the data is completed let's see in the data transformation so guys what I see from the logs I'm not able to see the error the reason is that because I did not write this accept block so what I will do over here I will say accept exception as accept ion as e and here I will just raise my custom exception now I think we will at least be able to see the error what error we are able to get okay and here I am going to give e comma 6. this is fine okay now let me do one thing let me again execute this now I think you'll be able to see the error see the error is coming as occurred in this this is 86 init got an unexpected argument strategy okay fine I understood what is the problem so in the data transformation will be seeing that this spelling is wrong I guess strategy okay it should be something like strategy uh which line that was so this line was 42. line number 42 okay so here also the spelling is correct simple imputer and here I just have to use strategy okay now I think it will work so it is necessary that you have this custom exception raised right so now if I go ahead and read it everything will work fine but still I'm getting an error line number 101 Center cannot cannot Center sparse Matrix with mean is equal to false instead doc string instead C doc string okay this is for 101 line let's see where it is oh 101 nine okay no problem I will go and see 101 line see this kind of Errors you'll be getting over here so guys uh I got to know the error what exactly it was here I had written train underscore DF this will basically be test underscore DF okay so just have a look on to those kind of Errors now I think it will work fine it looks good I have all my saved Crossing objects this is my artifact and the preprocessor dot pickle file is created finally so yes it is not that simple always you will be getting the error the but the main thing is that how you are able to fix the error that that skill set you should definitely try to find out okay so great uh we have done all this modification we have saved all the file now what I'm going to do I'm going to commit this code for you all so that you can use get add dot get commit minus M okay um here I will just say data transformation is added data transformation done saved it and then I can say get push minus U origin to Main so once I do this all my push will happen and finally I'll be able to see in the GitHub link the GitHub link will be in given in the description of this particular video so guys this was it for my side we did each and everything in an amazing way we found out all the data transformation technique everything as such what all things we specifically did we transformed our data set we handle the categorical values we handle the missing values and here I have just shown you by writing line by line code so that you will be able to understand so yes this was it for my side I will see you all in the next video thank you