End To End MLOPS Data Science Project Implementation With Deployment

Key Takeaways

hello all my name is krishnaik and welcome to my YouTube channel so guys yet another Amazing Project for all of you and again this will be an end-to-end project with deployment in our previous video I had uploaded one video regarding ml flow and dagsub right and we saw that how we can perform all our experiments model training with respect to ml flow right and dags up was a kind of a remote repository which you can also probably see experiment and collaborate in a data science project with multiple team members now after that many people were requesting Chris try to create an end-to-end project wherein we can probably use this ml flow along with data ingestion data transformation model training model deployment and probably try to do it in some AWS ec2 instance because in companies there many of the companies use awcc to instance right so considering this this particular project will be an end-to-end machine learning or data science project where I will be taking a specific data set and then we will be implementing each and every steps with respect to data ingestion data validation data transformation model deploy model training and model deployment and evaluation everything we will be implementing along with this will be integrating ml flow also now what will happen if once we start using ml flow in this with the help of ml flow we will be able to integrate and do most of the tasks with respect to experimentation model tracking everything as such right and then as you know that ml flow is completely open source so you can probably integrate with any kind of clouds right let it be a AWS easy to instance or Azure or anything Sr or Google Cloud also and you can also do this with any kind of libraries let it be said it learn tensorflow pytorch right so what we will do with respect to our experiment we will try to work in such a way that will try to integrate ml flow and then we will Implement all this life cycle and then probably we'll do the deployment with GitHub action in the AWS ec2 instance so please make sure that guys while practicing practicing this I will be giving the code entirely in the description of this particular video go ahead and check it out and as you practice go ahead and implement it you know and if you have any questions let me know in the comment section right now before I go ahead I really want to make a quick announcement if you probably want to learn data science from me you know probably implement this kind of projects in a much more detailed manner which much more implementation which much more explanation then you can probably check out our full stack data science Pro batch which is starting from August 12th so here you'll be able to see from August 12 2023 it will be completely live the classes are between 9 am to 12 p.m IST and again the doubt clearing session will be after between 12 to 12 p.m IST every Saturday and Sunday right here in this particular program it will be somewhere around 12 months duration all the classes will be live the instructor will be sudhanshu and Krish which are the main mentors along with this we also have some more mentors that is Sorento Paul and we also have Sunny Savita right so all these are the mentors one amazing thing about this particular course is that we have also included you know some amazing thing with respect to open AI so prompt engineer and LM we used to not have in our previous batches but now we have implemented or included everything right from prompt engineering to large language model to everything right so go ahead and check out all the information will be given in the description and in the pin comment [Music] so guys now let's go ahead and implement this entire project step by step okay so you just have to implement along with this and if you probably face any kind of concerns or issues please let me know in the comment section so let's go ahead and implement the project uh so guys now let's start with our project implementation so before that what I will do I will create one GitHub repositories okay of our project and uh we'll be creating the project template okay then we'll start without project creation so first of all I will uh open my GitHub account so here I will click on new Repository okay so let's give the repository name as uh end to end machine learning project okay uh with ML flow now uh I'll make it as public then I will add readme file okay then get get ignored python and let's take a license okay so let's take MIT one and I will create the Repository okay all right now I need to clone this repository I will copy uh select https and copy the link address okay now I'll open my local folder here I will open my terminal now just write git column and paste the link okay now this was end-to-end uh machine learning project okay so I'll just write CD end to end machine learning project with ML flow okay this is the project now let me just open it manually this one okay now I will open my vs code in this folder I'll just write code space Dot all right so first thing what I will do I will create the project template okay so I'll create one file here called template dot pi yeah so why I am creating this template dot Pi because let's say uh here we are creating end-to-end project okay and here we need lots of files and folder okay to manage my code so instead of just creating those folders and file manually if I create one python script okay for that and I'll just write one logic here okay it would be just one time effort so after that every time whenever I will execute this template dot Pi it will automatically generate this folded structure for me okay so you're just putting just one time effort and you're trying uh and you're creating these kinds of folder structure okay but let's say you had to create this manually so you need to click here again you need to click here okay they need to manually create all the folder structure so it so it would be a little bit time taken for you okay so instead of that actually uh try to create one python script okay and try to uh create this kind of folder structure okay this is the best way usually I follow so I usually follow one template structure okay in my every project so uh let me uh show you like what other things actually I usually create in this template dot Pi so first of all I will import some of the libraries here okay so first of all I will import operating system library then path leap okay from partly where I I also need path why I'm using path I will tell you okay then I am also importing login so first of all I will initialize One login string okay because I want to see the log in my terminal okay that's why this logging string is needed so basically first of all it will log what kinds of log label it is so it is information level log okay with respect to that it will save the time stamp okay the time actually you executed your code and the error message okay what is the message actually or let's say execution message it will save in the terminal then after that I will give my project name okay so project name should be ml project here you can give any any project name as per your choice but here I have given ml project okay so basically you will create one folder called SRC inside SRC it will create this ml project inside that I will be creating all of my component okay so this should be pretty much Clear whenever I will create this follow instruction okay then I will explain okay what I have done so here I have taken one list okay let me show you so here I have taken one list and I am named it as list of files okay so the first record what I am doing I'm creating one folder called dot GitHub inside that I'm creating another folder called workflows okay then dot git keep so that's so what it will do it will create one folder here dot GitHub inside that again it will create one folder workflows okay again it will create one file called dot gatekeep okay that's how you can see I'm arranging the folder okay folder name and fold like file name here so first of all it will create SRC folder inside that it will create this project name folder okay ml project inside that it will again create one components folder and inside that it will create this Constructor file because this is going to be my local package okay so every time I need this Constructor file to to make this folder as my local package because I want to import something from this folder okay that's why you need to keep this Constructor file ready okay so that's how you can see I'm creating components utils okay and then configuration okay then pipeline entity okay constant these are the thing and what is pipeline constant entity I will discuss each and everything whenever I will start the coding okay coding let's say uh implementation at the time I will tell you like what is this okay why I am using this thing okay as of now just consider these are the folded structure actually you need and with respect to that these are the files actually you need okay then I need config ml DBC okay so DBC I won't be using in my project so let me just remove it okay so DVC I don't need I I have already removed the DVC here so I'm again creating one file called params.aml schema.aml okay main.pi F dot Pi Docker file because I I told you I will be integrating Docker with that requirement.txt setup and I I'm also creating one folder called research instead that I'll be getting ipu NB file because before implementing our actual components we'll be doing the experiment okay in the notebook then I'm also creating template Okay index.html because I'm going to use flask okay to create my web application that's it okay now let me show you the logic I have written to create the folders and files okay so here I have defined two for Loops here so I'm first of all looping through this list okay first of all I am converting everything to the path okay as you can see I'm giving the path inside that I am passing these are the record okay so basically what you can see here here I am using Windows operating system but here I have provided forward slash okay but by default in Windows actually it always consider backward slash okay instead of forward slash okay so to handle this kind of issue okay this kind of path issue if you pass this path here so it will automatically convert to Windows path okay and it will give it to here okay after that what I'm doing I'm separating out these folders okay separating out these folders and these files because as you can see the first one is our folder okay and the last one is our file okay so using uh voice dot part dot speed okay I'm splitting that and I'm storing my folder in a separate variable and file in a separate variable okay so this here I'm first of all creating the folders okay as you can see if I am checking a folder directory is not empty just create the folder structure okay and log the information and here I'm creating the file first of all I'm checking this file is exist or not if it is not existent this file size is zero okay then try to create the file okay and try to log the information and if everything is okay then just try to give file is already exist okay this is the simple python code I have written okay now let me show you let me just execute and show you like how it is creating the uh folder structure for you okay so by default as you can see I don't have any file here okay I don't have any file here files on folder here so let me clear my terminal now if I execute my template dot pi first of all let me activate my environment on the I think this was the activated uh I think this was the my environment okay yes now I will execute my template.pi now guys can you see left hand side it has created all the file structure for me okay it's like very easy right we have just done one time effort any time you can execute this template dot Pi you can create folder structure for you and even we see it is also printing the log in the terminal see it's created the folder up folder okay empty folder empty folder okay now now let's say I want to create another another file okay so what I will do I will just come here and I'll give the file name here only okay let's give test.pi okay test dot Pi I want to create now just save it again open up your terminal and just run your template dot Pi C test dot Pi has been automatically created that's how using this template.pi you can anytime I mean like arrange these kinds of order structure now let me show you what other things it has created first of all it has created this dot GitHub as you can see at dot GitHub we need because for the cicd deployment we'll be doing okay using GitHub action that's why this thing is neat see workflows and dot gatek dot git keep I have given because whenever I will commit the code so this folder should be also committed that's why you should have uh some file present inside any kinds of folder otherwise GitHub won't take empty file that's why I've kept it in this dot gate keep okay return I will remove it now Pi test would be automatically generated for me because I have the extension okay so this file you won't be getting so you completely fine just you can ignore it okay now config as you can see I have I'm creating config here config.aml so config config.aml so here I will be keeping all of my configuration of my project now research inside research I have trials to Type U NP as you can see now inside SRC I have created C it's created the project name ml project inside that it has created the components okay again all the Constructor file as you can see config constraint entity pipeline okay and utils inside utils I have created common.5 because here I will be keeping all of my utility okay template index.html okay and these are the file as as you can see here as you can see params.aml schema.aml requirement.txt setup.pi everything it has created successfully okay so that means we are able to create the folder structure now what I will do um I will do the project setup okay and requirement installation for this project like what are the requirements actually you need to run this entire project okay so for this actually I need to install some of the package okay but before that let me comment the changes in my GitHub so what I will do I will open my terminal I will clear it uh I'll just write gate add space Dot and gate commit I've been in uh folder structure edit okay now I'll just write git push origin Main now it should push my code let me see let me go back to my GitHub and refresh the page here see yes it's done okay now let's do the my requirement installation so I'll open my base code again so I'll open this requirement.txt file okay and here I need some of the package okay so let me just copy paste the package here yeah so these are the package actually you need okay you need pandas ml flow okay see I'm using the same ml flow version notebook numpy scale and okay matplotlib and then you can also remove these are the theme but I kept it here okay uh just for a reference okay uh let's say if you're doing any kinds of visualization in your project you can also use the smart prop and all okay then python config box I will tell you like why I'm using python config box okay this python box why I'm using and uhml tqdm okay and ends here okay I'll I'll discuss this package okay why I'm using in my project because this this is like new package I have integrated with my project and these two package is like pretty good okay so you can do lots of things in these two package and it will make your code like very professional okay that's why I'm using this thing and then uh I'm also importing flask and Flash scores because I already told you I because at the end I will also try to create a web application okay so there actually I'll be using class quality okay now requirements is done now I also need to install this these are the folder as my local package okay so for this I will add this line called hype any space dot okay so it will look for the setup.pi okay inside setup.pi you need to mention um about your local package installation okay so let me just show you the code yeah so this code actually you need to mention inside setup.pi basically first of all it will open up the redmi file okay so these are the basics code okay you can write it otherwise you can also ignore it completely fine but I am giving just to make your code professional okay as you can see I'm mentioning the version I'm giving the name of the project okay it should be only ml flow so I'll remove the DVC okay now author name you can give here okay this is my source repository name as you can see Source repository name uh ml ml project okay author mail address you just replace with your mail address okay and setup tool.setup you need to give provide these are the information here okay so basically what it it will happen it will look for everywhere this Constructor dot uh okay this Constructor file and it will install this folder as my local package okay so that is the use of this setup.pi okay because let's say I want to import something from components okay let's say I want to import from src.mlproject dot components dot uh import dataization okay if I want to import like that so first of all I need to install this like folder as my local package okay and to install it you need the setup.i okay I think this should be clear now okay now let me install it so see to install setup.pi you need to mention this line okay it will pick up the setup.pi automatically now I'll open my of terminal I'll clear it okay then I'll just write peep install ipin add requirement.txt okay now let's install so guys it may take some time let's wait okay I will come back when it is done so guys as you can see my installation has been completed okay and there is no error that means you have installed successfully okay now what I will do uh I will quickly commit the changes so I'll open my terminal again and clear it I'll just write Gita I think I already have the command V that okay then git commit I'll change the commit message here requirements edit okay then I'll just write git push original Main yeah it's done now if I go back refresh okay see requirements added and if you want to verify you can open any kinds of file here setup.c code is there okay now we have also uh completed our requirement installation of our project so the first thing what I will do first of all I will complete uh writing my login exception and utils module okay because as you can see if I open my SRC ml project here I have utils okay then I also need to create login and exception okay with it so how I can create login and exception see exception I won't be creating separately because I'll be using something called box exception okay from this Library called python box okay and uh and like you uh logging actually I'll be creating in a different folder okay let me show you like where I'm gonna create the login functionality okay so here if you see inside SRC okay inside SRC I have created one Constructor file okay so instead of creating one like separate login folder okay you can also write your login functionality here just to make your import easy okay now let me open this Constructor file here okay and I already prepared my custom login okay so let me show you see guys this is the custom login so this is nothing new first of all I'm importing operating systemcs and login I'm initializing my login stream so basically it will first of all save the ASCII time okay then it will save the log label okay like whether it's the information related log or it's a it's a bug level log okay it will save that information then module like which module actually you are running this code it will save the model name and the message actually you want to print okay so this is my simple login string then what I'm doing first of all I'm creating One log folder okay here inside that I'm creating one folder called Running log okay and after that I'm calling this Basics config method okay and I'm initializing everything my long listing and all then I'm calling this to Method called file Handler and stream Handler so file Handler what it will do it will create this log folder inside that it will save all the logging okay and the stream Handler it will print my log in my terminal okay I think you saw whenever I was executing the template.pi just printing the log as well okay in my terminal that's why using this stream Handler you can also print the log in your terminal okay then I'm initializing my login here okay ml project logger you can give any name here okay so now how it will work let me show you so for this actually what I will do um I will let's say open this um let's open this main dot Pi okay and here let's import my login okay so what I will do I'll just write from from uh ml so sorry from ml project ml project okay it's not giving suggestion because here I have any selected my environment I click here I think it was ml props right ml prods Let me refresh again ml process this is the environment okay now if I write from ml ml project okay as you can see this is the name of my folder ml project okay and as well I'll just write import login okay see I think one thing you have mentioned although I have SRC folder here okay I have also in search the folder here but I'm directly calling this ml project okay I'm directly calling this ml project okay why because I have initialized my login functionality in this Constructor okay that's why I don't need to call this SRC okay separately I can directly call my ml project okay I I can also call like that from SRC dot mlproject.login I can also import like that but if you want to ignore any kinds of folder okay you can mention this thing inside the Constructor file and you can also import it okay so this is another approach actually I usually follow in some of my project okay that's why I thought let me show you as well okay so instead of creating logging file function like logging functionality separately you just keep it inside Constructor file inside ml project okay and you can also import like that okay now we have imported the login now I'll just log one information here uh let's say I'll call login a little logger dot info okay and here I will give some message let's say welcome to our custom block ing okay now let me save it and let me execute now okay now let me show you like how it will say now I'll just write python main.pi see you guys first of all it will give you the login in your terminal see first of all it is saving the time stamp okay time is time you run your code with the date also okay then it is telling information related log now it is also giving the module name okay see I have run from main.pi it is giving the men okay now it is giving the message as well the message we have given called Welcome to our custom blog as you can see Welcome to our custom blog okay now left hand side you can see it has also created one further to the logs inside that it has created running logs okay now if I open this now you can see the same message okay that is the like use of my custom blogging okay so using this custom logging functionality uh you can debug your code very easily let's say you have hosted your uh project okay you have deployed your project in a cloud platform they actually won't be getting any terminal to monitor your code okay so at a time if you have this kind of blogging functionality added with your code you can download these are the logging file okay from your uh remote server and you can see like what happened in your code okay if it fails somewhere okay so you can see that and you can directly fix that code okay for you so that is why actually we add these kinds of custom blogging functionality okay in our code okay so this was like custom log now what I will do I will create my utility okay utils so here left hand side you can see I have one folder called utl okay now let me open the utils and here I have created one file called common dot Pi okay let me open up now first of all let me now first of all let me tell you like what is utils okay see utils is nothing but those functionality you'll be using frequently in your code okay so we call it your tools okay let's say you want to read yaml file okay in all of your components okay so instead of writing that radioml function everywhere what you can do is just keep it inside command dot Pi inside YouTube okay and whenever you need it just try to import from here okay so that means those functionality will be using frequently in your code okay instead of writing them every time in the component just keep it in the utils file okay and whenever you need it just call from here okay that's why you are making reusability of your code okay so whenever we try to follow modular coding passion so at a time we follow these are the structure only okay so that's why I will be keeping some of the utility related functions okay here of this project okay so I already prepared some of the data functions so let me show you what the function actually I'm you I'll be using here so let me just show it here okay see guys first of all I'm importing some of the libraries I'm importing box abstraction as I always tell you I'll be I'll be using something called box exception okay box value added okay every time I'm calling this box value added okay instead of writing your custom like exception okay you can either create your custom exception but uh this package is pretty good okay you cannot follow this approach also it's completely fine okay then yaml I'm importing then logger I am importing also okay the logger we have created then just on job leap okay then ensure annotation I will tell you like what is ensure nutritional box I am importing config box okay I'll tell you why I'm putting config box path okay I already told you why I'm using path okay whenever I was creating that template dot pi and typing okay so the first method I'm initializing here called redml because I want to read EMF lots of yaml file as you can see I have lots of yaml file in my code okay I want to read them okay so for that actually I need some of the functions okay to read my ml as you can see I'm creating one read ml file and this is the simple python code to how to load one reml file okay and I'm returning as config box okay the content I'm reading I'm returning is config box tell you like why I'm using config box okay then on top of that you can see I'm using ensure annotation okay again I will tell you like why I'm using this decorator here okay then again I am creating another method called credit directories because I also want to create some of the directories okay I'll be creating artifacts inside artifx I'll be creating data in this data validation model trainer okay so to create this I need this function okay again this is this should be frequent use function that's why instead of writing into my components I'm keeping inside my utility okay then save Json then load Json okay save bins then load bins okay then get size so these are the basic functionality actually you need actually in any kinds of end-to-end project okay that's why I'm keeping here okay although I will be using all the functionality here but I'm just keeping just for your reference okay so I think now it's clear like what is utility okay in my code now let me explain what is this config box and NC orientation okay so for this I will open this okay I'll open this one okay I'll explain this example here let me save it here okay first of all I will select the kernel python environment I created ml procs okay so this is the environment I'll be selecting okay all right so first of all Let Me Explain like what is config box okay why you need config box let's say I'll Define one day a dictionary here okay so D is equal to uh let's say I mentioned one key you can replace any key with that and with respect to that I have one value here okay then I will again take one E1 okay then let's say it has again one value okay the limit as well one all right so this is a simple dictionary I think you know in Python okay now let's say if I want to access this value so what I uh so how I'll access this value okay I think in dictionary you know we usually call like that okay d I'll provide this name okay name here okay if I pass key here so it will return me the value okay but can I read this value okay instead of giving the key like that can I give like that D dot key if I give like that will it work okay now let me execute see it will give me error dict object has an automated key okay it's selling like that okay although I have key but again it's not it is not recognizing because in dictionary actually we usually call like that okay but if I want to call like that so I can use something called config box okay so let me input config box here from box so there is a box package now I'll import my config box okay now what I will do I'll again Define one D2 and instead of writing this dictionary directly here I'll call my config box class first of all okay and inside that I will mention my dictionary okay the dictionary I created another copy I'll paste it here okay now if I print my D2 here now you'll see the object name object type should be config box type output okay now if I call D2 dot key right now see I am able to access this value okay so what is doing here actually so I am making this simple dictionary to this config box type output okay as you can see config box type object so that I can also access any kinds of value okay using the key name okay instead of giving like that because whenever you are let's say lots of configuration okay because why I am doing like that as you can see inside Commons I will be reading lots of yaml file inside yaml file I will be having lots of configuration okay so if I want to read those configuration instead of giving the name like that okay if I call like that so this approach should be very much easy to call any kinds of configuration I will be using in my code okay so that's why we'll be using this config box okay so everywhere as you can see whenever I was reading this ml file I am written as config box type output so that I can access those value using the key name only okay instead of giving inside the list okay that is why I'm using config box okay I think it's this should be clear now let me explain what is this ensure annotation okay hence your annotation I'm using on top of every function I have created okay so for this I'll Define one simple method here okay called get product so this method what it will do it will take two input X and Y and this should be integer type and it again after doing the product operation it will written integer type output okay so this is a simple product I think you know in Python okay now if you want to call it just call by the function name and here I am providing X is equal to 2 Y is equal to 4 okay now if I execute C to the return me the correct number okay to multiply by 4 it's 8 okay now if I do like that okay instead of giving the code like that okay instead of giving 4 as a integer if I give to string will it work okay just think about in your mind we need to work okay let me execute and show you see still it it is working although I mentioned this Y type should be in desired X type should be integer X type is completely fine for me I have given integer but Y type I am giving as a string okay but still that is able to work okay and you are getting some different output okay so let's say you have written a bigger code bigger project at the time you did these are these are these kinds of mistake okay these kinds of like functionality you have already written and you are getting unexpected output at a time you will go mad right what is happening in my code I'm not getting my proper output okay although I have mentioned my data type here okay to overcome this kind of issue I have something called ensure annotation okay now let me open uh first of all import The ncn annotation so from ensure you can use ensure annotation okay let me import now what I will do I'll copy the same function okay I'll copy the same function and on top of that I'll just mention this decorator and share annotation okay decorator just give ad direct and give give it okay here now if I execute it now again I will I'll call my function okay again I'll call my function I'll give integer integral because I have given integer integer here okay so it is working okay now I'll call the second one as well now let me see what happens so if I give second one now now see it's it is throwing error it's telling argument why type okay 2 does not match The annotation type because it is telling you have mentioned in these are here but you are trying to pass string okay that is why it's throwing error okay now see now see one thing you have learned you can fix this kind of small small box okay using these are the library you can see your annotation config boxes are the thing okay now see this will make your code like very professional so whenever you are writing any kinds of production level code so at a time you can use these at the library so I was exploring this at the library and I thought let's also integrate with my code okay so that I can show you how we can handle this kind of issue okay now here every time as you can see whenever I'm creating any functionality function or let's say method and calling this ensure annotation everywhere okay because everywhere I am mentioning my data type okay and if user is passing some different data type so it will be throwing error okay that is why this ensure iteration is needed okay now I think this thing is clear now let me save it yeah so we have successfully completed our exception logo and our utils okay now let me just quickly push the changes so what I will do see instead of pushing from your terminal if you're using vs code left hand side you can see something called Source control okay you can just click here and you can also commit from here just give the commit message so I'll give exception logger and utils edit okay now I'll just click on Commit so those who are doing for the first time it may ask for the authentication with your GitHub just try to approve everything and you can also able to do it okay now I'll just sync the changes sync the change means even you are pushing the changes okay but for me I've already configured everything that's why it's not giving anything okay now if I go back to my GitHub now if I refresh here now see it has competed okay so yeah guys uh I have done and see if it is not working for you you can directly do from the terminal itself okay as I already did okay previously you can also do it like that okay so yes guys we have completed our logging exception and utils okay so now what I will give you I will give you the project overflow actually you'll be following okay because as you can see here we have lots of files and folder okay now one question in your mind which file I need to test first okay which zip file I need to change last okay here so these are the workflows actually you have to for him to understand then you will be updating our project okay so first of all let me give you the workflows overview like what are the things actually we're gonna update okay which file I will be updating first which file I I'll be updating last okay let me give you the workflow of the project okay so guys uh now let's uh discuss like what should be the workflows of our entire project so let me write down all the workflows here see these are the workflows actually you will be following throughout this entire project okay so first of all I'll be updating my config.aml file okay as you can see inside config folder I have one file cell config.aml file so this file like actually I'll be updating first okay then I'll be updating my schema.aml as you can see schema.eml file will be updating what is schema schema is nothing but it's just a file actually here you mention all the columns actually you have in your data okay with respect to the data type okay so we call it schema uh and we will be also updating params.aml as you can see params.tml so inside parameter ml actually will be writing all of the parameters actually we'll be using in our project okay let's say uh I think I already showed you with the elastic net that actually I was maintaining two parameters called Alpha value and another ratio okay so instead of giving that thing in the runtime okay in the command line I'll what I will do I'll give it inside params.yaml okay I'll write uh all the parameters here so that user can change here okay and it will reflect in my record okay then I will also update my entity as you can see here I have folder called entity inside that I have config entity so I will be updating this entity then after that I'll be updating something called configuration manager in SRC folder config okay here I have another config folder inside I have configuration.pi Okay so this folder will be making the change okay after the entity then I'll update my components okay components means like data injection data validation data transformation these are the components I'll be operating then I'll be updating my pipeline as you can see I have affected one pipeline folder okay basically all the components I'll be integrating with the pipeline okay I'll be making my training pipeline separate and prediction pipeline separate okay then I will be updating my main dot pi as you can see main.pi will be updating then instead of dvcml I will also need to update my app.pi after Pi okay yeah sub dot pi as you can see I have also updated Pi will be writing my UI related functionality and I'll integrate this an Android Pi with my app.i so user will run this app dot pi and it will access all the code okay here so yes guys this is my enter workflows of this entire project okay so this file will be following and we'll be updating the entire project okay uh so guys now let's start with our first component okay which is nothing but data in addition uh so before starting uh without actual implementation like our modular coding implementation uh first of all I will do the experiment okay uh with this uh notebook then if this notebook is working fine then I will try to convert that notebook to our modular coding okay so for this actually what I will do I will create one file inside resource okay and I will name it as let me create one file zero one because this is going to be my first component okay or you can talk about stage so data underscore ingestion dot IPO NB okay because this is the extension of your notebook file ip1 can be okay fine so first of all let me close these are the things this is not required okay so first of all I will select my kernel here so uh ml project okay this is the current so the first thing what I need to do uh I'll import a voice okay operating system module input OS and if I show you one thing okay if I execute let's say my project working directory PWD so you can see I'm I'm inside resource folder okay but uh I want to go back to my project folder okay because this is my project folder okay every job I I'll be partnering from a project folder only so that's why I need to go back okay I need to go back just one further back that means uh machine learning project with ML flow this folder okay I need to go back so how I can go back so there is a method inside voice called change directory you can see HDR here you just need to give dot dot slash okay that means you want to go back just one for folder okay now if I execute it now if I again write PWD so now you can see I'm I'm inside my project folder only okay so the first thing what I need to do now as I already written in my readme file I need to update my config.tml file okay so let's open this config.aml file and let's mention our data and decision related configuration okay so I already prepared the transition related configuration just let me show you see guys these are your data injection related configuration so here uh basically the first thing what I'm doing I'm defining one artifacts root okay so basically whenever you will execute your pipeline okay training pipeline so it will create one folder called rdpix okay inside artifics it will save all the artifacts related to your data Edition data validation data transformation model trainer model evaluation everything okay so that's why this is the root folder I am creating first of all inside that I'm I'm defining my data in decision related configuration again so first of all I'll be creating one folder called Data injection inside artifics okay then this is the source URL my data download URL so basically what I have done let me just copy this URL and if I go to Google so this is the same data I have downloaded from this uh okay well I showed you okay and I have already mentioned I think in that ml experiment okay link there so I've already downloaded the data and what I did just let me show you make one zip file okay I've just made one zip file if I extract it okay see this is the wine quality CSV data let's just make a zip file okay so if you're using any kinds of zip option I'm using 7 zip here okay and I have made it to zip file okay now after making a zip file what I did let me show you I just uploaded this ZIP file in my GitHub okay so let me open my repository ID uploaded this thing so there is a repository I have called bunch in branching tutorial okay you can create any kinds of dummy repository and here you can just upload it okay so let me just open it up now what you need to do just drag and drop here okay just you need to hold it and drag and drop here so it will upload automatically here okay see it's uploading okay then after that just click on Commit it will upload then what you need to do I have already uploaded I will just cancel it okay let me show you like how I've copied the link now this was one quality data right so let me just figure out the name yeah this is the wine quality data just click here and you can see the view rows okay just right click on it and copy the link address okay copy the link address now if I face the link address it is the same okay link now copy this link and just mention in your code okay so let me open my word just mention in your code okay that's how you can also zip your data you can upload in your GitHub repository okay although I provide my depository link okay but if there is some issue with my link okay you can upload your data in your repository okay it's completely fine then I'm creating another directory called my local data file okay that means when it will download that data okay where it will save it will save inside data injection as a zip or data.zip okay this name okay with this name it will save inside this folder then you need to perform a Sandeep operations after unzipping again I am saving inside my data transition okay that's how you can like assign different different path or if you're using any link you can assign inside that okay so that in future let's say you want to change this data URL just change it here it will reflect in your every code okay you don't you are not supposed to go to your uh for this for that okay inside the components and you you don't need to change it in inside the entire code okay only just open this config.aml file and change it here and it will reflect in your library code okay instead of hard coding this value okay so that is the convenience of this config.aml file okay now config.uml file we have created successfully now schema.eml as of now I don't need schema.aml whenever I will work with my data validation at the time I will do it okay then params.tml you are not supposed to write it as of now because whenever I will be writing my model trainer at the time I will Define my parameters example okay now I'll update my entity okay so a notebook let me prepare one entity here so entity is nothing but it's just a written type of a function okay so let me show you I already prepared The Entity my data injection related entity see guys I'm importing data class okay from this data class I am also emitting path okay so here I'm creating one class called dataization config and I am decorating with data class okay so this is not a python class it is a data class okay so in Python class actually if you are defining class variables you need to give self here okay but instead of self if you want to define the variable you can use something called Data class okay so here you can see I have different data class and this is my root directory resource URL local files directly okay if you just open the config.aml the same thing I'm just copying okay this key name and I'm just pasting here so basically what it will do so whenever I will Define my function okay like configuration function at the time this should be the return type of that function okay these are the like value variable it would be written okay nothing else only I have mentioned four variable it will be running these four kinds of variable like Road directory Source URL local director and unzip directory okay so only these four variables will return otherwise if you mention anything it won't be returning okay that's how you can also Define the return type of a function okay using this data class I'll show you like whenever I will Define as a return type okay as of now just try to consider this is a return type of a function okay this is the entity now let me execute so the first thing what I will do I will open my constant because now I need to read my ml file okay read my ml file to read these are the configuration so for this uh how I can read it so there is a constant I created I think you remember you just open this Constructor file and here just write this line of code okay so basically I'm using path and again this is a forward slash N2 prevent this forward slash issue you can use path class okay and you just mentioned it here it will return all the paths so I'm returning my config file path param CML file path okay here left hand side and my schema ml AML file path as you can see okay so these are the path I am returning and these are the variable okay now I need to import it inside my notebook okay how I can import it so simple just call from ml flow sorry from mlproject.constant okay import Star that means I am importing everything okay whatever variable I I have here I'm importing everything okay this star represents now I also want to import my radial method and credit directories method for my uh YouTube's Commons okay if you open my comments.pi I'm importing these two method pdml and my create directory is here okay as you can see ml project utils.com okay that means ml project dot evidence dot common import and create directories okay that's it now let me execute yeah now what I need to do I need to update my configuration manager inside SRC config so there is a um configuration we have created configuration.pi this thing we need to update but we are not updating in the file we are doing everything in the notebook okay so let me Define it here so this should be a configuration manager okay so I'll Define one class called configuration manager inside that I'll be reading all the like EML file as you can see I'm calling it from here okay config param schema path and I'm mentioning inside class variable okay then after that I'm using radioml method and I'm giving the path here it will return all the configuration in the variable okay so first thing what I'm doing I'm creating the root artificial root okay that means if I open this one I'm creating the artifex folder here okay after that I'll Define my data injection related configuration okay so let me just show you how I have defined this configuration see guys this is a simple function I have written called get data induction configuration and I am defining this data in addition config I have created this entity this should be the return type of this function okay so here I am not doing anything I am creating the root directories first of all root directory means I'm creating data in decision directories okay this directories because this configuration is coming from here config right I'm reading the config here as you can see config data in decision okay config data injection and I can access all the configuration from here okay that is what I'm accessing One By One Source URL again if I come here Source URL okay then local data file path okay then my unzip directory if I open it unzip directly okay that's how you can access okay you can access uh from this configuration okay and here if you see this function only will return these are the variable on the root directory Source URL local data file and unzip directory okay this this for for uh variable only because I have defined my return type as you can see okay using this entity I I have only told my method you need to only return these four variables okay that's why I'm returning this thing okay now I think you've got it what is like entity okay why we use entity because if I open this one here if you see I have defined everywhere my written type of a function let's say this function is written as a config box type output okay and I already discussed config box okay now if you want to create your custom return type of any function you can use this data class okay you can use this entity and you can create these are the thing here okay now see this will return these four variable only inside is writtenization config this object okay now let me execute now what I will do I will update my components okay so components nothing bad data injection components like how it will engage the data so for this again I will import some of the libraries here so these are the libraries are needed so request I will be using request to download my data from this URL z