🚀 Automate Experiment Tracking of Machine Learning Models using MLflow, DagsHub & Jenkins Pipeline

Hello friends welcome to my channel on our second video for the let mlop series that I just started we're going to look at how to train a machine learning model which is packaged using flask and how to track experiments using mlflow now first I will track experiments of my machine learning model using ml flow locally on my computer and then I will do the same remotely using duub to do the remote experiment tracking I'll automate it using Jenkins cicd Pipeline and then after we generate our model and the artifact we will version it and then add the model to a registry so please stick around for the entire video and let me know if you have any questions in the comment section below like comment share and subscribe to my channel for more videos like this okay let's get started now in mlops experiment tracking is very important let's say you have a machine learning model and you have data coming in so you're constantly training the model and so you constantly have different experiments so each training you do on your model is an experiment and each training also results in a new artifact of the model now that makes model experiment tracking very important the more data comes in makes the machine learning performs better whilst you are tweaking around the model in the best possible way all right so in the first part of our video you will look at an overview of the codes that we are using for this project now I have the code in the GAA repository mlops 02 I've also cloned this code in my local machine which is the same thing here to clone it just open a location click on the code and copy this link and on your local terminal you just do get clone if you have a terminal like this or you need to do is get clone and then the link that you copied here you just paste it in that location and press enter and that's going to clone your code since I I've already cloned it to my local host I will not do that again now let's see what the code actually does so first you have some templates now this template is what's going to serve the web page so you have an index.html file inside it where simple one and then I have some test cases here if you want to test your code you can use this to test it now this code is actually based on the first projects that we did mlops one so it's pretty much the same thing and then I have app.py because I'm using python so I'm using flask as my python application packaging which is still the same as the app do p that we saw in the mlops one project the only difference is that I change the port number instead of 5,000 I've made it 5,000 And1 but you can still use port 5,000 so it doesn't change anything I have some requirements.txt file here again we use this in the first project the only new one that I've added to this project is the ml flow for our experiment tracking all right so that was up in requirements now let's go to train.py which is the file that actually trains our model when you run it all right so first we import some packages here pickle this will help us to build our artifact we have ml flow and then we have some other packages inside escalin library in Python now to set up ML flow the first thing you need to do to set up some experiment that will pick up the job that you're trying to run so if this experiment is not available on mlflow this code will automatically create it and it will set that as our experiment and this will help us to work under the project name Iris model training so that is the first time you see mlflow here next is MLF flow. startor run now this command will help us to start an mlflow run so that we can track the training of our model and then over here you will see ml flow. logor parm this will help us to use mlflow to track log parameters and in our case our three log parameters are ncore estimators Max dep and random State and of course that's our model we using a random forest classifier model then you see another ml flow here where we are logging the metrix of our model to ml flow some well known metrics in machine learning accuracy precision and recall so when we host our mlflow server we should have these logs we should have these parameters we should have this project name as well and of course the last part is right here so after we've saved the model as a pickle file it is also logged into mlflow and then from there we can version it and register the model so this is a brief overview of the code that we going to use in this project so in this section we will see how to run our experimentation on ML flow locally on our machine as you can see I have a Macbook here this is a MacBook Pro it's my Local Host now from the code that you saw earlier there are some function in here for ML flow to ensure that experiment is logged properly on ML flow you have a set experiment ml flow. startor run and then we have log of Matrix and then we have log of parameters log of matrics and then we have log of the artifact all right with that let's go into our terminal so I'm opening a new terminal by the way the code is in the GitHub Link in the description below and please don't forget to like comment and subscribe to my channel so I have my terminal ready so the first thing I would do you should install python in your local machine just go to Google and search for how to install python you can install it using an AA which is the one that I'm using using currently and then for this project if you go to the requirement. txt these are the packages or libraries that we need for this project and so for that we need to have them installed to install these packages I'm going to use the command pip install I don't want to cash any directories so of course this is going to take a lot of space that's why I don't want to do that other than that you can just use pip install dasr requirement. TST run this command and this should install all the packages that are in the requirement. TST with that let me clear the screen so all our packages has in are installed now we have to train the model to train the model we use the command Python and the file of the model is train. pi so Pon TR not P press enter now you can see some information here and you can see some two new directories created in our home directory here that we are running the project in now the first ml run contains all the different runs that we are going to make different experiments that we will run okay and then the model it's a directory where the artifact is saved the artifact of the model remember it's a pickle file so this is the model package as a pickle file of course ml run it is run zero run that it gives it some number and then over here so what ml flow did is it checked the ml flow server and realized that an experiment would this name does not exist if it does not exist then it will create it as a new experiment if it already exists then it will log this experiment to it as a new experiment with the time stem and so you see a model is trained and log to ml flow successfully with an accuracy of 1.0 the accuracy is just the accuracy score of our model now this is a very simple model so it shows 100% accuracy but that's not very realistic for normal machine learning projects right this is just for testing so I'm model is trained next I'm going to open a new terminal this our old terminal now on the new terminal this is where I will run the mlflow server now I have prepared a command here to run the ml flow server you run the command mlflow server D- host and since I'm running locally I'm using Local Host local is the same as 127.0.0.1 right and then the port is 51 now remember in the flask application I'm using the port 51 so this has to be the same okay so that you can listen all right now you can press enter to run this command okay so you can see that it's running now it didn't exit the terminal which is why I opened a new terminal to run this that's because if you exit out of this ml flow will also not work on your web browser so you can use this open this link on your browser or just go to your browser and type Local Host colon port number 50001 press enter and that should open mlflow dashboard now you can see the iris model training which is exactly the name of the experiment that we have right here created that successfully so if you click on it you see 3 minutes ago and the Run name is called the precious rat 460 now this is the training that we did earlier right here this is the first training we did look at the time stamp it's 1166 if you click on this you will see the time 11606 or 08 in this case but it's pretty much the same thing you see some logs here remember we loged the parameters these are the three parameters we log the metrix accuracy precision and recall this is all the logs for that and then we log the artifact the artifact is a pickle file Iris model. pickle which is right here this file all right now at this point I'm going to train the model multiple times and see if you will find it on our mflow dashboard so I'll run one more the same train command okay so it's trained again now remember now the experiment here exists so there's no need to check or to create a new experiment it would just do the training so that's the second one I'll do another one whilst I'm doing all this the ml flow server is still active so it's listening still so I've done two trainings here let's go to mlflow dashboard refresh this and you see these are the two trainings that we just did and if you click on each one of them you should see the logs the parameters and metrics and the artifacts so if you make any changes in the model just retrain it and that will log it on your mlflow dashboard and if you click on the latest one you should see the new model that you can use for your own stuff now in the next session we will look at how to track a machine learning model whilst you're training it remotely now you can see that I'm running all this locally if I go to my visous studio code and I do control C on my terminal to abort this mlflow master if I refresh this right here it's off now a lot of times you'll be working on a team so you'll be collaborating with other colleagues right on a project so how do you make sure that you have the model the artifact the parameters and all the locks in one place such that it's accessible and that if you exit out of your terminal doesn't shut it down for every single person okay so in this part of the video I'm going to go to DX hub.com you can go to dashboard you can sign in with your Gmail account or you can create an account for that that this will help you to track your model experiment with mlflow remotely show has an inbuilt Version Control kind of like GitHub which also makes it very easy to version your code or even models so so first I will go here and create a new repository so click plus new repository and I create a blank repo now for the repository name I'll call it mlops 02 ml flow remote I'll keep this as public create Repository so you can see Zero experiments zero data sets and our repository is empty right now similar to GitHub this gives you a a link or a command to clone it and add some files and stuff like that right so I'll copy this first link and on my terminal here it has the gate clone already so I don't have to add anything I will just paste it here and this has cloned but it's currently empty mlops 2or ml flow so this is the directory LS and there's nothing in there so this was what I just run now I can create a read me file using this command and then to commit all the changes or any files that I add here back to this repository on duab I run all these commands now I'm not going to add the r me file because I'm going to take the previous files that I used in the previous part and I'll modify that to use it here so I will get all the commands in here at this location and then I will push the changes to our repository okay so as you can see in our repository I've added the commands that we had or the codes that we had from the previous section you have the templates the test files if you want to do some tests have the I have a jenin file here an app.py requirement. TST and have train.py I have the same info on this side on my Local Host this is the train.py this is the Jenkins file it's pretty much a skeleton I'll fill in the commands later then you have the flask application here of course you have the requirement file right here now you see that there are only a few changes that I've made compared comparing this file to the one that we used in the previous part of this project the only new thing here in the train.py is right here everything else is the same thing we are still logging with ML flow everywhere Lo login metrics and the artifact now here we are going to put the URI because we want to log it remotely you use the URI for duab for this first variable and then the username find the username right here for this and then we will create a password for this now the password is going to be a token similar to creating tokens for GitHub accounts to sign in or to to use that as a password similar thing for Dax help and then to automate the whole process that's why I have this Jenkins file here so I create a Jenkins server that would automate the whole training and everything and then we will check the results on the daub repository to track our experiments and in your doop repository you see files data sets experiments so this is where all the experiments that we run will be they'll all be lined up here if you have any models they will be in this section all models will be here and so forth so currently we are in files and these are our files so before we even start training the model I'm going to open a new terminal and I'll use this terminal to create my Jenkins server I'm using Jenkins to automate the whole process of first cling the repository to duub installing packages training the model and you can add more steps but I decided to keep only these three steps I don't want to do it manually and so that's why I created this Jenkins file for that so I need to have a Jenkins server and for that I'm going to use a Jenkins container to set it up now if you go to my dockerhub repository by the way my user username on dockerhub is Ion C I have several Docker images here now I have built a Docker image that is ready to use on Mac OS and the name is ionc Mac OS Jenkins Docker now you can use this Docker image to run Docker in Docker commands on your MacBook again if you want to know how I created this if you go to my YouTube channel the process of creating that Docker image is based on this video it was out of this video that I created that Docker image now if you you are working in on a Linux machine you can use this command to do the same thing so to First do this on my dockor Hub account since this is the image I'm using I'll click on it and if I go to Tags by the way you can see the description here this is a Jenkins Docker image for running Docker jobs on a Mac OS computer that's running desktop so you should have docket desktop installed on your ma all right let's go to tags and this is my image so first I will copy this and use that to pull the docker image to my local machine so this will pull it press enter okay pulling is done so if I do docker images this is my Docker image right there that I'm going to use so clear the screen and then to run it I use this command so Docker run in detou mode and I'm calling my server Jenkins master of course it's in privilege mode for 8080 for Jenkins UI for 50,000 for Jenis agents and I'm mounting the docker socket of my Local Host which is my MacBook to that of the container and I'm mounting the working directory for my Jenkins container I'm using user root and this is the name of the docker image that I just pulled so press enter and that should start your Jenkins container so if I do Docker PS that's my Jenkins container right there now it takes a while for the Jenkins UI or UI server to come up so after a few seconds or a few minutes go to your local machine on a browser go to Local Host p on 880 now the 8080 is from right here the Jenkins is exposed on Port 880 that's the Jenkins UI this one is for the any Jenkins agents that you may have currently we are not using any agents so you see that I will not use this by the way if you want to know how to set up jenin agents I have some videos on my channel right here you can set up Jenkin agent using a container another Docker container you can use kubernetes parts and I have another video on how to use a virtual machine or a cloud ec2 instance or virtual machine to create a jenin agent as well so these are the reasons why we need this 50,000 Port but currently we don't need this in this project so I just skip it let's do uh setup all right we need administrator password and for that you have to do Docker logs and the name of our container think Master now this right here is the administrator password copy it and paste it in here continue and then I'm going to install suggested plugins now this may take some time so I'll pause the video and come back once it's ready okay now for the username I'll just give it admin password admin confirm password admin full name ad admin this is has to look like an email address so admin admin.com continue save and finish start using Junes all right so our Jenkins server is done now you can see we have some error or something here in a production environment it's best to add a buil-in node okay which was what I was showing earlier about using Docker container or kues or a virtual machine as a build agent currently I'm not using it so we can dismiss this but it's best practice to create a node attached to this jenin server and use that as your build node in production environment so I just dismiss this one now let's go further to do some installations on my terminal I'll clear the screen here doer PS it's my container first I'll try to log into my Jenkins container I use Docker exact it the container name let just copy this I'm using bash to connect to it so I'm currently inside the Junkin skin as root so first I will just update and I'll upgrade so I've done my update and upgrade still within the container now my code everything is in Python so I need to Ure that python is installed if I do python version on my container it's not found so just shows python is not installed so I do app install I'm installing Python 3 d y all right so if I do Python 3 Das Das version this is done but if I do python Das Dash version it's not found that's because you need to add a three to see the version but I want to be able to run python with the command python take for instance if I want to train it I want to run the command python train.py not Python 3 train.py okay I don't want to keep adding the number 3 and so what I will do is to create a symbolic link that will link Python 3 to Python and I use the location of the user binaries or python three Now link this to python press enter and now if I do python version I have the same version as the Python 3 version so that if I run python train.py that should still give me the same results next that when you look at the second stage the first stage we will just clone the Repository the second stage here we using pip to install some requirements by the way the requirements has these packages that we need to install so this pip also has to be installed before I do that I'm going to check python DM pip Das Das version now you see that pip is not found this module is not found so I need to install p and I can install it using app install by the way I will use Python 3 or python either way still fine Python 3 pip that's Y and this should easily install pip okay so pip is installed if I run the version here here you can see the PIP version is also found right there I've installed everything I need to install so far so I will exit out of the container so I'm currently outside the container C the screen and then I will restart my container after all this installations so Docker restart container name press enter and it's restarting when I go here and let's say I go the dashboard you see that will it will be loading all right sign in with the username admin password admin enter okay so there we have it we have our Jenkins ready let's go back to our daub this is our daub repository okay so I'm going to be making changes to our Jenkins file first I will need duop MLF flow URL I'll need a tracking username for mlflow and I will need a password for mflow and then I will need to clone the duab Repository so I'm going to do all this one by one all right so first let's go to doop here then on doop go to your account go to settings and then on the left side go to tokens by the way you see default access tokens and you have managed personal access tokens I'm going to use this to create a password now that password is what I'm going to use for the mlflow tracking password it's the same thing I'm going to use as a credential inside my Jenkins this an old token so you can ignore that I'll generate a new one so generate new token for the name let's call it our project is ml flow let's call it ml flow remote ml flow remote duub that's the name that I'm giving it generate token so this is the token so copy this save it somewhere save now if I go back to tokens this is the token I just created that was the name I gave it I will copy that and save it as well and then I will go to my Jenkins go to manage Jenkins credential here it is global add credentials and it's going to be username with password next of course our password is exactly what we just created the token wherever you pasted it copy and paste it in here for the username on doop account if you click on this this is the username the username is iant consult or if you not too sure go to profile and this is the username you can change it of course so I just copy this and paste it here for my username and for the name I'll use the same name I used in Dax hub for the token just use the same name I'll copy that paste it here and there create that should create the token now let's go back to the dashboard I'm going to create a pipeline job that will run this project now before I do that let's do the changes in our Jenkins file okay first off in our train.py I have this values I have the tracking URL and I have um the U username and password the tracking URL since it's duub this is https D hub.com SL account consult slash by the way you can get the link from up here if you go to the repository let's go back this is our repo this is pretty much the URL that we need right so I can just copy this and in my train. Pi I will replace this whole thing with it and then add ml flow so that is our U for tracking username our username we just found out is icon consult we just use that for the credentials for the tracking password by the way the train.py file is currently not on the Jenkins container so I can just copy and paste the token that I created here like this by the way this is just a practice so or in the dev environment so I can paste it in here like that for the tracking password but in a production environment the best thing or your best bet is to create a do get ignore file and then add this content to that file put it in there so that when you push it it doesn't expose this token so that you can reference the file in here but since it's Dev environment I will just keep it as it is like this as our password now with that I'll save it everything else remain the same so train.py is pretty much fine now let's modify our Jenkins file first is the tracking URL so this is it I will copy this this the first one that's a doop mlflow URL past it here username iqu consult past it there and the password because we are currently this Jenkins file is actually from the Jenkins server you can reference this password instead of the plain text or referencing it from a file you can reference it as credentials and then the name that we gave it in the credentials so when you went to the manage Jenkins credentials this is the name that we gave it this is it so I'll copy this and reference this inside credentials paste it here next let's fill in I'm going to close this terminal for for the meantime I need more space let's fill in the commands here first is the command to clone the dark sub repository this is the dark sub repo if you want to clone it click the remote this is the Clone link this is the https link so I'll copy this and then in my jings I'll go to dashboard I create a new job or new item this will be ml flow remote dtop let's use that name pipeline okay you can add some stuff here here but I will leave it as it this and then for the pipeline script I'll use script from SCM and the SC is get based now for the repository URL that's exactly what we just copied from here okay paste that select the credentials for it the branch the branch here is main so this has to be main everything else can remain as it is by the way we are looking for a Jenkins file so let's go to pipeline syntax and this will help us to generate the script so go to get past the URL again going to copy it one more time branch is main credential is right here generate pipeline script this is it I will copy this command and then inside my Jenkins file I'll paste it in here it's quite long but I just pasted it in here now save this apart from this there's nothing more I have to do in here this second stage is just using pip to install requirements. tstd and this is doing python train.py which will train the model everything looks good so far let's go ahead and use our terminal going to open a new terminal by the way this is the one store here by the screen here and then I will do get add dot get Commit This is let's say added get clone command H push that should be pushed let's go to our duck Hub refresh this now you can see this is the Jenkins file here and this is exactly what we just it we added a good Branch command and then this information that we added you check the tr. P file that should also show you something like this this environment variables and then the model Itself by the way if you have any questions keep it in the comment section below and I'll be checking them and also don't forget to like comment and subscribe to my channel for more videos as I'm a small and new YouTube channel and I'm trying to grow so please subscribe and like my videos for more videos like this all right with that updated I'm done with the pipeline script so I can close this by the way in our duab the file I'm going to use to run my job is Jenkins file so this has to be Jenkins file apply and save go to build now then we check the console output by clicking this three dots you can see the first step it's obtain the Jenkins file from duop it's recognized it so you can see that the get checkout part is done and now it's installing the packages inside requirements. TST by the way you see that pip is installing but you can't see any commands showing on this console and that's because if you look at the requirement. txt the way I reference it inside the Jenkins file is that whatever output that is generated from the installation should be sent to a file req output.txt so that's why you are not seeing the console output of the installation okay sometimes it's a little too much you have to scroll so much that's why I'm redirecting it to another file okay so you can see that the installation is also done now I started training the model and you can see that the model trained successfully with an accuracy of 1.0 which is exactly what we got when we run the training of the model on our local machine and you can view this training by the way whenever it trains it it assigns some name some random name to the to that particular training if you click this this should take you to Dax up directly you can also view it right here right you see that everything is a success you can click on the links or you can go to your duub account here refresh it experiment you see this zero will change to one exactly because we just run it once there's one experiment all right here it is and you see the name is traveling chimp 208 which is exactly the name that is given right here chimp 208 accuracy is 1.0 same here accuracy is one right now these are the stuff that we logged you have the random State let's go to the train. pi file here so in our model we loged these three parameters okay those are these three and then we want not to log these other three um Matrix which is also in the the same train. PI right here and that's exactly what you see here now you can run it multiple times and it will start appearing all of them will appear on the side so just going to go back and build one more time maybe multiple times like that just to see how the experimentation looks like so you can see all of them are successful go to duub Let's refresh this you see four experiments right exactly the number of runs here there are four of them and all of them will appear here with the random names assigned to them this was the first one second third and fourth and for all of them the accuracy is still one everywhere this a simple machine learning model so it's fine by the way this is the experimentation on duab if you want to see it on mlflow you can go to ml flow UI right here using this button by the way remember the name of our experiment is Iris model training and these are the four trainings we just did and if you go inside each one of them you see more information about it recall accuracy and precision these are the parameters if you go to artifact this is the model the model is called Iris model. pickle right that's the model file or in the daab repo just click any one of them let say this one see similar dashboard these are the parameters these are the metrics if you go down you see artifact this is the model iror model. PCO okay so that's pretty much how you would use daub to train your machine learning model and to track your experiment with that particular machine learning model especially if you want to do it remotely let's go back to experiments now if you want to share from mlflow you can click here to share the model on duub let's also look at how to version the model and to register the model as well now for that click the particular experimentation maybe this one gave you good results click on it and you will see register model on this side you can click this and you can give the model some name let's say add new model and the model name I'll call it this is resilient zero save it and this shows that model is registered I go here you should see see that this is a model top you click right here you see all models that are registered this is the model you just registered now this is the first version right so by registering you are versioning it at the same time go in here and you see the version one so if you rerun this and you want to version the same model the next one will be version two and so forth click on this and you should see more information about that particular model this is the artifact this a some metrics and you can delete it you can also promote it so as you know in an experiment it's not always you get a good model especially with the data that you working with if you make some tweaks in the model sometimes the performance is not that great or it gets better so you can have multiple versions and so if you have particular version that gives you a very good accuracy or a very good metric you can promote that model as being the best one for you compared to all the other ones or all the other versions of that same model so on this tab here you have your EXP experiment you have your models can collaborate you can notate and if you go to files you have your contents of your repository right here and everything is automated in one place on duub before I end this video one more thing that you can go to settings here and you see that duab can integrate with some other tools as well go to Integrations see that you can integrate it with mlflow like we just did DVC New Relic S3 Google Cloud Azure blob Jenkins label studio so in this model or in this experiment we use Jenkins and ml flow you can use it to integrate with these other tools as well thanks for joining me on this video please like comment and subscribe to my channel for more videos like this and I'll will see you on the next one by the way before we go don't forget to stop the docker container so dock PS you can close all of these or you can log out either way it's still fine or if you do to dock stop this container the container name is Jenkins Master this will stop the container and so you see that Jenkins will not work anymore just close it and duub this is an account I can sign out and close it so that's it I do dock PS so all my containers have exited or have closed all of them so thanks for joining me on this video please like comment and subscribe to my channel for more videos like this and I'll see you on the next one