Running Machine Learning Experiments in Python

Key Takeaways

hello everyone and thank you for joining today's code long my name is Ree and I'll be your moderator today we're going to kick off today's session in a couple of minutes we're just waiting so everyone has a chance to join in the meanwhile though we'd love to hear from you so let us know where you're joining from using the chat or the comments depending on what platform you're watching on and tell us something that you'd like to get out of today's session if you have any experiments that you'd like to try that you uh are maybe yeah thinking of applying stuff to from this session then let us know what they are in the chat as well uh we are going to be using data Camp workspace in today's session so if you don't have a data Camp account yet already then please uh sign up for one you won't necessarily need a paid account to code along with us today uh but obviously to get full access you will do so uh yeah make sure you get signed up so you can code along with us I'll be posting a link so that you can code along with us and I'll also be posting a QR code on screen so you've got two ways to get on that as well um this session is going to be running uh for an hour until 12:00 p.m. uh stand E standard time uh or until uh 5:00 p.m. in the UK grwi meantime um if you haven't already make sure you get registered for the event on data Camp so that we can send you the recording and the resources as well I've posted a link to that in the chat you can also head over to dat camp.com webinars but I'll also be posting it again in the chat just so you don't miss it so yeah make sure you get signed up so we can send you the recording and resources and make sure that you uh also keep an eye on the chat so you can C along with us as well brilliant I'll be back to repeat these messages for any of you join us shortly but until then enjoy the background music hello everyone and thank you for joining today's Cod long my name is Ree and I'll be your moderator today we're going to kick off today's session in about 20 seconds or so we've just been waiting so everyone has had a chance to join let us know you're joining from using the chat or the comments depending on what platform you're watching on and yeah if you've got any questions or thoughts at any point throughout the session let us know in the chat as well uh we are going to be using data Camp workspace today so if you haven't got a data Camp account already then please sign up for one you won't need a paid account to code along with us today uh but yeah you can code along with us by scanning the QR code on screen when it appears and I'll also be posting a link to it in the chat as well so keep your eyes out uh for that um also in the chat I've sent a link so you can register for to today's event if you register we can send you the recording as well as the resources as well um I've sent a link so you can do that in the chat and you can also head over to datac camp.com webinars brilliant I think that's everything from me and I'll hand you over to your host for today's session Richie Richie please take it away hi there data scamps and data champs this is Richie now a naive perception of machine learning is that your Project's done once you have a model that makes predictions but in reality what often happens is that you need to make several models and then run experiments to see which ones perform better uh once you expose them to real world data and that means you need to start worrying about managing all these different models and managing results of the experiments so in addition to your standard machine learning tools like psychic learn you also need management tools like ml flow and so today you're going to create and run a machine learning experiment today's guest is folit Steinman he's the founder at fit. data he provides science and machine learning consultancy to uh services to companies and uh he previously worked as a machine learning engineer at engines and he's also the instructor of data Camp's mlops Concepts course and the predicting temperatures in London project uh on which this uh codog is based over to you fit thank you Richie uh yeah indeed as you mentioned the general thought about machine learning is that you're done once the machine learning model is done but of course that's obviously not the case um so today we'll be um doing a project on running machine learning experiments in Python um which as Richie mentioned based on a project uh which is also in the workspaces on data Camp um yeah so let's dive into it before we do the code along I just want to give a big bit of background um which I think is um useful for getting a bit of context on this project so today's use case is about um London weather data which is sourced from kegle and we're going to try and predict the mean temperature in in London um this data source obviously it's really hard to predict the mean temperature there's lots of stuff in involved so just see this a sort of dummy data we use um which is easy to understand it's not the main focus of today the main focus is to structure your machine learning experiment that's what we'll mainly be doing um and also learn how to experiment with different models and parameters now why is this important well um I'll dive into this a bit uh in a bit but um mainly uh what you see is machine learning is quite a complex thing to do uh from start to end getting a machine learn to production involves a lot of steps and doing this more uh efficient um is very useful so I want to give a big of background on where sort of this ml flow tool originated from and so running machine learning experiments efficiently is part of something we call mlops and mlops the definition of it is that it's a set of practices and tools to design deploy and maintain machine learning models in production continuously reliably and efficiently and just as I mentioned there's a lot of steps involved from the beginning up to the end right you want to plan the machine learning use case we need to prep the data find good quality data then do model development and then once it's in production we need to continuously Monitor and evaluate it and so doing this more efficiently um can really help and the focus of today is on on this part so the model model engineering and the model evaluation um and yeah the a big thing in doing model development is machine learning experiments and as in any experiment the amount of configurations can become huge so see it as sort of turning knobs to create the best machine learning model we can turn a lot of knobs like we can uh have different machine learning models different parameters different versions of the data different execution scripts and environment configurations um and if we don't have a structured way of doing these experiments we can lose the configurations and the outcomes we found out before so imagine you're working at a company you're doing different machine learning uh use cases uh you end up with a model that works well and then in in a couple of months your manager is going to ask you hey can we improve on this model that we produced a couple of months ago and then you might start to think oh I I didn't really track or monitor anything I've done in these experiments and it's hard to find what you did before um and so that's what we'll focus on today is use a tool called mlflow and mlflow is an open source platform for the complete machine learning live cycle so this life cycle I just showed you um today we'll focus only on machine learning experiments but mlflow can do much more than that um and using mlflow today we actively track the configurations and their outcomes um to make sure the experiments reproducible so in the case that your manager asks you hey this experiment you do done can you redo it um you can easily say hey I've really tracked the configurations I had three months ago um and work your way from there and so um that really structures your process um so yeah let's dive into the code along um let's see so yeah today we'll focus on the London weather data um and again obviously it's very hard to predict the weather but we just use it as sort of a dummy data to uh use it as an example of example data um the focus of this code along is to run machine learning experiments um and as in any machine learning experiment you first want to do a bit of uh exploratory data analysis and then use then we're going to use ml flow to run experiments on what models and what hyper parameters to use um so before we can start we want to um all right let me just make a copy of the code real quick yeah and so before we can use mlflow we need to install it because the data Camp workspace doesn't include mlflow by default and so we can do this by running the um pip install command here and then it's hidden here but it installs mlflow so that's done and then we can import all the libraries including ml flow so we have pendas here we have numpy we have ml flow for the tracking for tracking the experiments uh we're going to use caborn to do some simple visualizations and then we have psyit learn for the machine learning stuff um talk clear so we're going to be working with that stored in the London weather CSV this contains the following columns the date the cloud cover Sunshine Global radiation maximum temperature Min Min mean temperature minimum temperature precipitation pressure and snow depth and yeah let's start here so I think this is something you probably know how to do do weather. CSV so here we're going to load the data using Panda's read CSV function and we're just going to show the first five rows to make sure we've loaded the data correctly so here you see the date column and all the columns we just mentioned so that's great and then let's just show the information of all the columns nothing really special here um all the columns except for date column or floats um yeah I think this is not very hard to do then we have the exploratory data analysis so now that we've loaded the data set let's do some exploratory data analysis to understand the data better includes handling missing values feat engine Ing and visualizing the data uh what we first want to do is convert this date column to something we can work with so now as you see here this is still an integer and pendas has a really nice two daytime function so we can easily do some manipulations on the date um let's start by doing that uh oh F could you just bump up the text size one spot just so it's easier for people on small screens to see is this good that's better thank you yeah great um so yeah the date column we're going to use pandas to datetime function uh and then we have the dates and we're going to specify a format in which we want the date or the which in which the date is now so like as we saw here the dat's currently in this format so we want to give that as input to the to daytime function to make sure it gets the right date gets the year month and date from the column uh let's that first and now you can see that panas um transformed it into this date column we also want to do is check the number of missing values just give a measure a bit how what the quality of the data is you see here that most of the columns don't have that many missing values except for the snow depth we missed quite some values there um that's on a total of let's check the total of columns so that's on a total of 15,000 rows so about 10% of the rows in snow depth are missing then we will group the data by year and month and this comes in really handy when we want to visualize it so what we'll do here and this is why it's so nice to use the to daytime function from pendas because we can now easily um get the Year from the date column and let's also do this for the month now we've created separate columns for both the year and the month so let's just check that to see if it worked and now you see here we have the year separated and the month separated um and why this is nice well if you want to visualize for instance the um mean temperature the thing that we're going to predict today then you can make a line plot using caborn um using the year on the x-axis and the me temperature on y AIS uh oh wait I'm going a bit too fast so now that we have the year and the month um we want to group them it's one step I forgot so what we'll do is we group by we make a separate data frame in which we Group by the year and month list we use this parameter to make sure that it doesn't group them use use uses this group buy as an index and then we can just find the mean per year and month for all these parameters so I'm just going to run this and then we can see what the result is here now what we've done for this data frame is we have the year and the month and the mean all these mean um values from month and now we can easily um visualize this because visualizing it on a day-to-day basis is probably not that interesting to see so now we can do what I wanted to do namely use the year and the mean temperature uh and then the data frame we just created and then we see the mean temperature per month you can see that it goes up a bit over the years and this is just to get a bit of sense of what the data looks like uh um let's do the same for the sunshine this for and then we get the average sunshine per month and now one more thing we want to do in visualizing is this is important for doing the machine learning experiment um we want to visualize a heat map of the correlations so what this means is we look at the uh how each column influences the other column um this will probably become clear once we visualize it so we can make a heat map we can use the correlation function and now as you'll see here we can see the correlations between all these columns so how you need to interpret this this is that for instance the sunshine has a high correlation with the mean temperature namely a correlation of 0.4 which is quite obvious because there if the sun shines the temperature will probably be higher so more Sunshine um increases the temperature and we can for instance also see that um the precipitation so the the rainfall has a negative influence on the sunshine which is also quite obvious because if the sunshines there's probably no rainfall sometimes that happens but quite often it doesn't um and now what we want to do is we want to make a selection of columns that we're going to use to predict the mean temperature because as you see here not each column influences the temperature that much so you for instance see that the snow depth has a very low correlation with the mean temperature um sorry is that correct no so again what you see is that the month and the mean temperature they correlate and for instance the precipitation and the pressure they have quite a low correlation with the mean temperature um and so we what we want to do is we want to take the columns that are important for predicting the mean temperature um as a predictor of the mean temperature so that's what we'll do here let's call this weather metrix and we'll Define The Columns that we're going to use as predictors for the mean temperature so we see that the cloud cover has some correlation Sunshine Global radiation um we're not going to use the maximum minimum temperature because of course that already indicates the temperature so using this as a predictor would be a bit unfair um we'll not use the precipitation because it's quite low we also not use the pressure we will use the snow depth um and we'll also use the month then let's define our new data frame with only these columns cols Matrix getting a global radiation is not an index I misspelled that and now we can see that we've selected only these columns all right um oh yeah there's one thing I forgot we also want to include the temperature here that's thing we're going to predict so I have to reload the data frame again one second let's redo all these blocks and then when we again select the relevant columns uh use the temp then we have all the relevant colums so we're going to use these features as predictors to predict this target variable but before we do that we want to process the data into TR tring and test set um and because the the main focus of today is doing machine learning experiments I've already put this code in the notebook um but I'll go over it to make sure that everyone understands what's happening here so we have a function here that takes the data frame feature selection and the target variable as um parameters and then what happens is that we drop the nuns for the Target variable because we can predict something that's not there then we select the features with which we want to predict the target variable we put the target variable in the Y variable um and then we split this into a training and test set um which we normally also do in in training a machine learning model and then what we want to do is for the missing values we want to impute those so we saw before that we had some missing values in the snow depth uh column and here we can impute them using the mean uh which means that if we miss uh the variable in one row we're going to use the average of the uh rows next to those to to fill in this um value um then we're going to split or impute this for the training and test set separately because what you don't want to do is impute the values on the whole um on both the training and the test set together because that would mean that we can have some data leakage um but that means um we we can also share a source on that um from psyit learn that goes a bit more in detail on what that means but um in in short what it means is that if we do this imputing on all the data it means that we also it might happen that some of the imputing um leaks into the test set so the the mean value that we impute some value with it's G to um leak into the test set and we don't want that um then we're going to scale the values um because it helps to scale values to improve the performance of the machine learning model if all the values or all the features are in the same range and here again we're going to do that separately for the training and test set um yeah that's what the function does um and it's nice to put this into a function because let's say we find out we want to use different features in the future or I find out that my first performance of my machine learning model is really low I can just easily use this function to um use different set of features um it makes it easier to reproduce so already made this uh feature selection before a bit we have the target variable which we need as an input for the function and then we're going to split uh call the function to split everything into a train tring and test set now this all ran fine and now we're going to come to the interesting part here for today which is the machine learning experiment um if you've ever done training a machine learning model um the simplest form of this would would look something like this so we call the linear regression model from pyed learn we fit the training dat data we predict based on the test data and then we can use some Metric to test how well our model Works um now what happens if you do this uh numerous times so for instance you do this for a different data set you do a different training and test split it means you have to rerun this multiple times but what happens is that you don't track the results and that's where mlflow comes in um so if here I'm showing a way uh on how to use ml flow so what we do here is we have we call mlflow create experiment for now we call this example experiment one and what we can do with these functions if we start ml flow start run we call the model we fit the model um we predict using the test set and then we we calculate a metric on well on how well our model does and using ml flow we can log the metric and also log the model so that when we look back in the future we can easily find what we did there and this is nice because if we do this multiple times ml flow automatically tracks the results so here's the example for the example experiment we just run uh let me see something going wrong right this is what I wanted to show so what you do when you hit search runs with mlflow and you use the name we just Ed for this experiment you can see everything in a nice table here so you see the Run ID and experiment ID the start time so this is today just now the metric that we tracked using the log metric function we also had this tag that we used so all the data from this experiment is in this table now um which is really nice and and actually I think it's nice to show what mlflow does in the background it just saves this run with the things you see in this table here so it saves the run with the metrics you used and then we'll come to this later uh the parameters and the artifacts so that's how ml flow Works um and that's also what we will be going to do now um using the London weather data so we will now do this for the London weather data set using two different regression models um and we will just use one parameter but in practice you can use ton of parameters tons of parameters for the specific to the model that you're using um so yeah let's dive in we're first going to declare the experiment name something we track the experiment with then we do a for Loop to go over the depth values um we make sure to use random state which is a parameter to make sure we can rewind the same experiment and get similar results um and then we fit the models to the training data get the root mean squared error of each model lock the parameters and metrics and then we can easily see these results in the table and ml flow has tracked them uh um so yeah let's go um first we want to define the experiment name so what we're going to do is declare variable experiment name um we call this um regression V1 then we're going toare experiment idea ID um we do by calling ml flows create experiment and this gets the experiment name as input then what we will do is set up a for Loop to go over different different depths um that we use as a parameter in our regression models so let's that uh now we're going to use these depths so it's going to Loop over these numbers and we're going to use this as a parameter in each model um we create a parameters dictionary which contains these next depth which is one of the parameters that is called in the um regression functions and as I mentioned we're going to use random state which is a parameter from py learn that allows you to rerun the same experiment with the same uh at the same start sort of now let's define a run name for this experiment so each experiment can have different runs um and we're going to use this ID x value um which we defined here in the for Loop which just uses enumerate so we go over these depth parameters and then using this enumerate we get the idea 01 two three four from this for Loop um sorry to butt in um it's notice the the original bitly link that was posted to this workspace it was posted to the original version not to a copy Reese has just posted a new bitly link so for anyone who has jumped into the original version of the workspace you need to click the the new bitly link that was just posted in the chat go to that and says you got your own copy to work on and you can uh join in COD coding along so uh sorry about the confusion there please use the new bitly link not the old one all right uh V sorry carry on yeah all right thanks um so just as in the example code I showed you we're going to start the run and here this takes the experiment idea that we've defined as a parameter and it also takes the Run name the parameter which we just defined and then what we're going to do is uh get the decision tree progressor model we're going to use the parameters we just defined as input and then we're going to fit the training data on this model and then we'll do something similar for the a random Forest regression that we're going to use parameters and fit it to the training data now what we first do is get a prediction from this each model uh so we'll Define this as predicting the test set using the decision tree regressor and again we'll do the same thing for the random Forest regressor and then we want to calculate the root mean squ error for each of these models um to sort of score how well the model does we can use this our two score function um we're going to compare the target variable test set to what we predicted and again let's do the same thing for the Forest regress this [Music] Direction all right now what we want to do is use the mlflow log parameter function to log the parameter that we're trying out here so parameter that we have in the for Loop the max depth um we're going to log for each run what the depth was that was used in this run and then we also want to lock the metrics for each model so we can use mlow metric log metric function to get the R2 score of the decision tree regressor and let's do the same thing for Force regression um yeah and then this should be all set so to go over it again we've made a for Loop to um try out different parameters for both of these models uh we Define the parameters here and we use them as input in the model um during each run um we predict the test set calculate the score of the models and then we log this um to make sure that we track what the results were of each run let's run this experiment um this seems to be all done then what we want to do or what we can do again using mlow is search the run that we just did use the experiment name variable that we've defined before now this is the nice thing about ml flow we we see here all the runs that we did so we Loop five times through this for Loop which is why we have five runs here and then what's great here is that we see here what we looked we looked the parameter the max depth that we looped over so one two five 10 and 20 and we can see for both models what the score was so for the decision Tre regression R2 score we see that it goes up up until 10 and then it becomes worse after we go to 20 and for the random Force regression we see something similar and this also allows us to easily compare the two models um of course you could also do this in an Excel uh or or just note it down using a piece of paper but that's not very scalable um and if you want to do this for a lot of models want to do this for a lot of parameters you can easily do it like this and then um uh compare the best result and take the model with the best result so um yeah I think that's um that's it for the code along I think it's now time for the Q&A right absolutely so um for anyone in the audience if you got a question for ful um please do add them to the chat in the meantime I've got a few questions for you so um in this case the experiment you running you had two different versions of the model so there was um like just two different types of model are you wanting to compare the performance is that a sort of standard format for an experiment or other other types of experiment that you might want to run um I mean a bit of an annoying answer to that would be depends on your use case of course um but I'd say most of the times um you sort of iterate over over different experiments right so you might want to start with three different models just to see what performs best and then uh zoom in a bit on like for instance the result I had now was that the the random Forest regression on a certain depth was was quite uh or was best on this 10th depth um but what you would the next step here would be to zoom in and maybe use the depth on 12 or eight and and then find the the sort of best score there I think going over it iter iteratively start high high level and then go more precise to find the best model I think that's the the in my case that would be the normal way of working okay um so you're you're looking at like different classes of models and then you're also looking at the so in this case um the depth of the um the random Forest that that's a a hyper parameter you should be looking at hyper parameters as well it seems like there are lots of like there are many different possible models you could you could have and then also I mean you can look at like the different features that you're putting into the model so I mean the number of models you can have it could be hundreds or thousands so how do you go about selecting like which things you want to include in your experiment um yeah so there's there's I mean we we only mention here the parameters and the models of course there's tons of other stuff as well to include like you um you could have problems that uh or use cases that involve more that where you need to involve more the the data quality um um so again it really depends on the use case um what you w to do for experiments okay all right um and I just want to make sure that I've understood the the workflow correctly so it seems like um you basically you want to write your code for a standard machine learning flow so you're um selecting your features you're fitting the model you're making predictions you're crunching some statistics I think you had like R2 score here as As the metric um so you write that code first and you do it for each of your models and then ml flow is a sort of layer on top of that so you create an experiment run the code for each and then ml flow is just over there to help you compare the metrics is that correct or do you want to add some um yeah I think for the machine learning experiments that's indeed correct um afterwards there's a bunch of stuff like um saving the model logging the the one that performed best deploying it that's something that mlflow can can help as well so this workflow we shown today is just focusing on the on the experiment but that indeed would be the workflow okay all right and so you mentioned deployment um I think that's perhaps something is that's mysterious to some people who aren't who haven't worked in a production environment can you just give me a bit more detail on like what do you actually mean by deploying a model um yeah so now uh we're just running a notebook right that's not something I can run to do my predictions every day or or depending on your uh uh use case it might work in a notebook you might do the predictions here but most of the times what happens is that you deploy your model somewhere could be as an API for instance um so how I explain this in in layman's terms would be that um um we could run this model as an API I'm not sure whether the audience is aware of what an API is but um yeah do you want to give a a brief explanation yeah so so what you do with an with an um API in simple terms is that you uh you host it on some sort of uh what you could see as a website or a web service that I could send a call to and I get a prediction back so what happens when I call an API is that for instance you're still seeing my screen right that would be a bit too small but um an API could just be a URL that I call for instance datac camp. ai.com the regression model and then uh what an API call does is that I can send parameters with an API call and a really simple version would be that I just sent 0.1 as a URL call and it would give a prediction back and and in in a browser that would just be like a a bunch of text but normally that would be returned in ad Json um and so that's what also would happen in this case that we deploy it for instance with the cloud service like AWS or Google cloud or Azure uh and we run it as a web service that we can can call using an API call send data to it for in this is very simple example that we only sent 0.1 but in the example that we have here with the weather data we can send the the the cloud and the the radiation and that kind of stuff and then it predicts based on the data um the mean temperature does that make sense okay so uh the idea is you um throwing in giving it some data some Canon request it's running the model maybe making predictions for a specific date or whatever conditions and it gives you the answer back um so in this case um would you be using ml flow to create this API or would you be using other tools um yeah I think ml flow is capable of doing it that would be a tool that can do it for you automatically yeah indeed okay and I guess another question I have it is that um how does it work once you go beyond working on your own to working in a team uh how do you do collaboration on experiments um that's a good question um so now we have this mlflow here in our notebook right where it just saves it sort of locally here as files but what you normally would want to have is that you have ml flow somewhere running centrally mlflow also has a really nice UI um and with centrally I mean that it doesn't save it here in my local notebook file but it sends it on some web server where ml flow is running so instead of having then the results local here on my laptop I would have it in the web server where mlflow is hosted such that we can all work together on experiments and see the results there um and then moflow like I said also has a UI just something you can uh click like a data Camp also has a platform and then mlflow would be the platform where you click through and you see the different expression okay so you perhaps getting your data infrastructure team to set up some Central ml flow and that's going to be available for everyone in the team okay yeah indeed and then also what happens so I mean just in this one example we've got a single experiment imagine if you've got a data team you can be running lots of experiments all the time you could quickly grow to having hundreds thousands maybe even more experiments um there's ml flow of a scale to this um what like how big can you go with this um yeah that's a good one I I would assume but that's an assumption that mlow is capable of like and the web server you're hosting ml flow on would need to be scalable such that it can run multiple instances of ml flow such that it's easily can I think it's a scalable solution in yeah that's my short okay all right excellent uh yeah uh maybe get some from Mar flow to figure out the exact details there um all right yeah uh so my next question is um for people wanting to get started with experiments like what's a good first experiment to run um yeah would you say uh go do it go try different stuff uh take some dummy data there's tons of stuff on Keo but also on data camp that you can use um and try to see if you can make it as structured as possible so that if you want to turn the knobs to get the best model that you can easily turn those knobs so for inance um uh the function I created here for doing the the feature um pre-processing that's a really nice one to have because you can easily rerun it with different feature selections and such and so if someone would be starting doing experiments try to set it up in a way that you can easily run a lot of experiments at the same time to get to get good results um yeah does that answer um I think so yeah um just is this sort of example of having just like a couple of different models and comparing the performance is that like um the sort of the simplest kind of experiment you can do or what what is like what's what's the easiest possible thing you can get value out of ml flow from um yeah so now uh what I just said as well is that we're really focusing on just the model and the parameters but of course there's also a lot in in data and data quality like data is often the first uh uh step where you start improving your model because if you have bad data you can have the most fanciest of models but it's not going to work um so I would say start with the data first making sure you have good data looking at the correlations and seeing making sure you don't have a lot of missing stuff um and and seeing if the correlations make sense um because mlflow you can also also but that's something we didn't touch upon today but you can use different and log different data versions um so I think that would be a good uh good place to start ah so trying running the same model on two different versions of a data set yeah so so taking taking all the knobs you can turn like the data the model the parameters U make sure you can tweak them easily um such that you can see from running multiple experiments where the best results come from and then like we had in the first question iterate from there on what what the next okay so I'm also curious as to whether this works for things like machine learning competitions so I think well uh data Camp competitions are generally structured slight differently but um for like haggle competitions where it's all about can I get the best possible performance um yeah and like most accurate predictions would something like ml flow and running experiments be useful for trying to win those competitions yeah definitely um I mean um I I remember someone that started on C competitions when kegle started and it was really hard for him to monitor all the results he got and uh this is is like a great solution to see where the best results are and then zoom in on how I can tweak it in the best possible way yeah Okay cool so uh for anyone interested in competitive machine learning then it sounds like this is a very useful thing um and beyond that careers what sort of um what sort of companies what sort of Industries uh tend to use um this sort of uh techniques like who's doing machine learning experiments um yeah I would I would say that definitely like Tech heavy companies are using this so um I mean you you would probably have multiple machine learning use cases if you want to use these kinds of tools like mlops becomes interesting once you have multiple use cases working on it with multiple people um um so I think like really Tech heavy companies um uh data intensive companies that they would be really interested in using these kind of okay uh could you maybe give me an example of like uh some business use cases for um these sorts of experiments um well yeah there's tons I mean um you have this big Dutch company called aen who does a lot of machine learning in in in fintech use cases so for instance detecting fraud um that's a clear business case uh I mean in retail I've seen uh usage of machine learning in predicting how well product is going to sell and then doing different machine learning models for different categories of products um that would all be use cases of uh of machine of of these kinds of experi okay so it's very pretty broadly applicable across like a lot of different Industries then yeah but then then still your company needs to be quite far in its in its data Journey so to speak so so a company that's just starting out seeing getting Insight from their data doing some data analysis maybe having dashboards would probably or maybe even experimenting a bit with machine learning wouldn't yet use a tool like this I think okay so there's a sort of order to this so you need to get like maybe your descriptive statistics done first and and dashboards and then a few uh tried just oneoff machine learning models and then you get on to experiments later is that yeah I think this this becomes this becomes interesting once you have at least one use case in machine learning and you have have two or three which are doing multiple experiments for then it becomes interesting to use this okay and um it seems like um the start of this code long was all about um processing your data making sure like all the dates were in the right format and things like that so what do you need to know about um data quality before you start doing your experiments yeah I think um just like I said like data quality um is is often the most important thing in doing the experiment um again bad quality means a bad model in any case so um having high quality data and making sure you understand it is really important in improving your model I mean you can run tons of experiments with tons of parameters and different models but if the data is is bad the data is bad and you're going to get a bad performance so I think the data is is key and um we're also seeing in the past couple of years that there's there's a way more attention in getting good quality data because we all say there's big data and there's so much data but we don't know if it's relevant if it's good enough if the quality is high we just know there's a lot um and that's also something I hear at companies often as well like we have so much data we can get so much insights out of this but that's often not the case because the quality of the data needs to be good so knowing how to load data also how to to to for instance impute it like we did and and uh scale it um that's are those are really um important skills I think okay J you want to talk a little bit more about um imputation then because uh we we lost over that like fairly quickly in this but it it's incredibly important technique so do you want to talk about like what imputation is and why it's important yeah um so what you often see and and in the data set we use it's it's quite uh the data is quite okay in terms of missing values but what you often see is that you miss values and this can have numerous reasons in different use cases but um uh imputing means that we can uh means that we or or what imputing results in and that we can use more data so instead of a way of imputing or a way of handling missing values would also be to just disregard all the rows that we don't have values for but that means that for instance if we throw away all the rows in this data set that we didn't have the snow depth for that would mean that we throw away 10% of the data and so imputing it means that we just find the value that we expect to be uh the row to be alike so for instance with the mean we just take the mean of this value or of this column to fill in the row and so we can still use the rest of the row and not throw it away because it's missing um but still uses data um I do love that is like you basically you're making up data but you're doing it in a sort of a scientifically rigorous manner so it's not just I've uh randomly generated some numbers um okay uh two more questions then I think we'll wrap up so you also uh briefly talked about data leakage I have to say um yeah for people who are doing the data Camp certifications um and not understanding data leakage is one of the the big failure point so this is like quite um an important thing to understand so can you just talk me through what is data leakage and how do you like when does it occur how do you avoid it yeah um so here we do the the imputation and the scaling um uh and for instance when we use we impute with the mean what we actually do is we take the mean of this column to fill in the the values that are missing right and so if we do this for the complete data set before splitting it it means that the the the test set sort of knows the same thing as a train set but they need to be separated because you want to train your mod model on um or yeah maybe to give a sort of a more intuitive example um if I want to learn myself something I'm going going to learn all these examples that I'm going to learn from where the the the what the relationship is between the variables and then for me to if I make an exam and I already have seen all the questions and the answers before then I'm going to make a perfect exam but I really didn't really test my skills so I think that's also a good example of what happens when you impute on the whole set you're sort of giving me already the questions and answers that I'm going to be exam examined on um which means there's data leakage I really like that analogy of just looking at the uh the answer page to uh to an exam before you uh start taking the exam um yeah so I think um yeah the the secret is really just split into your training and testing sets right at the start of your workflow and as that actually makes it clear why you've got this pre-process data frame um code as a function because you've got to run the same code then for your training set and your testing set so you got to do it twice I guess that's why I use the function there yeah yeah indeed and also to um like here it's nice that if you want to experiment with this easily um uh rerun it instead of having to change the whole code in nice um okay so last question for me um so we're focused on machine learning experiments today but they're all sorts of other data experiments so perhaps um AB tests are maybe the most um popular kind of experiment can you use um this same sort of tool set like um ml flow um for ab tests and other data experiments or is it just for machine learning experiments um yeah Thinking Out Loud like um you can use different data versions in this experiment so you can use mlow to do different different versions of the data so that would also include that you use one set of data in the a test and a different set of data in the B test to see how well the model does um so I think you could maneuver your way uh of using ml flow for ab tests indeed yeah yeah I'm not sure whether it's really set up for that that task okay okay maybe use a dedicated tool for that then all right so we've got one question from the audience um before we get to that I just want to say we've got loads more we got three more um webinars this week so I'm gonna be back again tomorrow for a webinar on all the announcements that open AI made last week so you're going to find out about GPT for Turbo custom versions of GPT the assistant API all the good stuff that they um samon and the the rest of the open AI team announced on Thursday uh got um a session on uh revolutionizing decision- making using AI so like just going Beyond like using data for stuff like decision making is like the big New Frontier um that's definitely worth turning up for on Thursday and then on Friday we've got another AI sessions this is on fine-tuning your own llama 2 model so llama 2 is one of the uh the GPT rivals it's like um it's a very big uh large language model so loads of U stuff for the rest of the week please do uh come back I hope to see you again in future sessions uh and now we have a question from Javier so Javier you'd like to know about the pros and cons of ml Flow versus DVC I think D DVC as data Version Control if I remember my acronyms right H oh that's something DBC I haven't heard okay um does he mean DBC maybe well maybe it's worth talking about like are there any alternatives to ml flow that are worth um knowing about um yeah there are indeed um I mean there's each day there's new mlops tools coming out so uh there's tons of Alternatives there I think EML flow is one of the bigger ones um there's also uh Neptune I know of um and a and a bunch of others but not on the top of my head that I know um I I think certainly for experiment tracking weights and biases maybe the the most popular and then there's Comet ml I think there's another one there's a few of them yeah yeah there's loads there's loads if you if you look for it mlop tools I think it's it's it's growing every day you're probably gonna have to if you want to buy one of these things do your due diligence and just like see if the the feature set meets your needs um but MLL it was free in this case actually is there a paid version of ml flow I'm not sure maybe for deployment that's often what what what these uh products do is that they have a free open source version and if you want to like for inst with hugging phase you have a fee version but if you want to deploy it you can pay for it but I'm not sure whether that's Cas with ML FL it seems all open source well anyway yeah you managed to use ml flow for free today so it's probably a good place to get started so um all right uh with that uh we're at time so uh thank you once again Fula uh I was really nice just like seeing like what the flow is for running uh machine learning experiment so uh excellent stuff uh thank you to Reese for moderating thank you to everyone who asked a question uh thank you to everyone who showed up today uh don't forget I want to see you tomorrow for the the gd4 turbo stuff uh it's going to be a lot of fun uh and I hope to see you all again future webinars