Machine Learning in Production with Python | Feature Engineering & Model Training

Key Takeaways

hi there data scamps and data champs this is Richie now often when you write code for data analysis or machine learning Target audien is yourself and maybe a few other colleagues and in this situation it's often possible to get away with writing some Scrappy code that just gets the answer quickly however once you start thinking about putting code into production perhaps this part of an application then you need to start worrying more about code quality so the code May encounter new data sets that you hav't thought about or users May provide inputs that you hadn't previously considered and on top of this if there's a problem with your code then a lot more people are going to see those errors and that means that you need to be sure that your code is robust now there are two important techniques for this to get you started writing robust code suitable for production and the first is making use of functions rather than scripts and the second is writing unit tests so those are what we're going to focus on today now our F Our Guest is Michelle Conway she's a lead data scientist at Lloyd's banking group and Michelle is a full stack data scientist with experience in retail Insurance Finance telecommunications and IT consulting and as a senior manager at Lloyd she leads a high performance mlops team creating data products Michelle is also an ambassador for women in data and was on the women in data 20 data and Tech 2023 list so uh very uh well credentialed uh guest and uh really truly an expert in uh in uh putting code into production so uh great to have you on the webinar uh please uh take it away Michelle thanks Richie thank you for the lovely introduction it was very very nice um so hi guys I'm Michelle like Richie said I am a lead data scientist and I work in an MLS team which is machine learning operations it is where you deploy code for machine learning and data science models often and frequently using devops best practices so I was going to talk you through a few bits and pieces um so on my screen it's just a quick introduction I'm sure you all seen this title when you signed up um and Richie has summarized it very very well so I won't a um stay too long in it but we have a data set for you that is from kle and to keep the theme um we have an open source banking data set that had some marketing campaigns on customers we thought be quite useful so the main goal is to help give you some coding standards that we use in production that are very very good to know and apply and we'll also give you some tips and tricks on how to do that using um two main uh libraries from python pandas and Pest and it's just to help give you a solid understanding of how you can do feature engineering to make your code reusable and redeployable and also to make it testable so if I get rid of this little PowerPoint excellent I'm hoping yes you can see my screen but I think it has been shared this notebook um it has been sent in the chat and I think reys has also said to register if you haven't I assume that's to get the link so it please to register if you need it um like I said we will be working with a kaggle data set which is here so it's on blanking bits and pieces I wanted everything to be publicly available so people can go away and code up in their own time so I've used the train CSV from here my main focus is on writing really good python code so this is the data and I've also created a GitHub REO and in there um it has the sample code that I'm going to talk you through how you would productionize that sample code and then how you would engineer tests which are sitting within this test folder um I have set this up for local development so it gives the background to what we are covering in today's session and also how you would set this up locally with a virtual environment and a pymel so that's there to look at outside of this call so like I said we're going to be covering some really good feature engineering uh I'm going to be talking through the pipe method which is really cool and fun for when you create um decent functions to engineer new features in Python and then pipe it onto your data frames uh it's quite Nifty and Nimble and then like I said we're going to be uni uh doing unit testing uh doing Pie test I am mindful I'm going to try keep an eye on the chat if people have questions as we go but please feel free to um so that is for the moment so I think let's get cracking so the train CSV like I said is on the kaggle one thing to know about the CSV like when you read it in normally from pandas you do your normal import pandas as PD you need to be really careful about the separator because if we go and actually look at this St set it look a little bit funny it Should Crash on me um I need to do a print most likely so see the way it's rded in a little bit funny it's not a traditional CSV where it's separated by commas it's actually separated by semicolon so you kind of have to be careful of that on the files that you read in and make sure I put in a semicolon and hopefully this will make it look a little bit better when we do it nice okay so we can see we have C CSV read in I've just done a head where I look at the first five rows but obviously you can change that to look at more rows hopefully it will behave and print out cool it does and it's got more colums and because of the notebook it doesn't show the whole thing entirely I'm just going to talk you through how we would do some feature engineering because you can play around with different bits and pieces about like creating new colums so I might just I'm going to move backwards and forwards between my local IDE pie charm and this and I'm going to make sure I update the notebook with what I'm doing so let's say we go to create a new colume and we do that by doing a really easy way of just assigning it directly to the data frame and I'm going to do it this way but this time I'm going to be like I decided I wanted to do a nested list and I wanted to do a few boundaries so I'm going to look at ways of creating that so I'm going to get it to Loop through the duration so there is a colume call duration in the features and I want to look at the annual one but I want to do a Bound by dividing it by two just to see what it looks like so that's one way of creating a new feature and then another way you could use uh topples I'm just trying to give high level examples of creating stuff but I will focus more on other features that you can engineer and also how would you make that quite nice with in functions so another one is if you were using a topple again using similar syntax to how you would do the above one and how I'm getting again I'm going to get that to Loop through um The Campaign colume I think CU this data set had different bits and pieces from whenever um customers were contacted there okay cool so these are two examples of how you would engineer two additional features so annual duration and campaign duration just going to give this a quick run and then I'm going to upload into the notebook y so I think ree has put in the chat about registering to get The Notebook so that is thinking about it I think you can as well Cod along from here too this is where we're talking about ceration and then campaign limit these are just examples of how to create and hopefully they have created nicely that's just queuing you can see here it's created a couple um a nested list and it's done what I've asked and it's dividing by two on the values that are originally in the colum and producing it out I just want to give you flavors of how you might do this in a really nice way um rather than going through for loops and keeping things ized and in one line and then here I was using Topp of format and the nested list and this was sing the values okay there was so it didn't come up quite well you through it to get different ones um so some of you might be familiar with group eyes so I this is where I wanted to give an example of sample code of doing a group Y and I wanted to do it on um marital status um colume I was going to apply a Lambda function on a couple of colums I was going to look at the balance I was going to look at the age and I was going to look at annual duration and I was going to create um new group by returns of those so I might look at what is the balances for each marital group what is the mean age for each marital group and to help code that up um again I'm just going back to my ID because it's just quicker for coding and it also spots when I make a little spelling mistake which I quite like I'm going to group by marital status and I'm going to use the apply method and then I'm going to do a Lambda because when I'm grouping I want to do a few little things and lambas are nameless functions and I'm I'm going to use a kind of series but in there I'm going to construct dictionary where I can start to do so like I said I wanted to look at the balance um of each marital group I think I'm going to just look at the max I just want something nice and easy uh because this is the warmup this the sample code that we're going to then productionize started my spelling already amazing and do [Music] Mage and then [Music] I'm going to more but I'm just mindful I'm going to stick to these two now when I run this locally I do get an issue I do get a warning error which I silence by doing and include groups as false but that doesn't come up when I do it in the notebook so I'm actually just going to take this little bit here cool so this is the group I was saying about that I've just described above I have applied everything I want and I'm actually just going to assign this as a result and then I'm going to work on printing out that result um and because I've mainly focused just on creating two group eyes the balance Max of the group and the mean age that should do us I just want cool so this has gone through the customer data set and it's looked at all the different marital status of the customers and it figured out what is the mean age of them uh most single people were 33 married 43 divorce 45 and then this was the max balances on their account um so now I want to look at how we make this code more productionize and a lot of the times really good stuff to do um is to create functions uh in particular if you are changing or if you're adding in a new feature sometimes it's better to create that as a function so that you can pipe it around and apply it in different faces because then if you need to change that you don't need to go into multiple different places and change a new configuration um like let me give you an example um okay cool I do production code so let say I want to create a function I want this to calculate the no balance and I'm going to really descriptive with what this function is doing because it's better than going death balance um and it's good to do type inss when you do create functions so here I'm using the I pulled in data frame in series yes you can do the import pandas as PD it's just to kind of make it nicer and tier it's a stylistic thing you can do like import Panda size PD um and then do that way as well my data frame so I'm just basically saying hey I'm creating this function I'm going to pass the DAT a data frame and I want it to turn out a data frame and you should always do like a good dock string ion into too much detail because it's going to be a oneliner function um but I basically want to add a colume with the mean age of all customers okay Co amazing and then I'm going to do death and I should be just want perfect just keep an eye on the chat I have a few screens open so I'm moving backward forwards so I'm going to call my new feature balance Max and that's basically going to be taking I'm going to use the balance colume I did too many things too fast I got extra brackets which I don't want and then I'm going to do Max and I'm going to return the whole data frame and the reason for doing this is because then I can pipe it quite nicely and I'm going to be doing piping with you soon so that's one example of function to engineer any fature so that we can then go and pipe it um um and I might do a similar one but I'm going to do it for age so instead of balance [Music] um I'm going to do age and I might want to do it as the mean age like we did sample code again same thing I wanted to feed a data frame come out now I need to change this so I change this to customers mean age so this is my function for cost is mean age perfect um and then this will work amazing so I'm just going to bring these two bits and pieces back into the notebook um I also just optimize this I will probably leave these lovly bits and pieces so this is the first function getting Max customer age and then this is the second one so now let's see if they're going to work and do what we want them to do so if I take this function and I apply DF on it so it's just running it it's having to think and it's compiled so it's returned to lovely data set that we've gotten from kagle and there it's put on the max balance first which is what we were looking for and let's see now if it's done the same with the customer age just so because um pie charm gets my spelling a lot better it's thinking about it amazing and let's see has a done customer AG in here yes it has now another little bits and pieces is I'd quite like to pipe these functions so I'll just do um an example of like how you would do a pipe I won't do it as a function but I'll just do it as so if you take your data frame and you can use the pipe method and then you can say hey I'd like you to pipe on a function that I've made but the this pipe is looking for whole data frames to return it back on otherwise it gets a little bit funny because it'll be missing stuff and it'll bomb out I'm saying please pipe on the max customer one and then I wanted to do the customer's age and then that should print out very me nicely perfect so that is an example of how you would apply those functions using the pipe which is quite nice I know it may seem like Overkill because I've just done two small functions but I'm being mindful of time but could you imagine if you had 50 functions or 20 functions piping them on like this is quite useful and the reason for doing it as a function is like let's say you go back over your code and you're like oh actually I really don't want to do the max I want to do the M um the medium value or I want to do the minimum value value you would just change it once and if you had it in loads of other places in the code it would automatically change via that function but if you had a hard coded you're going to have to change it in multiple places so it's quite good to reuse the code and minimize dependencies of when you need to change bits and pieces um this [Music] one okay and then I need I might have a look jump into the unit test because this is where things will get um interesting because when you do unitest you're basically assessing what good values to like limit for bits and pieces um let me just amazing so um there is is a very nice this is importing production Cod do we want this I've not used it use this okay so there is a thing in py test called um fixtures but first I'm going to show you how to do it without fixtures it's a little bit longer and can be a bit clunky but it's fine so when you write your test you normally say test underscore and the name of the function you want to test so I want to test Max customer account balance amazing and you can write a d if you want but I'm going to move into I need to get a test data frame so for this T Test data frame I'm just going to use Handa one going to create a little dummy one doesn't have to be real data but I just want to be thinking and calculating correctly so I'm going to create one it'll have balance a colume and I just want to give balance a few values so I might just give it like five so balance of 100 balance of 200 100 400 okay cool and then I want to um I need to see what the actual value actual um data frame what I call it actual will I yeah I'm going to use that function that I've created what is a customer Max balance and I want to pass it this test data frame because I just want to see like what will it generate and will it be what I want I'm not going to run it just yet but logically the test data frame is saying hey here's five balances for customers now logically the function we've written the max value should be 500 so I would expect this test to return 500 um so I would want to do an assert I'd want to make sure that balance Max which is the new colum we creat is in the test DF volumes and I'd also want to start that like the actual DF now because everything is the same because I put it all the way through because I was designing this for a group ey which I'm going to go back and slightly tweak I want to just make sure that or actually so I might change this to actual maybe I'll keep this as actual DF then I just want the actual value which is going to be from this data frame and I'm going to do the top ey lock so I just want to P the first row value and I want to seert and see if that's equal the expected so the actual value and then what you expect that there so this is a Dy test that I've created for our function to make sure it works bring that back the notebook okay amazing now let's have a test to see if this works ah it's sparting things which is good because it's giving different values in series so there is a thing in pandas where you can assert the series are equal which I need to be mindful of let me just have a back cuz I need I didn't put in one actual colume that I needed here now let's see do my indents I'm going to get this one because I BL out Okay cool so that worked now like I was saying we logically went through and went okay cool if this is the test data frame the test on me data and we're saying it's 500 but actually if we put down or we're expecting to be 400 then this test should fail let me swich see a little L message pop up yes that was as expected um so it's saying this expected value does not match what actually happens because it's meant to be 500 so it's a good way of testing your fun and as well if data changes or if some like if someone changes a function without changing the test it'll catch that in the logic and in the pipeline so we can also do another test function for testing the mean age let's have go with that so again it's going to be quite similar um actually I'm just going to use this as a base so again we want our test to match the name of our function and we are testing customers mean age customer mean age and we want it again be thecore function perfect oh it's customers I forgot my S see this is why I like this ID so we're going to use that but again we're not using balance as our test data we're going to have to use age so I replace this with age then I will so would say we do so I need five values again so we do some 30 40 50 60 70 um perfect I need to put the eyck back in here and make sure I'm flagging it as age mean um medium age here should be 50 not 500 and I need to change this to AG mean and hopefully that should work for usol and then I'm going to make sure to pass it through the data frame make sure I watch my indents so I do test and passing the data frame that we're using that should pass me but if it does I'm also going to make it oh it hasn't take zero repetitions but one was given oh yeah sorry didn't need to call it because it's not being called in the thing why is it not happy [Music] so age mean it does not like maybe I have renamed it differently in the function which I'm just going to test check not test I've called it h Max trying to reuse other functions so let me fix this to mean I pipe this [Music] again okay that's looking a little bit better and now if we come back let's see if it complains cool [Music] um sometimes it's good for stuff to buom out because being able to read stack Trace is like hugely important um so that should be fine okay amazing and then that's how we do our testing there now we've created two nice functions with two good tests and if we go back to the sample code let's apply it so that it looks nice and clean but I will need to edit the functions that we've made so the function function but actually for the group ey it's going to want to do this but with just a single value that will like comput it so I'm going to use these functions but I'm going to alter them so that they can handle group eyes and turn an integer value because then they'll be more handy for piping if I just take these functions and just say hey just for group by customer mean age and then I will want this to do age mean and just return one value so it'll help with Group by and then I also want to do this for this max value for getting the max account value okay aming do the same just for that demo of the group by I want to make sure we do Max I need this to give me some python okay cool amazing so these are altered functions that'll help with Group B so remember we did a group by further year and it was like a little bit um like it's it's good but like I said if you had this groupy let's say in multiple files and multiple places and you need to change it from being like the max or the Min by having it in a function it so instead of having this bit where we're like okay cool I want to do the balance you now can use these functions and they will deliver the same bits that you were expecting so instead of do here you pass the X through the group by [Music] function likewise here let me pass the X true here perfect I'm hope this works if not I'm going to enjoy debuging it and it has um if I don't use the original functions it would error because it's actually not let me can I copy this out and just run it so you'll see why I need to create new little group by functions I get rid of the group by ones and I just use the regular ones like it it should not be happy with me so it's saying the output is too large because it's passing in a whole data frame and returning a whole data frame it's kind of like what are you doing to make there's too much going on so that's why I had to make like tiny little edits putting in the ey loocks and just editing it so that it'll be group by compatible but to make it like pip usable I create them as pure functions that pass in the whole data frame pass back the whole data frame so it's just a lot of it is style and functionality on what you need to make it work um so they're just examples how you would make Scrappy code not Scrappy code but how would you make code more durable and last than um having it around the place so I'm just going to delete this one just to show you how it works and yeah this is here how you'd pipe it on so there's lots of loads of different methods um did have one more I'm not too sure if it will be compatible in the note but we will try um do it locally but see the way we've done a test data frame here um you could use pyc fixtures whereby um create a dock string or a darker decorator saying darker and what we'd say is like hey I want you to create a test DF you create that as a function and the reason for this is because because we're using p test you can then go pass that around to be your test data frame and you pass it through as a parameter of the test functions so I'm just going to create a very basic one and I'm just going to say hey uh data going to be this and I'm just just going to use these values because I think they're nice and I quite like them and they worked um one and I I'm going to take age as a [Music] number perfect I think that needs to be better forming it is not happy with me but it is fine age that's our data frame that's our colums maybe it's I went too high up oh yeah it doesn't like when there's spaces okay cool and then instead of creating the data frames here and passing them through you would just have them once and then pest lets you pass them through your fun functions as fixtures that needs another return cool so that's one thing you could do you could use that pext fixture your functions are backward oh sorry someone is messaging me hang on I've had my camera off because my computer likes to hiss at me when I am doing cording and screen sharing at the same time and I didn't want it to crash out now when we were doing the unit testing like I said was another way of doing your test with the use of P test and C let me just have a check over creatures peline I need to put these the right way around cuz it's not and I have all bits and pieces that I need to help do proper testing so yes that's one thing keep in mind is this P test sure you can pass that around rather than having to create and especially if some functions use the same data it's it's just handier uh bits and pieces and I think that concludes my little demo if anyone had any questions feel free to F them in the chat uh yeah thank you so much Michelle that's very cool I have to say um being able to write functions and unit test it's basically it's a productivity superpower um you just spend so much less time just trying to hunt around going what's wrong with my code because you've got proof that it does actually work or not so yeah this is s such important stuff I hope everyone's uh learning something here all right um there is uh one question from the audience so far so let's put this on the screen so question comes from Kyle saying does the output of one function pipe into the next function so for example can you pipe the output output from the first function to the second so I think the answer is yes but do you want to just go into more detail on like Kyle you're right the the order in which you pipe does matter like so yeah so see have piped here and I put hey can you please pipe the function Max customer account balance it's done Max balance first and then it's done customer mean age and it's done that second now if I had another function that was like I'm making this up because I've not created it but it was just like some the balance and age that wouldn't work if you put that first because it would be reliant on the first two outputs so it kind of chains it in order of like when you need it so you need to be mindful of that because it's kind of like instead so a nonproduction way of writing it python so other way you would you would have your data frame you'd say hey I'm going to change this data frame by putting it through Max customer account oh then I want to change it again because I want to add on the like customer mean age it's just a nicer way of writing it in one line and likewise if you wanted to get like a sign on this is not created so it's probably not to let me um do it what I given me a warning so instead of writing three lines of Cloe you write it as one line but obviously if you have about 10 pipes you can like format them so they like stack on top of each other and that's quite nice um like they would look something like this but they need to be um ah here return I shouldn't have hit return your IDE would form M them in a particular way so they loop around so yeah the order does matter let me just fix that again but it's a nice way of just writing things in one line nice and clean and because the functions are clearly named and remembered when you're in an IDE um I don't think I actually hyped these down here let me just so let's say I oh yeah I think it's not it's got the pretend function there yeah um so let's say I decided had it in so these are going to bom out for me I'm just going to import them so Cod base you just you find the of where they are and you know that it's just used so this situation I'm using it in my sample code and I'm also using it in my test so it's kind of like it's to reduce repeating yourself as well as typing it in a really nice constructive way nice um we do have another question the Audi before we get to that um this should work oh no oh was thinking about it all right I actually have um a question for you I'm keep in on your opinion on when you should write tests because I know quite often people will write the code and then just at the last minute once they got the correct answer think okay now is time to write tests uh and it's right at the end of the project but I suspect that's not the best practice can you talk me through when you think people should be writing tests yeah so it um it's a balance between it's a not at the end of the project but also not at the very very St because if it's a proof of concept you just want to do the analytics but at the same time there's nothing stopping you from using functions and I would recommend using functions because you'll end up with really messy notebooks that loads of scripts whereas if you write a function in a p script and import it it tidies up your notebook so I would Recon is like definitely do functional Python and there is this lovely thing that you can do you don't have to write the test if you're stuck for time what you can do of the test and pass have it as a placeholder to come back to so there's a quick way WR test test but I I would recommend yeah it's quite good and like it runs but obviously um this will run it it's it's not ideal this will help you and as well if you're your functions refactor your tests but I would recommend using functions as much as possible and then test when it comes to productionizing it getting it ready like look this this is asked for me so I would give if you don't want to um do your test I'll put um copies of that in actually help let me get another box for yeah you could do that as well skeleton test all right uh super thank you we got one more question from the audience this is from Kyle again so well done Kyle for all these questions um so how do you suggest organizing tests is it better to include tests in your scripts or maybe keep them in separate scripts and same question for defining your own functions yeah so they they say when you're like organizing your project like you should have a test folder and that's where all your test scripts would live and the recommended naming convention is create Python scripts and go test underscore and have the rest of it match the name of the Python scripts that you've written your functions in so I've done production code. py and I've called my test test production called oh sorry I aness in a spelling mistake um dyslexic thinking but the structures normally have a test folder and name all your test scripts in line with the Python scripts that has the functions in them so that everything's clear and it's labeled and it's like it's easily structured because if you let's say you have um your repo structure has like five different folders and there's 10 different Python scripts in each one that's 50 scripts I'd recommend having a similar folder structure and having a test underscore the name of the folder it's trying to do the test on just because then people can pin in and out and as well by having it separate like if you run a cicd pipeline you can just go hey P test do it in this test folder please here's all my tests it's just it's quite nice nice and Nimble and easy to to use but You' be able to like track back everything that you're using by again control clicking into functions um when you're in an ID and it'll tell you where you've used this you've used this function you've imported it in this um sample Codey um and then you've used it in a pipe and then you also have a test for it yeah that's nice and yeah worth highlighting that in general when once you're doing tests you're not going to be using notebooks you're going to be using just raw python files um I think Michelle this is something you mentioned to me earlier that's your preferred way of working as well I think yeah it's just because um you end up bundling up the code and like deploying it into production and having new changes um and again that's the way all code is written for all python libraries it's done similar structured and like compiled up and then that it makes everything pip installable um which you'd be used to from pip installing bits but if the notebook is to just give a bit of commentary and kind of flavor as to the structure that we did like what we were doing breaking it down for my spelling and making sure I get things in like a nice grammar you'll see everything is structured really well um in my repo I have mentioned linting linting is where like you can do formatting to make your files look nice so in here I've applied I've applied black a lot where I could you probably didn't notice me doing it but like if I had so of double quotes I had everything as like single quotes um black doesn't really like that and it would kind of change it for me or if I had like those nothing spaces but if I had trying to think something where it would I want it to break on me so yeah so see this it's got like it's got the series if I have bundled up into one line it's not very readable but if I put linting to it it's going to be a little bit nicer and it's normally like best practices so if I do I'm running black on this you can see it's formatted quite nice for me so that's why I was keen on using the ID because I I was like cool this is in a nice format and then it's easier for everyone to read from The Notebook um so that's where I'm getting the yeah nice formatting it's from the link again everything yeah I'll go for it um yeah uh I have to say certainly once you start collaborating with other people then you won't have exactly the same code style written by everyone so the uh that python black uh code style is very very useful excellent um all right we have no more questions from the audience so I think uh yeah just uh thank you again Michelle this is like very very very informative session uh very cool stuff all right uh yeah thank you so much uh all right before everyone dashes off as we say we've got some upcoming events uh so the next webinar is going to be tomorrow so that's on building effective data teams so uh we've got uh two data leaders talking about uh how they hire people uh how they uh uh make sure their data team is performing on Friday we've got a POV session on analyzing uh Euro 2024 data and in fact there's another code log on Euro 2024 date at this time sequel next Tuesday so for soccer fans we've got plenty coming up for you I hope to see you again in future events so yeah thank you again Michelle for uh presenting uh thank you thank you for having me uh thank you recent moderating thank you to everyone who asked a question thank you to everyone who showed up today hope to see you all