Using Scikit-Learn Pipelines for Data Preprocessing with Python

Key Takeaways

what is happening guys welcome on back to the daily data live stream today we're going to be taking a look at psychic learn pipelines it is something that i realized i probably need to get a little bit better at so i'm thinking that we can do it together and ideally see what we learn i do know a little bit about it because i've used it in the past but i realized that there is an absolute ton of stuff that we can actually get get started and actually do with the scikit-learn pipelines it's like a whole wide world out there in terms of what is actually possible now i don't know if you saw the short that i put out today but the reason why scikit-learn pipelines are so so powerful particularly when it comes to deployment is that you can actually save down the whole pipeline so it means that you don't need to go and bring specific functions or bring specific classes along with your model when you actually go and deploy it which as you go and progress in your data science career you'll realize is it's such a powerful skill i'm sure that a ton of you have been hearing about um ml ops or machine learning ops or that type of thing that is where it truly comes into play if you can package your model into the tightest possible little capsule that is possible that's gonna make just your life a ton easier the it guys are gonna like working with you the data engineering guys are gonna be like thank god this guy's got his stuff unlocked he knows how to do this he knows how to get his models running and all nice and tidy anyway henry why am i live so early uh it's actually 5 35 p.m right now so i'm just finishing work or like day job and actually switching over to doing a bit of a live stream damn 233 my guy you're here early how you doing kevin how you doing aaron how you doing keller malone how you doing by you sarah md how you doing when is the new data science project videos are coming i'm working on them i'm working on them got a bunch of stuff coming hey hey henry hey abdullah how you all doing all right again today's going to be pretty chill probably not as hardcore as what we did on the weekend i don't know if you guys checked out the weekend video but that was like my first like major hardcore live stream where we did it for longer than like sub hour so who knows we'll see how we go tonight but we are going to be doing a little bit of pipeline stuff and at least doing a little bit of practice so again i'm not an expert in this by any means i'm just i'm doing this to get a little bit better alongside you guys hold on before we actually test this out or before we actually get into pipelines let's test down the breakdown board if you've seen any of the tutorials you know the breakdown board you know it's important let's actually see if this is gonna work monday please go easy man yeah we're gonna take a pretty chilled out pretty chilled out i don't know i know i know the feeling oh okay all right we're working we'll work it alright so this is the whiteboard on my what is this on my left this wait on my right anyway so ideally if this is gonna work we should be able to draw on this bad boy so imagine let's actually draw this up so pipelines type lines so imagine you've actually got your machine learning model and let's say we're going to make it purple in this particular case so imagine we've got our scikit-learn model and i'm just gonna draw it as a cube because whenever i think model i think a cube let me bring the mic down a little bit so you can hear a little bit better so let's say we've got our model there right then what our scikit-learn pipelines are going to allow us to do is not just have our let's say for example our scikit or like our random forest classifier model in it so rfc it will allow us to have all those additional pre-processing steps so if we actually draw this a little bit differently right let me just rub that out i don't know why i put the model on the left hand side but what we would actually be doing i feel like uh khan academy at the moment what we'd actually be doing is let's say we we put our model over here this is our final package right so what we can actually do is let's say i'm going to draw our pipeline in yellow we could actually stack a bunch of stuff together so we could actually take our standard scaler and then let's say we could actually go and then grab um i don't know like uh like a one hotting you probably will do this before after anyway one hot encoder so you could actually stack these up together inside of this pipeline that's what i'm actually going to show and we'll actually experiment and then we could actually do like a random forest classifier right now the cool thing about this is that we could actually then yeah i know kevin the breakdown board the cool thing about this is that we could actually then grab all of these different components and let's say for example we were going to use like a pickle or something to save down our model or serialize it so that's an important term serialized s e r i a l i s e a s a whatever anyway so when we actually save this down when we actually get our our like our psychic loan model back what you'll actually see is that it returns pipeline and then inside of brackets you're going to actually get all of these components inside of here which is actually really really cool now there actually is one thing called a like a column transformer so that's not gonna work i'm gonna have to learn how to use this a little bit better but there is also um another transformer called a column transformer and this actually allows you to do a bunch of additional like you can get very selective with what types of transformations you do and like you can say for this column i want you to do this because as of right now if we went and used like um standard scalar on a binary column actually i've never dug into it but i think that this would actually try to scale your one hot encoded columns which is interesting because it's probably not the best way to actually handle that so we're going to dig into this anyway i figured i'd uh bring back the breakdown board let me know if you guys like it in the comments but i do like it so i want to try to bring it back because it helps me explain a little bit better okay all righty so now we've sort of taken a look at the game plan or what we're going to take a look at let's actually do a little bit of coding um in the sign language model you have built you used in the sign language model you have built you use pipelines too right did i oh yeah i think i did for the action detection one let's actually go take a look at that so i think if we go back to what we did for sign language actually the action detection for sign language one i believe we did use we did use pipelines because this was based on landmark estimation wow this was a while ago when was this 2021 damn wait that's over a year ago quite a fair few videos ago just a tad wait see i didn't um we did this with an lstm model i don't think we use pipelines for this one i know we did use pipelines for one of them i can't remember which breakdown board looks good love that you guys like him alrighty pipelines let's uh what do you guys prefer collab or me just doing it on my desktop uh cd what is a voice cloner yeah cd12 let's activate our environment so voice cloner scripts activate jupiter lab you used um pipelines before tom i'm i would love to hear about your experience okay so glms okay all right what are we doing we are practicing pipelines today so i've got this csv this is just the spaceship titanic one that we actually you know i'm tired of looking at that one let's go get a different one let's go find a let's just go get one of the getting started ones what else we got something of a painter myself that's again how do we filter on tabula wow got to get better on calgo hey uh house prices this is probably oh yeah this is good yeah let's use this all right pipeline prevents from data leakage yeah uh guys any thoughts collab normal jupiter don't care let me know all right we're going to start with okay nobody alright well let's just do it in here so i am going to create a new jupiter notebook and i don't believe i've actually gone and set up kaggle on this computer so this could be actually let's go do it with collab and i need to grab my hi ram how you doing what am i doing i'm blanking out here create new api token so we need our api token we are going to create a new notebook and we are going to work with some pipelines uh and then let's just rename our kaggle file i'm gonna call it just kaggle dot json bring it to my desktop and i've already got one there but i ran and created a new one which is going to cause it problemos let's grab that okay so let's upload that and then all right let's go through our kaggle walk through again so uh first up let's create a text box so authenticate to kaggle and all right memory path again this is how i'm been practicing my memory path guys all right so first thing we need to make the directory and it's going to be squiggly line forward slash dot kaggle so that's going to be our first command and then our second command we want to copy this kaggle.json like how does that look over there yes we want to call that copy kaggle.json into dot the dot kaggle folder so i'd actually type in what is a dot ls we should be able to see the folders uh no we don't see folders sample data cd squiggly forward slash line end okay god my bash is a shocking my mate would absolutely smash me all right so there's clearly squiggly line okay so that is oh it's because it's a dot it's going to be hidden okay that's fine um i've got to learn how to actually render that command anyway what we want to do is copy our kaggle.json file into that folder says copy kaggle.json into squiggly line or tilde forward slash kaggle kaggle.json and then we want to change permissions chmod 600 tilda or slash dot i should really call it tilde permanently right i mean it's a squiggly line anyway right so that's going to change our permissions we should then be able to authenticate if kaggle is already installed which we can check so pip list is it here h i j k already there we don't need insult beautiful okay so then we can run exclamation mark kaggle competitions download dash c and then the where was our competition that we wanted uh we were in competitions getting started what was it the house prices one that we wanted advanced regression techniques all right so we can copy this we've already set up the cargo api can you see that there's i might need to move that chat hey uh dash c i've spelt download wrong download all right we're good okay so we should have our data somewhere over here we do beautiful now what we need to do is unzip it so unzip and it will be called we can just copy this all right cool so we got all of our data there so we're going to play around with train but you can see that we've got all of our data so we've got test.csv train.csv sample underscore submissions.csv datadescription.txt i didn't know they came with data descriptions that's sick what's in here oh it's beautiful i love it i like it a lot i like it castle all right cool so we are good over there we've authenticated we've got some data let's mess around now so pipeline practice practice okay so let's actually bring up documentation because this is always good thing to do but just for example let me actually show you how i do this traditionally so let's say we just wanted to be like really sketchy and make all right let's load in our data frame but let's say we wanted to be really sketchy and just build a super fast pipeline to to make a bunch of random predictions this is me really really go in a low low quality uh type prediction let me turn off this camera to make sure we don't that doesn't die um so what would would we do so we load up our data frame so df equals pd but read csv and then we grab train.csv which should load up this uh we haven't loaded in pandas so we need to import pandas as pd and then if we run that all right now let's make like a machine learning model in like a few minutes so or like less um so if i go df.head right so there's a whole bunch of nands uh okay so it's beautiful um and we've got our sales price over here which is that column i wish there wasn't so many columns with missing values anyway let's just grab a couple of columns as our feature columns then so we're going to go df and the reason that i'm doing this is just to show you how i currently use it because i want to improve in it and hopefully you guys learn a little bit as well so what i would typically do is go from sklearn dot pipelines import make pipelines this from my memory path um and then from let's say i wanted to use models sklearn dot ensemble import random forest regressor and then i would also bring in a transformer step so from sklearn dot pre-processing import standard scalar this is also pipeline not pipelines [Music] from sklearn or pipeline import make could be make pipelines make make pipeline my bad not pipelines okay cool so then let's make a really rough and ready ml model so we know that we've got our target column which is going to be the house price right so if we go and take a look does that say it in our data description read this okay so okay we're not getting so df.columns okay so sales price i'm pretty sure is going to be our target oh this is alphabetical that's um [Music] is it all right so it's right at the top sales price so you can see that there the sales price the property sales price in dollars this is the target variable that you're trying to predict okay so let's make a rough and ready machine learning model which will do that this is not best practice in terms of making this model but i want to show you how i use it if you've seen the um the end-to-end machine learning live stream that we did on saturday this is going to be very very similar right all right so uh let's grab a bunch of feature columns so df so we're going to say x equals df and we can grab a bunch of columns what's a set of columns that doesn't have a ton of net uh missing values so let's grab that that that that and maybe that that uh and this and this you see that all right so basically i'm just grabbing a bunch of columns that i want to use and we are going to limit our data set to only those probably didn't need that many how's your monday going so far guys yeah the description who said that kristoff yeah that description is like really i didn't realize they came with their data description uh okay that should be okay we've got a bunch of columns what have i done utilities misc val did i copy those together my guess is i did yeah this should have okay so we've got our x value then our y value is just going to be sales price df dot sale price okay now a little bit of rough and ready uh transformation so let's drop any missing values drop an a and then we are going to one hot encode this bad boy over here just using pd.get dummies which is it's very bit fluid when you use pd dot get dummies which is why i'm like there's better ways to do this type of stuff so let's just wrap this up here so pd.get dummies which will one hot encode the whole bad boy so we need to drop n a over here as well actually we should do this f equals a b t equals uh df dot drop n a no because that's going to drop the missing value so we want to grab let's grab this and let's throw sale price into their sale price so we're going to call this select data frame so it's just selected columns that we're going to be using and then that will give us our sales price and all of our different feature columns that looks somewhat okay boom okay no issues there and then what we'll do is there we'll append drop n a to the end of it just so you can see that there and then we will then go and say select df and we want to drop our sales price column because we don't want that in our feature columns so sales price and then i'm going to pass through access equals one we're probably going to do pipelines over a couple of nights because there's a ton of stuff i want to get through but i've been exploring today select after drop that's going to drop this column we're saying access equals 1 because we want to drop a column not a row that should work and then y will equal select underscore df and just give me the sale price the sale price or yeah it is okay beautiful all right so this is where pipelines come in right let's clean this up so what are we doing do we need this let's leave that there for now okay so what let's add some comments so what we've so far done is we've gone and said give me only certain columns columns so i don't need to deal with uh nands for now no it's funny i was watching the formula one this morning the the replay and there was actually a shot where they're actually using computer vision to track the car and typically they actually have a i think they might have a classifier that actually gives the name of the driver underneath the the what is it the the number or the car like so it's basically the car wait is it the car it's a picture or a tracking tracking value to say that this is the driver so in this case it was tracking charles leclerc and then underneath it actually had i think it was either to show the delta between the current car and the car in front of it and it actually said nan like for a brief moment so clearly the developers have got a little bit of work to do this is the f1 as well guys um if you're an f1 uh software developer right here you need a hand hit me up hit me up guys only certain columns i don't need to deal with nands for now and then all right so that's going to give us so let's quickly take a look at our what do we need mb i've minimized everything now uh what do we want to do so x dot head so let's just double check we've got our features that we want okay we shouldn't have any missing values and it's been one hot encoded all right so how i would use a pipeline typically so i would go uh we don't want this to be a text column so i'm having a complete mind blank adding so how i would typically use it is i'd say pipeline equals make underscore pipeline and then i'll go all right throw in a standard scalar and then throw in the random forest classifier wait we're using random forest regressor because it's a regression problem let's bring this back up here uh and then what i would do is that you can literally just go pipeline dot fit and then pass through x y and then whatever other columns you want i'd normally pass it through like a grid search tv but that's literally our model now fit right so i could then go pipeline dot predict and pass through x which is not right but like that's that's how you'd effectively be able to do it and that's making predictions now so that is the beauty of pipelines because what you can see now is that over here i've actually got the different steps that i would go and use to apply or to to build up a machine learning model but there's so much more to this is like i was reading over the weekend and there's um aurelie jerome's book he actually talks about this and like the column transformer and there's a whole bunch of additional stuff and it's wild but the beauty of it which i'm trying to explain to people but maybe i'm not articulating it well is that you can then go and save a pipeline save the pipeline you can actually then go uh so let's import pickle and then i can go with open let's say pipeline model dot pkl and then we're going to write binary as f we can then go pickle dot dump the that specific model so i can go dump that pipeline as f i've spelt pickle wrong [Music] right so you can see that i've got pipeline model here so that means that when i reload it i'm actually going to have the standard scalar so standard scale is not not a big it's not like a crazy transformation right but there's a ton of additional transformers that you can use with the scikit-learn pipeline api so i can then go with the open pipeline model.pkl and watch this magic i think it's fascinating as f and then i can go uh reloaded model equals pipeline wait no pickle dot load f which should reload the pipeline so then if i type in reloaded model so we've now gone and loaded up the entire pipeline now there are other ways so you saw that i went and defined the pipeline as literally just the steps so this is the nice bit about making the pipeline so i can type make pipeline and then i get standard scale and random forest regressor now i can actually go and extract certain steps so when you typically want to go and do um like get feature importance values or get coefficients if you're using linear models you can actually go into this this value over here which is i think something that not a lot of people teach because it's like oh it's done and dusted but you can go reloaded underscore model and i think you can get type in like get pipeline steps uh it's dot something name steps uh so there's name steps but i want i don't think that's what i was looking for so this gives you the dictionary so then can i go random forest progressor right so then i can get that regressor and then i can type in i can access everything that i would typically go through it's not name steps that's not the right thing get feature params feature names classes i think it's steps okay there so i can go steps and then i can go give me the second step which is index one which will give me my transformer there you go so then i can grab whatever i need out of this so uh we can grab the second value out of that tuple which should be our random forest regressor and then all the values available out of our random forest regressor i think can we not use predict here there you go so that that's returning values but that that effectively shows you what you can actually do with these pipelines there's actually a ton of stuff now let's jump into that before we do that let's quickly jump back on over to the chat that's at least the beginnings of what i want to go through again this is only meant to be five lines of code a day but uh maybe we'll do a little bit more we'll last we'll see all righty to the chat and i've turned this camera off let's turn it back on last time it died midway through the stream okay let's have a look um where did we end up all right so we got colab anybody else would collab please make a video about glms sure why not hey akash hey lakshman how you doing end-to-end machine learning project so many people have asked me about this but guys have you been to the youtube channel there are so many end-to-end machine learning projects on there if you actually go to videos right all right i'm gonna i'm gonna drop this so if you actually go to here like there's a this reinforcement learning for gaming there's like i think there's five four or five different machine learning projects in there and then if you actually go to over here so if you actually go to this playlist deep learning projects with python and keras it's there for you guys there i'm gonna event eventually turn this into one big video but these are all like unique uh data science projects so building an image classifier deep face detection common toxicity building gans building iris detection and building an audio classifier there's a ton in there if you guys need any inspiration as all please do go check that out because there's a ton of end-to-end machine learning projects they are truly end-to-end because it's it's about oh wait i'm not sharing my screen what the hell am i doing all right you guys probably missed all of that so if you actually go to the channel right let me zoom out of this if you go to the channel and go to playlists there's a playlist here that i've sneakily called deep learning projects with python and keras there are a ton of deep learning projects on there they're all very much end to end i go through a ton of stuff this is a monster one where i tried to recreate a machine learning or a deep learning model from a research paper so i think people were asking about that a while ago but there are a ton of deep learning slash machine learning projects inside of this playlist if you actually open it up a lot of these are unique as well so we go through deep face detection we go through common toxicity we go through gans iris tracking there's three more that i've got left in this series so before i make a big video of it because i want it to be 10 videos total but um just know that there's a ton of end-to-end ml projects in there if ever you guys want to check them out but please do i really appreciate it i like that i went and did all of that first time and i wasn't even sharing the screen um alrighty yeah there's a ton of end to end stuff um hey nick i think it would be better if i moved to the chat to the left because data won't change much yeah what do you reckon like move it like i was actually thinking just uh yeah i don't know maybe move it to the left i'll sort that out tomorrow i've worked on that data set yeah this it is a seek data set collected from two climatically contrasting regions in southwestern germany for the period of 2009 to 18. fascinating monday's okay slow monday i've got a model which is showing better accuracy when rf bootstrap equals false i'm confused out of the game i think everyone's always confused with machine learning let's be honest what else we got um it's an awesome monday of regression it is is it not better to use pd dot get dummies data drop first equals true what is drop first equals true do i haven't actually ever used that command anyway we'll check that out again i'm learning but i tend to i've heard using get dummies is not good practice i mean who says what good practice is or not but like i typically don't like to use that because you don't have as much control over the one hot encoding which is fine when you're training but then when you have to go and make predictions on real data you need to restructure your pre-processing pipeline to actually make a prediction so that's just something to keep in mind um well maybe try to find the f1 github page and submit an issue for that now why all right well i'm going to talk about this so why make pipeline and not just pipeline i need to know the difference i'm actually going to show that can you do a complete deep learning project from scratch guys ishan there's the playlist you gotta check it out i'm using a random forest classifier for sign language detection one big video it's coming all right let's let's go dig into that there's there's there's um there's i'm always going to be doing more tutorials on machine learning and deep learning and whatnot so um if there's more like actually this is a good question in terms of end to end what do you truly consider end to end is there like is there stuff that i'm not showing that that you guys need to or want to learn about let me know because um a lot of the ones on there i personally consider them end to end like if i had someone that i was interviewing and they showed me it to that level i'd be like that's end to end that's good you've done stuff you've gone and done an implementation to a certain extent i think it's good okay difference between pipeline and make pipeline so let's take a look so if we actually go and import pipeline up here so when we so i use make pipeline which automatically names my steps right so you can see that the step is named standard scalar and random forest regressor if i'm going to go and do build a pipeline manually i can type in pipe let's say it's not manual i don't want to class it as manual because it's not uh what are we saying i'm having a mental break uh manual let's just call it let's not call a manual pipeline let's call it non-make custom pipeline wow i like that i'm i'm having a mental dilemma over um over what to name the uh the pipeline all right so using the pipeline uh class so custom pipeline we then call it pipeline and then i believe so previously i was actually i didn't have to name the steps so here i now need a name i'm scaling and then pass through what i want to do for scaling so which would be standard scalar and then the next thing that i'd have to go and pass through is the next step so i'd go um rf mod model and then pass through my random virus classifier or regressor uh it takes two positional arguments maybe this has got to be inside of an array or something vegan so you can see there that there's just a little bit more work so that's that's what happens when you're using the pipeline right this is with the pipeline plus and then this is with the make pipeline class so let's take a look at the difference so this is make pipeline model so it would be make underscore pipeline and then i literally only need to type in standard i'm just making sure i don't cut it off on that side standard scaler and then random forest regressor so it does the pretty much the same thing right the only thing is that i can change what each one of these steps are called if we take a look at custom pipeline now that's what my custom pipeline looks like and look at the names this is the important thing so the name first this first step is name scaling the second one is called rf model if we go to make pipeline it automatically names the steps for you so you can see it says standard scalar and then random forest regressor so you don't need to go and name those steps which is why i typically use make pipeline but again to each their own there um there's a ton of additional stuff uh what else are we taking a look at all right let's go and dig in i'm probably only gonna do like five more minutes guys i'm starving uh pipe pipelines sk learn but if you guys want to do more on pipelines you let me know so okay so like say i wanted to the thing that i'm most interested in using or learning to use is the column transformer column transformer here we go with mixed types so this is really really cool i was reading this about this inside of um aurelie jeroen's book this is me trying to do a fake french accent that sucks but you can see here that you can actually stack a bunch of these pipelines together which i think is super sick so let's uh read into this so the numeric features so we create a numeric transformer and then we create a categorical transformer and then with our column transformer we specify what we want to apply to what do you see that how sick is that all right let's try doing this i'm going to throw that up there and let's make this a little bit tad smaller so we are going to look at column transformers okay so what do we need to import first so i think we need sk learner compose so from sk learn dot com can you see that is a bit small there we go it's a bit better from sklearn.compose import column transformer and then how do we structure this so we ideally would want to specify what the numeric features are the first handle numeric features i actually did like i went and studied this a while ago and threw it in the course but i had just i've got a i want to practice and get a ton better at it categorical features okay i've got a wedding this weekend so i'm just doing as many videos as i can until my best mates wedding and then uh we're gonna really get back into it hardcore hardcore um okay so we've got column transformer what's another pre-proc step that we might need like one hot encoder from from sklearn a pre-processing import one one hot encoder and then we would ideally want to identify what our numeric features are versus what are our character categorical features so if i said select df dot uh what is it i think you can type in select d types and then object right and then i can grab the columns right so that gives me the columns that are going to be categorical so i can actually grab that and then i can say so these are going to be categorical features equals that you know what let's just put this in another tab we don't need to see it side by side so those are going to be my categorical features and then i can go uh numeric features equals select df dot select e types can i type in exclude here yep object columns numeric features beautiful look at that all right cool wait hold on why does that say d type object we celebrated way too soon guys numeric features shouldn't be a slash there cool that looks okay and then categorical features like i think it's so so powerful if you can like have that level of a pipeline alright see that looks good all right then what do we need to do so then i believe we define what our pipeline is so let's say we just wanted to scale our numeric features we might apply a standard scalar so it would be um let's create a numeric pipeline equals pipeline and then we would pass through as different steps so we'd pass through i don't know standard scaler let's just create a simple one to get it working standard scala which should just scale our numbers using z-scoring and then if we were to go and use it on categorical features we would need a categorical pipeline so categorical pipeline and then we would say pipeline and then create let's say we wanted to one hot encoder categorical features we would then create a step and i'm just doing a single step but it's at least applying that pre-proc we'll combine it uh what are we doing one hot and then we would pass through our one hot encoder here boom and then i believe this is where the magic happens so we can then combine them so uh let's take a look at the column transformer nope i don't uh my my question mark let's bring up doco okay what do we need to do how do we use this okay so column transformers we then but this is saying for normalizer and then for what is it doing it on is that just specifying the columns there docker is not great okay so we are saying that these are our numeric features we've gone and set up our pipeline similarly in a similar fashion we've gone and set up our category all right so then this is the interesting bit so then we go and structure our column transformer like so so this is naming it this is saying okay all right i got it so we would then go and say so this is our transformer not to be confused with transformer deep learning architectures then we would say transformers uh here we go we could probably just pass through what's called a list right and then we would go step one or for this particular part of these particular columns we'd say this is for our numeric transformers pre-processing we would then go and say that we want to use our numeric pipeline numeric features so then we want to specify that okay i think i'm getting the hang of this so we'd go and specify what columns we wanted so what we want this step to be called what we want to do to it and then what columns it's going to apply to then we'd go and specify the same for our categorical uh pipe line so this would be categorical pre-processing we then go and use that categorical pipeline and then we'd go and apply it on categorical features beautiful what did we import we don't need that if we go and take a look at our transformer that is so cool so you can see that we've now got a oh this is awesome guys this is exactly what i wanted to get to tonight because this is really really really powerful so we've got our numeric pre-processing step and the actual pipeline that we're going to use for that and you can actually even then go and create custom transformers if you need to do a specific type of pre-processing i'm going to use this so much more now and then we specify the object or the columns that we're actually going to use it on and then we're doing categorical pre-processing and then the columns that we're going to use it on ah this is good i like it i like it a lot i like it picasso all right cool so that's our transformer and then we'd actually go and create a machine learning pipeline which would be i think pipeline and then we would go create our steps so first up we'd pre-process so complete so all column let's call it all column pre-processing and then we'd pass through this transformer because again you can stack them like you can go and create a ton of different pipelines and then stack them all together and then we'd go and create apply or pass through our random forest classifiers random virus classifier and then pass through our model that's i'm just going to bring this over to here so you guys can see that's doesn't help at all really got to work out where to put that chat um random forest classifier i think that's looking good uh regressor forrest boom boom oh my gosh that works so now if we take a look at the pipeline so this is obviously a way more detailed one take a look at that so we've now got our column transformer pipeline which is all of this and then we've got our machine so you can do a ton of stuff and keep it really nice and tight now i'm gonna have to make a memory path of this is that something you'd like to see me actually doing a memory path for something like this actually teach you or you actually see me doing it live yeah i really do like these pipelines i've been meaning to get way i've been very much focused on just tensorflow but i'm keen to get back into scikit-learn all right can we then uh i don't know what i've actually gotten done with my data but uh could we then go and just pass through select d after that dot fit uh we would say we need to pass through x and y don't we so select df and then we but this doesn't have any it would still have nulls select df hold on wait would it have nose where let's go back what did we act well we wrote a ton of so much for five lines of code today guys all right so where did we all right we've already dropped na so we're okay we don't need to handle that so we this should already handle our categorical columns then all right so let's do some magic so if i go select x equals select df and then drop transported oh hold up hold up hold up we need to remove so from my numeric features we don't want uh we don't want what's called we don't want to have the target column so if i said select the f drop uh to do transported it's not transported it's sale price getting confused with the this should be access equals one boom because we don't want our target column to be as part of our transformers because maybe it'll i think it'll throw issues numeric features yeah cool all right that's better so now if i run that run that all right so x will be select the f.drop transported comma axis equals one and then y will equals select will equal select df.uh we just want sales price fail price yeah i think that looks okay transported not found sale price i'm still stuck on the spaceship titanic model that we did on the weekend if i do that we get errors okay hold on a given column is not a column of the data frame sales price where is it throwing an error knew it i knew we're going to get it your errors all right so where is that column now stored categorical features is it in here it's not in here numeric features given column is not in is not a column at the data frame but we have stripped it out all right let's think this through so i select the after drop the numeric features so i've run that i've run that anyway i'm getting i'm getting preoccupied but we've sort of at least got the pipeline set up clearly i just haven't pre-processed correctly so numeric features categorical features we might finish this tomorrow it's the time 6 27 ml pipeline.fit i mean my transformer again you need all columns inside of your transformer you shouldn't wait so ms subclass lot frontage lot area misspell ms zoning if i go run bang bang bang bang bang okay no that worked magic guys how awesome is that so then again the beauty of this is that i can then go ml pipeline i know not perfect dot predict pass through x we're making predictions after going through that pre-processing pipeline ah that is so awesome how good is that and again the beauty of it i know i'm getting super hyped up but i can then go with the open um column transformer type model dot pickle and then go right binary as f and then pickle dot dump ml pipeline f beautiful so we've got we can then go and stack in like custom transformers a ton of stuff into here we can get really fine-grained with how we actually manage this because the beauty is that i can go and read it back in so i can run reloaded ml pipeline equals dot load which means that we now have all of those steps safely stored away how good is that oh my god that makes me so happy that we got through that and that we've got it working successfully but that is so so powerful guys don't underestimate what we went through tonight because there is a ton of stuff what that's actually possible with these transformers um if do you guys want me to share the notebook yeah let's share it uh column transformer practice i'll download this for you guys and i will share it let's go save it on github create a new what is column transformer practice let's see let's try saving a copy in github again it didn't work last time we need to create oh we need to create a branch i've got to work out how to do that uh download download jupiter notebook i'm going to upload this for you guys and then we'll answer some questions and we'll wrap it up should probably clean this up but at least you guys get to see it raw so and i did not wait for it to upload let's go back let's upload oh yeah you go guys if you guys want to play around with that link is in the chat you can play with that hopefully you guys have enjoyed this we've gone through a ton of stuff so we again we took a look at how to authenticate to kaggle i showed you some random uh some of my basic pipelines and we also took a look at column transformers which i think is going to be super super useful if we're really going to hit the ground running on some kaggle comps coming up because that's what i'm planning on doing i really want to go start smashing out some cargo competitions and giving it a crack so we can really get a little bit more competitive okay cool so we have gone through let's go through the chat see if there's any other questions that you guys had hopefully you guys have enjoyed this one all righty can do can we use reinforcement learning for regression problems uh typically you wouldn't need to like you you really want to be using them on like um unknown environment type problems is data analysis and prediction using ai the same as in data analysis and using machine learning like neural networks are a great capability but just treat them as well if you're using it on tabular data very very similar it's just you're using a different type of algorithm maru thanks you made my knight you said you are the best amy can we use a user-defined function in the pipeline yes so you can actually use the transformer mix in to do that it's something that i want to do and maybe like we'll do it uh after tomorrow's live stream but nikki thank you for nick thank you for the live glad you loved it me end to end is right from importing a data set using sql to deploying it on flat well that's one type of date end to end model i mean there's a there's a ton but um yeah would like to love an advanced video on pipelines maybe redo the spaceship titanic using a single pipeline i would actually really really like to do that i think i actually really wanted to i don't want to go back to the spaceship titanic video again unless we wanted to try to improve performance but i definitely want to do an entire video purely on pipelines memory part for the win okay all right maybe i'll do it with my door on wednesday we'll see make the camera in the top corner i mean you let me know guys i'll put i'll put up a poll if you think top corner is better i think it actually might be top better now top the top might actually be better because then it's not down the bottom actually that's not a bad idea solid suggestion who said that uh railway thanks man abdullah i'm a complete beginner and i found your videos quite helpful can you please do a live session on data analysis as well as data predictions sure thanks for uploading i'm trying to share a bunch of stuff you also have function transformer for udf sk learn function trend there's so so much stuff yeah alrighty guys i think i'm going to wrap it up now how important to know about model deployment in order to secure a data scientist role that's really really great question it is very important and i think it's something that you should take on as paramount in terms of a periphery skill set so just knowing how to build models is great but knowing how to do it end to end is way way better um but yeah we're going to probably keep on going with transformers and the pipelines on wednesday hopefully if you guys get some ideas between now and then let me know in the comments on the live stream because this is going to go up straight after hopefully you guys have enjoyed it thanks again for tuning guys love you all a piece i'll see you in the next one tomorrow peace bye you