How to Set Up a Machine Learning Environment with Conda and Pip-Tools

Key Takeaways

what's up guys welcome back to the channel in this video we're going to learn how to set up a hybrid conda in pipto's environment for machine learning basic idea is to our objective today is to start from nothing and finish with a folder containing some basic machine learning code and the entire environment setup configured from start to finish so the first step is to create a con environment and we'll specify the python version now let's go ahead and we're going to create an environment so we'll go conduct create minus n and then the name of this environment is the setup ml f all right and i will specify a python version to uh python 3.7 all right so once we've done that say yes so the first step first step is that now the second step is to we're gonna now we'll activate our our environment perfect now that we have activated the environment we can go to step three which is to install the right cuda tool for our pi torch environment in this case i decided to do a machine learning example using pytorch but uh we it would work the same for tensorflow the only things that we will have to install different packages uh so that's what we're going to do we're going to install and now that we have instead of installing so now the third step is to install the right cuda tool for our pi torch environment now in this example i'm going to be using pytarch code but for tensorflow the the setup will be very similar the only thing that differs is the installation of the tensorflow packages and instead of the buy touch packages but in this case we're not going to install piper right now we're going to first install the cuda tool that i need so in my case i'm going to install the kuda toolkit 10.2 because it's the version that is compatible with my with my machine so that's what i'm going to do so that's what i'm going to do here so that's what i'm going to do here so i'm going to just go auto install the tool kit 10.2 perfect i'm installing cuda toolkit perfect now that i've done that step number four is to export the environment to an environment.yml file so we can do that with conda and export and then we're going to say from history so that it doesn't export a lot of things that are they're not we're going to say condyle of export from history so that we just export to this environment oml file uh what we have installed so far which is in this case the python version and the cuda toolkit and then we say in what environment file perfect now to check we can just come here and we we can look up our file and we can see here that we have in the environment.yml file the cuda toolkit and the python uh and the python version perfect perfect now we can go back to our terminal and go to step number five which is to create the requirements folder now we can go to step number five which is to create the requirements folder with the necessary dot in files now these dot m files are files that will create so that they can work with a tool we'll use which is called pip tools and this tool is amazing because it allows you to keep to it allows you to compile a automatically a requirement.txt file from a set of simple dependencies that will as humans will manually input and it can allow us to keep uh mutually compatible versions of packages in the development code and the production code and so that we don't have to take care of looking up dependencies for packages we want to install to make sure that we don't run into conflicts and issues with compatibility so what we're going to do now is we're just going to create a folder called requirements uh and now what we're going to do is we're going to create a folder called requirements and inside that folder we're gonna create a file called dev dot in and another file called prod dot in perfect uh the product in file will keep the dependencies for the actual uh production code let's call it the the main source code the stuff that's gonna go to deployment and the dev.m file is gonna is gonna have the development packages the packages that we are using to develop our source code and to develop our project all right so now that we have created this file and now what we're going to do is we're going to put the dependencies in these files so in this case i'm going to go to my m file and in here the first thing i'm going to do is since i want to do a project with pytorch the dependencies i'm going to need are torch porch vision numpy and pill and pillow right and in the dev now i can go to the dev dot in file let's say we wanted to add a visualization so we're going to put like wallet maplin for example right and we're going to we're going to put it right here and at the top of this that at the top of this dev dot m file we're going to put a command saying minus c prod dot txt what this is doing is constraining the development packages to the production packages to make sure they're mutually compatible this is a great way to make sure they're keeping everything consistent and this line is constraining the development now what i can do is instead of compiling and generating the uh corresponding dev.txt file and product text files now what i'll do is i will write a make file because i just don't want to do this once or right now but i want to have a make file so i can update things as i go automatically and as easily as possible inside this make file we'll have three basic commands we'll have a help command and that will print the make file commands that we have we'll have a conduct update command to update our environment and we'll have a pip tools command to install compile and sync our dependency requirements in both development and production environments it's important to note that so for our make file i already i have one set up so i will just copy from my example environment so copy the make file here and now i'm gonna show you what it looks like so this is the make file that we have the our uh so the first command here we're printing all the commands that we have inside this make file this is actually was taken from this stack overflow response so these the first three line the first line here is printing the make file commands now instead of going through this entire command since i didn't write it this was actually taken from the stack overflow answer i since i'm gonna leave a github repo that you can access and see the make file you can reveal this command for yourself but basically here we're printing all the commands inside the make file next we're gonna we're gonna set up uh we're going to set up conda update next we're going to set up the conda update command so the condell update command does two things uh the yeah the update command is two things first it uh it does an update to our environment right with the conda m update and um it has an update in our environment from the yml file that we have and then it prints out a warning so that we activate the environment that we created and the second command and the and the third command is the big tool command which compiles and installs the exact packages that we need from the files that we created in this case product in and dev.end right and finally it uses the pip sync command to install the dependencies from these two packages from these two requirements files uh from these true requirement files at the same time which is great because now we know that we're working with dependency packages that we can actually install the dependency packages from two different requirements files and they will work inside the same environment because of this awesome pip tools because of the awesome pictures package so these two super powerful commands really help us out it's important to note that what i'm reproducing here is an updated version of the lab setup from the full stack deep learning course which is one of the best practical deep learning courses out there to learn how to do machine learning for industry and they teach all the practical aspects of how to set up not only how to set up environments but how to develop code train do experiment tracking data management all that good stuff and i definitely recommend you check them out and i'll leave a link to the description to their course that's about it with the make file now we can follow up to uh step number seven which is to run make conda update so we can quickly uh let's uh just be pedagogical let's deactivate uh we'll deactivate our environment right and now we're going to run a make on that update so perfect and now we can activate our environment now this is just the uh this is just to be thorough with you know how we're setting this up uh perfect we can check on that environment.our file so that you can see it's looking good so perfect this is the environmental ml file and now we can run make pip tools and it will install all the packages in the dot m files and take care of any potential conflicts that might exist now we can do that with now we can do that with make tip tools and now we're installing all the necessary packages for this project so now what we can observe is that we should have the uh dependency files generated so as you can see here there's a file called text and that dot x has all the dependencies that we need and it was compiled automatically which is awesome and the same thing for production right and that's pretty much it this is how you set up uh properly and that's pretty much it that's how you set up properly a development uh and that's pretty much it that's how you set up a hybrid conda and pip tools environment to work in machine learning and make sure that the development code that you have is compatible with the stuff that you have in production so as you develop your application or your machine learning model or whatever you're always uh just updating uh these files with a simple uh make uh conda update and make pip tools commands and you're always uh you're always sure that you're developing and you're developing your environ your environment is consistent will be consistent across machines and a really nice feature of this is that by having these two files and using pip tools you can make sure that you know your project will work in machines that don't accept condo for example so thanks for watching and don't forget to like and subscribe and see you next time cheers