Every Beginner Data Scientist Should Understand This

Key Takeaways

do you really understand what happens when you're on pip install pandas if not then this video is for you so this is definitely something that I wish I understood earlier in my data science career because it just overall helps you to write better code work better with external libraries so I want to talk about files modules classes and functions for the Python programming language if I for example come over to visual studio code and I do a standard import like import pandas SPD but do you know that when you import the pandas Library you actually import a whole set of files that is underlying that Library so you can see how I can click on the module basically that I am importing over here and look at all the code that is behind there and this can be really convenient when you're trying to look up documentation see how a library or a class works so that is really what I want to briefly cover in this video how to work with that because for me it took a very long time to figure out that you can actually click through on all your functions modules classes and actually figure out what's going on behind the scenes in the back end basically and this can really help you to not have to constantly switch to stack Overflow to try and find out what for example the specific parameters are for a certain machine learning model from scikit-learn for example so let's start from the very Basics what happens when you run pip install so when you do this you basically reference the python package index and that is just a large repository a large Library basically of all the PIP packages that are available so for example I can search for pandas over here and here I can see pandas2 and you can see how you can pip install panda to or just the original pandas pip install pandas and from here you can also go to the home page or also to for example the source code so here you can basically see what the pandas library is what it contains and usually when you work like in your code editor for example and you go to let me open up a new terminal over here and you do a pip install so for example I do a pip install pandas within this environment it's already installed but usually you don't really understand what's going on because you are looking up these tutorials and you see oh if you want to use this package just do a quick pip install but what is happening in the background is that your pip basically is referencing to a certain GitHub repository where all the code lives where everything is and it pulls that code onto your machine so now when I for example I import the pandas library and again I can click on this module over here by if I own Mac if I press command and if I'm on Windows you can press control and then basically see what's going on over there and then from here I can right click this file and I can refill it in the finder or the windows equivalent would be to refill it in the Explorer and I can basically see that I am here in a bunch of files and I am now in the init file from the pandas library that you can see over here and if we scroll a little back to see what's going on you can see that we are currently in a conda environment so this is a specific environment that I created for this project and here you can see all the various contact environments that I have and in that environment is a lip folder and in there is a python folder so I am using python 3.8 in this case and then over here are the side packages and here you can basically see all the packages that I have installed using pip and also like all the dependencies so usually what happens if you install for example pandas the pandas Library also requires some other packages and it will download those as well and it will basically all put it into this folder over here so that is basically what happens when you run a simple pip install it downloads the files from a GitHub repository and puts it onto your local drive so let's look at a few more examples to fully understand this so let's consider the scikit-learn library which you can install using pip install site alert then once I import that Library I can have a look at the finder over here and see basically what it's all about so here their top level we can see the scikit learn folder and in there are basically a bunch of other python files and also folders containing again other python files so coming back to the document over here basically when you load in or install that package you download many files including python files that Define the modules classes and functions and when Pip installs the package it places these files in appropriate directories on your drive so in my case that would be in the virtual environment or the conda environment I should say and then once we come back to our IDE for example our code editor and we load inside kitlearn we basically can import the modules classes and functions into our python script so basically when you run an import statement python looks for the corresponding dot Pi files on your drive reads the code in a file and creates new module objects in memory so what we are essentially doing over here is we are referencing to this folder over here and we basically tell python within that script that we have to load all of this information but usually you don't import the top level Library all the time so what often happens is that you import specific classes and now this would definitely be the case for scikit-learn so as I will show you in a bit for pandas for example it's pretty common to just import the whole Library so we just import pandas SPD but when when we are working with scikit-learn it is usually better to import specific classes and functions from this Library so what are all the differences between all the terms that I just described so we have files we have modules we have classes and we have functions and all of those work slightly different you use them in slightly different ways and scenarios so let's look at this again from the example of the scikit learn Library so the files so we already had a look at that so that is basically all the dot by file extensions that you see within the sklearn library that the modules so modules are a collection of related function classes and variables that are grouped together in a single file examples from scikit-learn are for example the pre-processing module that contains functions for data preprocessing tasks like scaling and normalization so let's have a quick look at pre-processing overhead so here we can see we have a folder which is called a module that has again python files and now what we can do if I for example come over here I can basically call the SK learn Library first then do a DOT and then do for example pre-processing and then we can import that whole module on its own so if I hover over here and I for example hit command and then look at it you can see that it is a module and you see description over here but I can also say that from this module I want to import a specific class instance in this example and that is for example the standard scalar so if I hover over here I can see that this is a class okay so this brings us to the next subject that is a class and a class is a blueprint for creating objects that have attributes variables and methods so for a lot of the terms that I'm using you will find out that people will use them interchangeably so methods functions are the same things attributes variables when it comes to a class are basically the same thing so keep that in mind so in the example over here so here is the example of a logistic regression class but we were looking specifically at the standard scalar class is if I go in here so this is the module you can see it over here in vs code and I can basically see everything that is in here so for example these standard scalar that is over here I can click through it and also find out where that is located so you see how in these files everything links and references each other basically and this is just Python's way of figuring out where everything is and how you can import them so we can have a look and so this is within the data dot by file and we were first in the init file so remember we went from standard scalar to the init file where the import is and then to the data file where we find the class over here and now we have also come to a point where working like this and looking through the classes and in a bit also the functions becomes really helpful because you can see that there is documentation in here so if you have a solid Library there is usually a good documentation string that basically describes first top level how a class works and then also going further into it even how the functions work but before we dive further into that let me also come back to the final topic and that is a function so functions are blocks of code that perform a specific task and can be called multiple types and then for example in the scikit-learn library we have metrics and we have the accuracy score so let me quickly import that so come back to the file over here and we say from Psychic learn and then metrics so two modules so first we have the dope level module which is basically the whole Library then we have the metrics and then what we can do is we can for example import mean squared error and now if I hover over here you can see this is a function and I can come in here and now I have the documentation over here of how this function works so I won't dive specifically into all the nuances when it comes to functions and classes that is a topic for another video but in general a function you define by using def you're probably aware of this if you have been working with python and data science for a while and a class is let me come back to for example the standard scalar class over here so you define a class simply by writing class and then you use camel case basically you capitalize every first letter of each word to be in line with the common python style guide so functions and classes and now where this gets really helpful is when you are actually start to work with these libraries so let me for example show you a model over here so let's import a random Forest regressor so instead of coming to Google over here and say like SK learn random forest and then take the regressor over here and then come to the documentation and then basically look up over here okay so it needs an estimator the Criterion and basically all the documentation everything that goes in there what you can now do is you can come over here and just click with command or control on this class and you can just scroll down and see basically everything that is on this page over here so you can see we have the Criterion you have the m estimators and you can usually if you scroll down you also have an example over here so I can literally come in here come back to my file and then just make sure to clean this up a little and we have an example so that is also something you typically find on a page over here but you can see how it's just the same and instead of switching constantly back and forth between your code editor your IDE and the documentation online it is already in here and specifically for the library that you're working with so when you are looking up stuff online you also have to be mindful not to like click this first hit and go to the random Force classifier now we're working with the random force regressor and you can't make that mistake if you just stay in your IDE so say you have this model you have the X and UI and you make predictions and now you want to calculate the mean squared error but you're thinking again okay what should go first and how should I name my variables you just come in here and it says Okay first it is y true and then it is why I predicted so not the other way around and then also you can see some examples over here so okay you can put a list in here or a numpy array and then it will work and like in the beginning of my data science career I would just work from Jupiter notebooks so I will do all my projects in here this that is also basically how I learned it in University but if you are just using jupyter notebooks within your browser you don't have the ability to do this so I can't I can't go in here now you can do that from vs code so that same notebook is within vs code and now I can click on that pandas library and go back so if you like to work with notebooks I would still recommend working with them in an IDE such as Visual Studio code and I know from experience that a lot of people work like this with just notebooks as well and you're just limiting yourself by not being able to look up the docs look up the documentation and constantly have to switch back and forth and find it out based basically like that and when you understand all of this you can also start to work within your own projects more dynamically so for example if I come to this new empty python file over here and I for example create a function over here so this is my awesome function and it will just do a simple print statement so let's save this in here and now I come back to my example and what I can now do in basically the same manner as I'm importing functions from the scikit-learn library for example I can say from functions import and now I can import my RSM function so see how now similar to how I would interact with the scikit-learn library I can also interact with my own files so as you can see understanding all of these Concepts so the files the modules the classes and the functions and how they work and interact with each other if you import them is really at the core of being an effective python programmer and like I've said when I started out I started working in single notebooks and just put all my code from top to bottom and that would be it and then later I transitioned to python files and then I would just write all of my code in one big file and I would of course use the import statements but I did not really know what was going on behind the scenes in the back end so I would just pip install pip install pip install and then look up the PIP libraries on the internet so go to the GitHub or the documentation and then learn how to work with them but as I've hopefully explained to you in this video there's a much better way to interact with your code all right and that's it for this video that is what I wanted to show you now if you found this video helpful then I would really appreciate it if you like this video subscribe to the channel That way you also let YouTube know that you want to see more content like this more data science content and that is of course helpful to you if you want to learn more about working with Fierce code and files and working like this basically then you should really check out this video next where I go through my entire setup and explain basically how you can work in a similar way as I do