How to quickly explore data in Python using the D-Tale library

Key Takeaways

in this video i'm going to show you how to perform exploratory data analysis or eda in python using the detail library and we're starting right now so the detailed library will allow you to quickly perform data pre-processing and also exploratory data analysis right inside the web browser using a jupyter notebook or even a google collab and so let's dive in so i'm going to provide you the links where you could download this particular collab notebook or jupyter notebook so the first thing that you want to do is to install the detail library so let's do that using pip install all right so now it's installed now let's start by loading up a data set and so we're going to start by importing pandas as pd and then we're going to create a df variable where we will be assigning a data set where we download from the github of data professor and it's going to be the solubility data set and then we're using the read csv function to read the csv data and let's run this and now we're going to be using detail so for one we're going to import detail which we will be using for showing the data frame via the detail interface and in order to allow this to work right inside the google code lab we will be needing this particular line so if you're using a jupiter notebook on your own local computer you could feel free to comment this line otherwise if you're using a google codelab then you want to run this line as well so we're going to run this cell and there you go so this particular line use ng rock will allow us to generate this url let's click on it and so the magic will happen here and so this is the interface of the detail and on the top left hand panel here you're going to see the numbers and what does the number here represent so the number in this area here on the right hand side is going to be the number of columns five and so there are five columns and the number here 1144 represents the number of rows because we have 1144 rows and if you hover your mouse at the top you're going to see the menu bar and also for the play button here if you click on it you could have access to all of the various functionality that you could do with this data frame so you could think of this as kind of like a gui or a graphical user interface for your pandas data frame so a powerful feature here is that you could perform eda for example let's click on the correlations here and within a few seconds you're going to see this heat map of the intercorrelation matrix so you're going to see how each of the variables here are intercorrelated and so right off the bat here we're going to see that there is a negative correlation between log s and the moloch p and the great thing about this is that the correlation is also color coded as well where negative correlation will be red and positive correlation will be green and then at the bottom here you're going to see that there is a scatter plot between the variables of your choice right here which is shown here is the molar p versus the molecular weights and so you could select the particular columns here and only the correlation between those are shown and at the top right hand panel here you can click on the code export and then you will have access to the code that you could use in a future run meaning that you don't have to rely on this gui for reproducibility you could also copy the code and paste it into your code cell of the jupyter notebook so let's try that let's copy this code let's go back to the collab paste the code and let's run it let's copy this correlation data okay so this is the underlying data of the intercorrelation matrix so the underlying code is generated for you automatically let's go back to the interface let's click on the action here so you could also see that there are functionalities for you to have a look at the missing data in your data sets and so here it is using the missing number library and so we see that there are no missing data so let me close this and let's click on the particular column that we want to analyze and then let's click on describe and so here are you going to be able to have some descriptive statistics so for those of you who wants to perform eda here you are you could quickly have a look at the total number of rows the percent missing data and here there are none and the top frequency of values are shown here as well as the various quartiles the maximum value the mean value median minimum number of unique values in this particular column the standard deviation the kurtosis the sku and also this box plot here and to generate this you could click on code exports and you can copy the code let's click on it again let's have a look at the variance report so this variance report will tell us that this particular variable does not have low variance and shown here is a histogram plot along with the kde so this report will allow you to determine the relative variance of this particular column so here we could see that out of the total of 1144 there are 931 unique values and the most common value is 1.41 and also 7.27 and there are occurrences of seven and as always you could click on the code export here to get the code or you could also modify the number of bins here and then we get the updated plot and the great thing is that the plot is also interactive you can hover your mouse onto a bar and then you see the respective number of frequency all right and so let's have a look at another data set with more data so let me comment the solubility data out and i'm going to uncomment the acetylcholine estuaries inhibitors datasets let's run it let's run it and let's run the interface again and let's click on the link and it will generate another instance for us and so let me close the previous one so for this particular data set there are 882 columns 4 695 rows as you can see here this data set is comprising of binary values zero and ones so we could expect that there will be several variables or several columns that would have low variance meaning that most of the value will be constant and have no variability and so we could do that manually or also programmatically but then that would require some time to write the code but then you could instantaneously click and have a look let me show you just click on the play button and then click on the low variance flag and so you can see here that it literally gives you a red flag and also highlights the particular column in pink color here and so these columns have low variance and so typically we would delete these columns because they provide no information owing to the low variance let's scroll through here so we can see that there are quite a few number of columns with low variance let's have a look at the heat map again of the correlations okay so because they have low variance it is quite impossible to calculate the values here so ideally we would have to delete those columns but also we could just take a quick look and so you're going to see that they're impossible to calculate so i'm going to show you in another video how we could remove those columns having low variance let's have a look at the action button again let's highlight the d types highlight by data type so they have the same data type so they're all in the same color let's use another data set let's comment this one let me add penguins here and let's run it again and let's click on the link and then the new interface is loaded up and as always here you can see that there are seven columns and 344 rows and so you notice here that there are some missing values so the data is not clean like the one shown previously let's try out the other functionality on the missing value let's have highlight missing so you can see here at a glance that the yellow color are the highlighted missing values and then you can scroll down so the orange color that you see is indicating that it is currently loading the data and the blue is the highlighted one so yeah there are quite a few missing values here here as well 339 let's click on highlight data types so numbers are highlighted while the categorical are not highlighted let's have a look at the correlations let's try out another function how about low variance flag so apparently there's no low variance here let's click on the column and let's have a look at the describe so these allow you to perform quick descriptive statistics or value counts and so you could quickly see the relative frequency each of the value of this particular column even for this one as well for the island so you can see that disco accounted for the most frequency about the gender so roughly similar but a little bit more male penguins here 168 versus 165 okay so you can see here that there are quite a lot of functionality that you could perform with the detail interface and so let me know in the comments which particular function do you like the most and if you're finding value in this video please smash the like button subscribe if you haven't already and also make sure to hit on the notification bell so that you will be notified of the next video and as always the best way to learn data science is to do data science and please enjoy the journey