Uber Drive Data Analysis | GeeksforGeeks

Key Takeaways

foreign [Music] [Music] [Music] foreign [Applause] [Music] right hello good evening everyone are you able to hear me can you just post in the comments here so that we can start with the session yep so hi uh welcome everyone uh thanks for joining the session yeah I'm doing good hope everyone is doing fine as well uh so I'll just introduce myself myself Rehan I'm the chief AI product architect uh at tapu which is a computer vision company here in Malaysia uh I am also the mentor for data science course here at geek for weeks so today's agenda is to discuss about data analysis so uh I hope you all are interested to pursue or you all are already pursuing career in data science right so so for data science if we talk about machine learning deep learning or computer vision right or so there the data is the key so so we'll in today's session we'll be seeing that okay how we can uh see the the data that we have we can do the analysis we can uh clean the data right we can remove some of the bad data and before we go for the actual applying those algorithms for machine learning deep learning or computer vision so this is a very important aspect which generally even the experienced people they are ignoring it so yeah let's get started I'll share with you the data set so if you want you can uh do the analysis along with me just I'll get the link Okay so so this CSV file is for the Uber Drive data right sure sure so let me share my screen and I'll show you the data so if there are any questions yeah you can keep it so this is a one hour webinar right you can post a session in the comments right and at the end of the session we will try to take on those questions uh and if there are any other questions after the session you can just post the comments on the YouTube video right we can have a look and reply uh after the session so is everyone able to see my screen right okay you can post in the comment if you're able to see the screen okay uh so this is a Uber Drive data for a particular user right so what do we have here we have the star date and date uh category uh and then we have the so category means which type of ride is it is it personal or is it business right right and then the this is the start destination from where you're taking a ride this is a stop destination uh so yeah we're the cap is dropping you this is the distance right and this is the purpose why you are traveling right so we have the data now we'll see that okay what are the different analysis that we can make from it okay yeah so data set has been shared uh Amit in the chat above I'll try to share it again yep okay so so just before uh plunging into the code right so let's have a brief background right so this was the CSV file which I showed you right so CSV file is comma separated file which I mean you're having all the data so now what we'll be doing we'll be using python we'll be using python libraries uh to do the analysis right so if you all are aware of those that's really good but if you are not you can just follow right and try to understand get the idea what we're trying to do and this thing you can apply to any data you can apply to labels uh uh that when you are training a model for deep learning you can apply to the labels there for the machine learning or you can apply to you can apply to Text data you can apply to emails data right for anything you can apply this concept so try to understand yeah how we are approaching the problem and after the session try any simple use case and see if this is what you can apply uh so first of all we'll be starting with importing the libraries right so numpy you already know I mean it's a uh Library uh for the arrays right uh pandas is uh python library for the data analysis manipulation right so you can do this with normal python also without these libraries but it would require you to like write a lot of lines of code with these libraries you have functions readily available with which you can do the data manipulation easily mat.lib and uh so this is for plotting the graph charts all those things C bone is also for plotting the charts but it is for plotting the uh Rich charge more advanced charts if you want to do and calendar is for the date time manipulation if you want to do uh yeah OS you might be only aware of like it's for like if you want to uh copy the files move and all correct you can use this so what will be the first step after loading after you import the libraries I'll just have a look at the comments if there are any okay so uh First Step will be to load the data set right so this is a data frame so what is the data Frame data frame is a two Dimension array right which we have uh in which you have rows and columns right so it is similar to SQL table right so there you have the rows and the columns right but here I mean it is in the in memory you're trying to do it so first of all you will load the whole CSV file inside this data frame so it will like just you how you load the value inside a variable right so similar way you are loading the whole CSV file into this so it will become a two dimensional array so how do we do that so this is bundles right so pandas YouTube the Alias PD dot read CSV and then you give the CSV file location right so for miles here I mean it is in the current directory so I'm reading this and then DF dot head so what DF dot head will do it will load all the data and DF dot head will just display the first five rows right so that you can know that okay if the load has been done properly and you can see the data and all right everybody clear till here okay we'll move ahead uh then you have df.tail so DF again to remind what we have we have the CSV data right dm.trail will give you the last five rows so this you can know that okay your data you can see the first records in the last record and you can have an idea okay what is there and your data right uh and then uh so we'll be going through uh various analysis and what what information we can see about the data right then you have DF dot shape right so it will give you uh okay what is uh the how many rows are there and how many columns right so one one five six rows are there and then these many columns are there then you have DF dot types it will give you the type of each column right so start date and date all the type is object miles which is the distance then float right and all the others are the object types so you can see here uh six category and one numerical right okay now uh the question is uh we want to see how much so you have the CSV file right so there might be many columns many uh values which might be null right so we need to check how many null values are there in the data because if you have the null value right it will affect uh it will affect the analysis so yes uh the uh the data set link is already provided so yeah I think again so maybe I'll write I wrote a heading also so now it should be clear okay so for seeing what values are null right so this was a data frame in which we have loaded all the whole CSV file and here I mean you use this is n a so is n a or is null is the same thing right so not available right so this is used to see the null values it is having right and then dot sum what it will do for each column how many null values we have right so EMC PLC started zero null values and date one category and if you'll see purpose it has 503 null values right uh and then if you want to uh zero down right which uh row is uh null right so then you can do the data frame and then so this is the nested right so if you want to see right what is null then you write okay column and then is n a right and uh if you like have that in the nested in the data frame you want to see this row right then you just list it here in the reader frame and you can see the exact record which is the null one right so now suppose if you have null values right in your data set and I I want to say that okay you have to calculate what is the average what is the uh like what is the most frequent date when you're traveling so if there are if you are doing the average and if Suppose there are 1000 values right and if you and there are 20 nulls values and you are doing the like 100 null value suppose and you're doing the average then I mean the total number of values that you have right I mean it is still uh so it is still uh uh 900 only not null right but average you'll be doing divided by thousand right so it will affect your average so that's why it is better that you remove those null values right so now your data set should be like 100 null value should be removed you have only 900 data which is not them and then you do the average that will be the true average okay let's go ahead yeah so how do we remove the null values right so you can drop the rows right which have the null values so DF I hope everybody might be clear till now so DF is which is holding all the CSV values here right so it is a two Dimension array which stores all my rows and columns for the data set okay then I use df.drop df.drop will be dropping whatever rows I want to drop or columns and okay how do I first of all I need to figure out okay what I need to drop so here DF and date so I am selecting what I'm selecting and date column right and then what I'm saying is n a if it is null right and then dot index and access equal to 0 right so what do we mean by x is 0 so x is 0 is means that I I want to refer to my rows if I want to refer to the columns I will write X is equal to 1. right in place equal to true so in place equal to 2 it will modify the data frame here right if I am writing in place equal to false it will not change the rate of frame right so suppose if you are making a copy of some variable and then you are doing any changes it will not change the original variable right but when if you if you just do the assignment to the original variable it will change right so similar thing if you do in place it will change the data frame so what you're doing for the row for the rows for which end date is null you are dropping all those rows right you did the drop and then you have uh you did again so hold recall the reader frame in which you had the whole Uber record you again did data frame is any is any will give you all the null values dot sum will be giving you the sum so now if you did you see now my end date and date now it is the end is removed right and you can see yeah okay yes it's here till here any questions anyone has I can yeah if if anyone is having any doubt you can just uh ping in the comments here else yeah we can keep on moving ahead okay foreign yeah yeah I want to see uh the the metadata right uh I want to see like the information about my data frame right so what do I use I use dm.info right and then I can see okay what are the columns available so start date and date right how many not null are there double one double five and what is the data type of these right similarly if you'll go here you'll see for purpose we have 502 null right and there's six five three not numbers so then you can think of okay how much null are there and what you need to do and you can see memory usages 72 KBs okay so now if You observe right so um more than 55 percent for purpose column it is null right so it makes sense that we drop this column because if it is having null values right it will not help us in the analysis right so how we do the drop right what we saw earlier uh d f dot drop which column purpose and X is equal to one so last time what we did X is equal to 0 we did for the rows now we want to drop a column so we here we do X is equal to 1 in place equal to true that it will modify the data frame right and then uh DF dot head to it will give me top two records so first two records I got it and you can see the purpose column is dropped yes okay now we have already uh seen the information about the CSV data we have we saw that how many null values are there we saw top 10 record top five records uh bottom five records we dropped the row we dropped the column right why we are doing this we are cleaning our data right so whatever insights or whatever analysis we do it is not biased or it is not affected so suppose you have thousand records and suppose all 900 rnl right and now you're doing the like analysis so then like those null values would affect the remaining 100 also right percentage pressure on it will be really bad so now you want to check for duplicate data right so there might be instances in which your CSV file is having duplicate data how do you do that so you'll do DF or duplicate it right and then data frame again in the indices if you'll place that you will get the exact record which is duplicated right so here I can see that okay 492 this row I mean it is to decade right so what you can do you can just drop this row uh okay so may I know uh if anybody can write the statement for dropping the duplicated row I can give you the statement how you drop a normal row I can give you here okay so this is to drop a particular row where end date is uh and then so now the question is that to drop okay I'll wait for two minutes and in the meantime if anybody has any query they can post no problem can we replace the null data value of purpose column with unavailable or something and continue yeah sure you can do that also yeah so there is also something called that you have you calculate the average and then you can also replace the null values with those average values right so that it's not affected so there are numerous methods that you can try to apply anyone who can post uh how you can drop a duplicated row attempt at least okay I think time's up okay so okay dropping uh the duplicate row right so we saw that how do we calculate the duplicate rule so DF dot duplicated right it will give me the duplicated row and if I want to display that right then DF then this square brackets and I need to write that so it's the same statement right what I have written for we have gone through for the for dropping the end date null we just replace that DF end date is n a with DF duplicated right so it's so here D after drop I will write to drop the rows right or the columns and then here which rows I want to draw it will be DF dot duplicated uh and then X is equal to 0 for the rows is equal to 1 is for columns and then in place equal to true so it will modify the data framework so if you write false it will just display here but it will not modify this data frame right in which and then you can display so now if you see now we don't have any duplicate data right okay so we earlier saw that the date and that the end date and the start date they were not the uh they were object type right and now we want to convert them into date time so may I know uh the reason why we want to convert into date time we can let it remain in object also right so that also we can do the analysis but menu why we are converting them to date time from object anyone can post the comment two minutes yeah I'll write the question here why are we yep yep great so it would be easy if you convert them to a date time right so then you can apply those daytime functions and you can do a lot of operations like if you want to calculate the difference right or any more operations also you can try to do that yeah okay so now how do we convert that right so uh okay PD dot so this is the pandas Alias that we are using two date times so this function is used to convert to date time right then on the left hand side first argument will be uh they started right so this DF and this is the column started so this it will be for this column I want to convert and what format I want to convert it it should be in form of this month day year hour minute right so you can express any other format also right ah so similarly for the end date you can convert it like this right and now you can see if you see the df.d types right so DF dot d types will return you the data type so if you see here you can see earlier it was object right now it is late time right everybody clear on this okay uh let's move ahead with the analysis part right uh so this we are going to do the Eda so Eda is explore exploratory data analysis so in simple words uh having Data Insights right so even before applying any uh machine learning or deep learning algorithms you can do some analysis using the data right because you really need to understand your data well before you go for applying those algorithms and experience with me while I'm working on the case studies here at my company or when I'm taking the interviews uh people are lacking uh jumping into applying any particular algorithms right but maybe I mean that is not required right so I'll give you an example uh we had the case use case of uh classifying different cylinders so you have the LPG cylinders right so you have to classify them into different classes so the class category was a b and c uh so a means uh uh that I mean it is good condition right c means the condition of the paint is very bad uh b means that it is moderate right so now uh first of all you need to see okay what uh like how is the like what other what is the data right how are the images how you can differentiate between this a b and c uh and then what algorithm would you want to use for it right so I mean I would not go into details because many of you might not be aware of the different um algorithms right so different uh so you can use different approaches for this right uh what you can do you can look at the whole cylinder and you can see if the paint is on what area is print present or not then you can label it as that or what you can do you can just uh the area where the paint is worn out right you can just label that area not the whole cylinder and then say that okay this is the one note paint right uh have an other like uh different approach can be that you don't label the cylinder or you don't label that OneNote area but you just label the whole image as something right so now what will happen that I mean if you are not labeling the cylinder right so then how do you first of all for saying that okay this the paint is not there you need to identify if it is a cylinder right so uh and then if you just okay label that part that okay where the paint is right so then uh there might be different it is regular patterns of paints and wants so that when you are labeling that particular one out area it can be quite confusing right so so I mean this is I'm just telling you the only the first step that where the data analysis is very important to see and okay after the labelers have done the labeling right and now you want to suppose do the training so in the model training we have broadly three parts four parts you can see one is the labeling of the data uh then is the data preparation or for organizing the data so that it can be trained by the model so that is called the data preparation part and then is uh the training you train the model right and also it involves the tuning the parameters so different parameters for training a model so that you can train the models from the best band and then the last you have you evaluate the model okay so there are certain cases in which you give the data you train the model and you get the results but and the uh and the results if you see the model is not performing good right so then I mean you check the model configuration or everything is fine but then you come to know the issue that okay the labeling what you have done that is wrong right so a b and c you have done some mislabeling some of the part you have you forgot to do the labeling right or there was one person saying this is a the other person saying this is C right so there is ambiguity so these different things you can save from those things if you do the proper data analysis and you do the proper Eda so Eda means so when you're training the model right so you can have uh so you need to have a balanced data set for all these classes in which you are so suppose you are training for cats and dogs right so if you have 100 images of cats and you have only 10 images of dogs right so your model is biased towards identifying cats more than the dogs so if anything uh the model is like having the confusion right it will be giving weightage as cats right so how do you so if you do the Eda right what we are going to do now you will be able to know your data actually right that okay what is the distribution of different classes how many null values are there how many data is not labeled right so if you take cat images your model is performing good what you did you took cat images at night and your model is struggling there right so now you are balancing the cat images in different environment condition also so I mean yeah this is the part when we do EDM okay we'll go ahead so univariate can anyone tell what is univariate Okay so uh yeah now we want to know right uh so we'll be doing we'll be starting with analysis by like different different columns right so now I want to do the analysis for category right DF categories dot unique right I want to know what are the different values which category columns has unique values right so you you can have different right it can be business or personal business or personal right but what are the unique value which you have for category so if you do this right you'll be getting the result is business and personal so you have two values only for this category there are two right categories right business for work personally personal travel right thank you yeah now what you want to do you want to do the calculation that you want to calculate the total number of miles right total number of distance for business and for uh personal right how do you do that okay first of all which columns you're working on category and Miles right so if you have studied like worked on SQL it is same very much similar to that what do you do you Group by this data in these two columns by category right and then you apply the aggregation right what will be the aggregation some for Miles column right for the miles data so this is the step right first part you select what columns you want to apply second is how you need to group by right like you need to when you're calculating the sum right aggregation how you want to so I want to group my business and personal okay and then what operation I'll be performing I'll be performing aggregation sum of all the categories right all the business and all the personal categories so it will be doing this some aggregation of all the values for this right so we have only two values it will give me this now you saw that okay business is 11k right and percent is 700 so this means you're taking more uh right for the business purpose right okay now let's see how we can plot the plot this in a bar graph right so this was matplotlib right so I mean this is used to plot the charts so you've plotted the figure and then again what you did same thing right you did the same thing here what you did earlier this columns you took category miles you group them by category and you have applied the sum on those different categories some for all the business some for all the personal and then what you did you plotted a Dot Plot and what is what is the graph it is a bar graph right okay you have on the x-axis you have the category right you have the category on the y-axis you have the total miles right and the title is total miles per category all right and here is the same thing what was there in a data table you have it in the graph okay so what is analysis from here that you that the user is using it more for the business trips and rather be in the personal trip right okay moving ahead uh so I want to see okay what is how many uh how many unique uh start uh location you are having it is 177 you need to start starting points and then I want to see what are the top 10 starting places right so then what you did so uh so it is DF start right so you write the column name right and then you write a unique and then you calculate the length right so you'll be getting how many unique elements are there okay uh top 10 so what you did TF start so for this call I want to count all the values right and it should be in descending order right and then I want the 10 values so for 10 for this I write this colon 10 it will give me the first 10 values so so here this means it will start from the first one 10 is the 10 values it will give you right so you can see here the maximum starting is for carry right that is unknown location and then so now the same thing if you want to plot it right so what we do we use the same thing a start column right you want to count and then the first 10 values right and then you are plotting it so for plotting the parameters are okay same so bars so this bar H is the horizontal so this was a vertical bar graph right and this is a bar Edge is a horizontal then you have the Y label places and the X label is pickup count right and then you have title is top 10 pick up places okay so okay now uh can you host uh the unique places for stop and the top 10 stopping places can you post that okay so since time is already going to be up so just one minute if you can tell me how many uh unique stopping points are there if you can just write the statement and just post in the comment foreign in which we have all the Uber data a stop is the column right so I'll just go through the data frame again in case yeah so you can see I mean these are my columns and this asterisk is the column name right so don't confuse with uh wildcard or something right this is a column name part of column name okay so DF square bracket uh start I'll get all this DF square bracket stop I'll get all this right okay I'll check this okay so I have 188 unique stopping points then how do I do the counting it's same thing only the change will be instead of a start it will be strong right so you can just plot it okay so you are starting maximum at carry and then you are stopping maximum at carry right so this means uh this is like a like his office right yeah uh so maybe the home right or the or the workplace so okay this is I mean you can do further analysis that let me see what's the time okay you can do further analysis that uh if the starting um if you want to see okay how many unknown locations are there right so same I mean you can do that okay if if like DF start equal to equal to unknown location right and then you want to count all those instances you'll be getting 148 right so yeah try to so here you are comparing this this DF installed to the unknown right and then this whole thing right you are counting the for the start column you are counting it right with values count then you're getting the answer okay similar way you can plot the total miles but you can see here I mean this is not giving you any good analysis right because I mean normally the distance you are covering is in between 0 to 50 right so you're not covering 300 miles or something so this analysis was not useful for us right so it will not be like that all the analysis will be useful for you okay now uh so we saw DF uh we saw the info right how we can see the info now this is ds2.describe.p so what it will give you it will do it will give you some useful uh calculation right so it will give you okay uh what is the mean right so but this is valid only for the numerical data right so it will give you all the mean the count the standard deviation right the minimum value for the right you have uh for the distance is 0.5 the maximum value is 310 if this is 25 percentile this is 58 and this is 75 percentile right so this is describe DF dot describe dot T it is Val valid only for the numerical data not for the uh text right because for the text you can't just create meaning and mean okay so we saw uh univariate analysis right now we have the multivariate so univariate means you are doing it for one variable multivariate is you are doing it for multiple varying so DF dot head you get the first five records right now what you want to do you want to group by so earlier we were doing Group by like one column site but now here we are doing Group by two columns a start stop and then the miles you can see I mean this is the like the values that are printed here right so it is being grouped by start and stop right and the total miles are being displayed here right so total miles and then you can see so many of these are the destination so let me scroll down okay to make it more cleaner right so here's the data I mean you are grouping by start in the stop right start in the stop and then what you're doing uh for the column miles you're grouping by start in the Stop and you're doing this some right so here you can see here you are seeing the data from 1 to 11 right and then so from if this is the start and this is the stop it is summing all the distances for this uh starting and the destination points okay so this is just to show that you can do univariate analysis you can do multivariate analysis okay uh now we want to see how many round trip the user has done right we'll just do this and I think then we'll uh is there a library you can use for autofill browser for my password manager in Python okay we'll just come to it we'll just finish this part and then we'll come to question answers q a sessions uh okay so how do you see how many round tip you have done right so you you try to function right in which you see that if the start and the stop is same it will return Yes else it will return no right then what you do I mean you just apply this function right so DF dot apply what you can do you can apply this function right for the round trip and then you can yeah so here what you're doing I mean in this function it will be so it is for columns right X is equal to one is for columns and what it is doing it is checking uh that uh starting the service name right and then you do a plot where your x-axis is the round trip right how many round trips you have right and then uh you are having uh this is the data right on the data frame so this uh on the y-axis you have what is the count of this right and what are the values that you're having for this round trip you are having either ES or either no so if you notice right we have added a new column into our data frame which is round trip so if you want to add a new column just write the if and then write the new column name and it will it will add that column automatically and what that column values contains right you can see here so DF round trip asterisk this is the column name value count so you can see uh yes is 288 no is 86x so this means uh around 75 a single trip and 25 in the amount right so you can do lot more analysis you can calculate the right duration for that we have done the uh conversion to date time right you can do lot more analysis using this so this was just to give you a brief idea right you can use it I think now we can yeah go for Q a session before we wrap this up so addict I think this will be um like we can try to do this on the client side right on the server side um like uh we should not be doing that so for the pipe yeah yeah anyone else has any questions regarding the data analysis or uh regarding uh data science [Music] um also let me post you the the different courses right which you can look into yeah so here are these list of courses right which you can go through or data science yeah I'll wait for uh two minutes for any questions or clarification on what we have covered so far yeah how to generate reports of analysis to post uh what do you mean by reports is it is it this charts and the graphs or or do you mean to say anything else because if you're talking about like having a rich reports right so I mean one thing is that you can use the this matplotlab and c bond right for the analysis and you can get this kind of a chart but if you want more [Music] Advanced Tool uh I mean you can use power bi right in which you can get more powerful reports right and the pie charts graphs data tables so I'll write down the name so it is by Microsoft so these are actually the data analysis tools so where what you need to do you just need to send them the data and they and you will you also need to like have the standard format in which you want those charts and the tables and then yeah it will be available for you for the visualization so there are a lot of data visualization tools available any other question anyone on boom okay I mean if yeah there are no other questions yeah then we can uh do you want like we should be having uh further analysis or if you want me to share on anything else okay so I have posted down the link uh for the give for me courses if you're interested you can explore that right and thanks a lot for joining uh today's webinar who I mean this was useful and you'd be taking a step ahead using this data analysis for whichever project you're doing right and you can explore it in like deep learning machine learning computer vision yeah sure please go ahead you Raj yeah power bi questions the power base is a visualization tool right there you just need to uh uh like import the data and you can see the different graphs and the charts and all right I think what anshuman was asking regarding like yeah what uh how like we can generate the reports and all right so our bi is a tool by Microsoft yeah yeah so you just give the data right and the format in which we want the data to be spread there are a lot of other asked in power bi for the interview so I mean you want to know what are the interview questions asked for data visualization and all I mean you need to know that okay um like how you can import the data right and uh if like different analysis you want to do how you can do it in power bi how you can calculate in what format the data will be displayed right so I mean yeah these no no I mean they will not have scored as a bar chart and all uh yeah I mean they will ask you that yeah how you can actually do that right in so suppose in I have a this data right for the Uber data right I want to see that okay how much uh the business and the [Music] personal trips are there right so then I mean like how do I visualize this in uh pie chart or a bar graph in power bi right so I mean this can be like one of the questions so mainly on the application so if you want to do try out a small project in which you can try to use power bi for yeah sure please go ahead yeah foreign ER to do the visualization so you send the data to elastic and then you can just visualize your Theta then you just need to select okay what are the rows and the columns sorry what is the X and Y axis right and then what data you want to use do they ask Advanced skill carries in data analysis so yeah a good good question Raj so yeah you need to be good having good knowledge of SQL databases right for because uh for uh data science you need to have uh so that I mean you can filter store retrieved in effective manner so I would say a decent skill queries right uh yeah I mean you should be knowing how you can do aggregation how you can do grouping how you can sort it right so okay guys yeah thanks a lot for your questions uh and thanks for joining oh yeah have a nice Sunday evening and uh yeah please post if there are any other questions in the comments yeah we'll try to reach out okay thank you how to create a data analysis resume okay sure basically uh like the key is that you select any you select any project right you to do that use case you can you can refer to kaggle right you can refer to geek for geeks you can select any project you can try to like create a small project post it on GitHub right and then you can put that in the resume so that will be really useful and uh got it got it okay I'll you can I'll you can also reach out in the course section you can also reach out to the contact and questions yeah we can answer there and in the YouTube comment also you can post so that yeah we can ask certain questions there okay thanks everyone uh yeah nice having you bye