Analyzing Data with SQL & Python | How to Analyze Survey Data

Key Takeaways

uh so uh make sure you got this uh open and uh we're going to start off going through you've got links to the paper that um describes the data set and the data set itself if you want to go away and uh read more about this um we're going to start off uh by uh looking at uh questions with a numeric response and visualizing those with a histogram uh we're going to do uh a little bit of stats for the manual EU test and then we're going to look at uh visualizing questions which have categorical responses using par plots right so the first thing we need H to use some uh python packages now because we're doing all the data analysis in SQL no python package needed for that we're just going to use the plotly Express package uh that's for drawing our histograms and bar plots uh we also need the manwin EU function from c. stats that's for performing the hypothesis test so uh first thing let's get started we're going to do import plot the Express and the common Alias for this is PX and next we're going to uh uh import The Man U function from scipi stats I'm copy and pasting here because I'm a terrible typist and uh better to not make mistakes uh so it's not auto complete for me okay copy and paste then so here we've got a whole package and then just one other function hit play to run that so those are correctly imported now the next task we're going to import the two data sets we're going to um if we have a look at the files uh the survey uh responses these are contained in the survey data.csv file and then we've also got a data dictionary in survey uncore questions we're going to put the data first then the questions uh oh this is actually out of date uh I shortened the name of this file because it was horrendously long so the survey data is actually contained in a CSV file named survey uncore data and the details of the columns basically we've got one column of zeros and ones which means is the current is the the company currently um classified as a growth company so there's a an oecd so this is like an international organization which Define gives you an official definition of what constitutes a growth company uh some of them are some of them aren't and then uh each one of the questions um questions are split into different parts I'm not not sure whether this like slightly dubiously translated from Finnish so question is like an overall set of questions really like a section and then row means like the individual question within that um okay so we got lots of questions and then the details of those are described in uh this data dictionary file so next task we're going to import the data set so we want to select everything from this file uh so we going do select start to select everything and then from uh I'm just going to show you what happens uh if we uh try and import the data set as is this is going to go wrong because the data set is in um a slightly different format to what we want so usually in workspace you can um just import uh data sets from a CSV file by in the from Clause you use um a string naming the file uh in this case um it's not correctly detected that you've got commas for decimal places so it thinks these are text columns instead of numeric columns so we're going to have to do a little bit of cleaning up in order to make this work uh there are some code hints here uh that explain how to do this uh and then they'll drop down so uh this is what we have to do is call uh duck DB that's the database that we're using uh and we have called read CSV Auto that should be a low case eight and we have to set some parameters so first of all uh the uh column delimiter it actually got this correct but I think we have to set it uh explicitly now so that's a semicolon uh the decimal separator so the thing that says this is a decimal point that is equal to uh that's a comma so that's these commas here and then also uh scrolling to the right a little we've got some missing uh some questions with missing values and these uh they've got a space in them so the value that we want to be that we want to consider as null the null string and null strp in this case is is a space so these three arguments together I'm going to run this again you can see now in question five uh Part 9 and 10 these missing values they're correctly denoted as null and then in question two where we've got numeric answers numeric responses you can see these are correctly uh described these are correctly numeric columns and they've got period as a decimal point all right so that's the data imported uh one thing you need to do is uh if it's not correctly copied for you is make sure that the the result of this is saved as a data frame named survey we'll be using this later uh next we're going to import the data dictionary so we're going to do something very similar so we're going to do select star from uh let me just uh copy and paste the name of this file now in this case uh this uses uh standard uh like uh conventions for CSV files um so that means each column is separate with the comma you've got a period for the decimal place separator things like that so we can choose default settings that means we don't need to explicitly call read CSV Auto we can just pass in uh the file name as a string okay so um if you notice it's not super helpful just having the name of the question as the column um as the column header so we need a look up here to determine B what does you got the column in the original data set and it tells you what was the question that was actually asked here so uh we can read like what what were the people asked if you want to know more about this um in the files there is a PDF here which um has all these sort of complete details uh in in in more depth all right so uh we have imported the uh the main data set we've imported the data dictionary and now we're going to start drawing some plots so first of all we're going to have a look at the numeric responses so you can see we have these first two questions these had numeric responses so that's what we're going to deal with now and we're going to take a look uh so basically this is the question was asked um so basically saying if the firm develops the way you'd like it to how much revenue would you receive how many employees would have five years ahead so it's thinking about like in the future how much do you think you're going to grow and there are two different metrics for that so we're going to consider the first part here about employee count and because we've got a lot of uh numeric responses we want to know what's the distribution of these responses and it's very natural to visualize this using a histogram okay so uh we're going to draw a histogram using the survey data so that um uh we should have uh got a a survey data frame available we're going to look at uh question two Row one and uh we're going to set an access label to make it a little bit clear what's going on so uh we're going to use the function px. histogram and the first argument to this is the data set and then on the xaxis we are um sing which column of this data should be be plotting which in this case is called question two row one transformed and that goes uh in quotes because it's a string and then we're going to set the access label to make it a little bit easier to um to look at actually before I do that I'll I'll just run it so you can see what this looks like so we got to um histogram here um and the XX is a bit mysterious because it's just got the name of the question so this is where we're going to override it so I'm going to uh add in this uh x-axis label now in the Cent you can see the uh the code pattern for this so we're going to use the labels argument and it takes a dictionary so this is a dictionary with the brazers and uh on the left hand side the the the key for this dictionary is going to be the variable name and then we want the text for the x-axis on the right side so I've put this in the question what we want and I copy and paste that in here and run this again you can see same plot but now we've got a more informative description of what's happening on the x-axis so let's just take a moment to actually look at the plot and see what it's showing us so you can see most the most common response is between zero and 100 100 so this is um the expected employee count in 5 years as a percentage available from last year so they're taking the current number of employees and saying in five years time do you think you're going have more or less employees and actually almost um what's that so 48 of them out of 120 so what's that that's 40% of them expect to have between Zer and 100% of the current number of employees so 40% of the companies think they're going to decrease their head count in uh in five years time and you've got nearly 30 so about a quarter of them think they're going to go between zero and sorry between 100 and 200% so they're going to be somewhere between the current uh number of employees and double the number of employees and then you've got a few very optimistic uh managers who think okay we're going to dramatically increase the number of so in this case uh between 1200 and 1300% so I think in 5 years time they're going to have like 12 or 13 times as many employees that's like we've taken a load of venture capital money and we're growing incredibly fast now one thing I'm not totally clear on there were four responses but have a negative um percentage employee count I'm not really sure what a negative count uh means in this in this situation sounds like they're just uh giv mov up completely or it might be some dodgy data there so this is worth uh maybe reading the paper and seeing what what do the authors think about this seeing if there's more information there otherwise uh there's some uh questions around what what the meaning what's the meaning of this all right so um that's pretty interesting but we also have this column uh where's the data set gone so at the start called growth firm so this this is the column I mentioned about is this firm currently considered to be um a growth company or not so we might expect that growth companies are going to be more optimistic about how fast they're going to grow over the next five years so if you're already growing fast you might hope that it's going to continue so it's very interesting to look at a difference uh in uh the values of this uh question two Row one column about employee growth depending like split by these two groups and if you've got um so what we can do is instead of drawing one histogram we can draw two histograms uh one for each of these groups and the way to do this is actually we're going to copy and paste this code uh copy that the command C and then we're going to paste it in here and we need need to add one extra argument um so let's add one more argument and the argument is called facet uncore row and we're going to set that to the column that we want to group things by which in this case is growth underscore burm so just noce capital g capital F this is case sensitive so let's run this and now you can see so these are the non-growth companies the ones that aren't currently growing so this is growth firm equals zero the ones where it is a growth firm that's growth firm equals one and actually I have to say just eyeballing this I can't see a big difference between the two it looks pretty similar to me um there's maybe a bigger percentage of um companies in this sort of area where they're going between um 100 and 200% and then between 200 and uh 400% so there do seem to be for these growth companies there seem to be more of those managers thinking that they're going to either double triple or quadruple the number of employees over five years compared to those aren't but otherwise there's not an awful lot of difference appearing like from this U histogram um but just to be sure we we're going to do some statistics to see if there is some kind of difference uh actually before we get to that you're going to practice drawing some histograms yourself so I'm just going to give you uh a minute or two to do this um and you can do the same histogram again this time you're going to be looking at question two part two uh so it's this plot uh but change the question and draw it for yourself I'm going to give you um one minute to have a go of that then we'll go through the the answers as Reese while people are doing that have we got any questions yet uh no questions just yet um but yeah if anyone does have any questions with anything to do with code let us know in the comments and we will get those answered absolutely happy to take all of your questions um all right so uh let's go through the risk through the answers for this so again the easiest thing to do when doing two similar things is uh copy and paste so I'm going to take the last histogram that we drew uh I'm paste it in here uh got someone called Luke in my uh workspace uh actually and go as well so uh re should have posted a link in the chat so you can get your own copy of the workspace you don't need to uh jump into mine uh you can have you can have your very own workspace to code along in uh Reese if you want to post the link in the chat again okay so um we've got the code for plotting question to row one going to change it to question to row two so this is a fairly straightforward transformation we got to change it in two places so I'm just changing the name of the column that's considered in these two places and the other thing is uh the title is now wrong we need to change uh employee count to revenue okay let's run this again and this case again is a a little bit hard to see so there's one really really optimistic manager who thinks that actually uh the revenue growth is going to be um what's that uh 150 times what it currently is um so that's tremendous that strikes me as perhaps it's someone at a startup um where they've gone from having a free product to they've just introduced their premium their first premium product so they're going from basically nothing to some kind of money because in five years growing your Revenue 150 times that doesn't happen often uh but you can see most managers are a lot more modest so if they're not a Growth Company you've got almost everyone saying well actually it's between zero and five times and then again that's the the most common uh for the growth firms as well but you got a few more people where they're a little bit more optimistic all right so this is a fairly interesting um plot to play around with because because of this outlier what you might want to do is say let's have a look in more depth uh at at the sort of um lower end so because it's plotly we can zoom in a bit uh so let's Zoom that work uh okay let's zoom in uh all right the panning is actually not amazing uh in this it's just showing us a little bit closer um one thing that might help is to go if you want to play around with this further is to uh try and edit the axis range uh so the x-axis you're probably want going to want to look at like between zero and where does it stop being interesting around 2,000 um and that's going to give you a little bit more insight as to what most people are thinking without just worrying about like well what does this like super optimistic manager think so that's something for you to carry on with and play around with later all right so one thing I mentioned before is that um those two histograms when we looking at the employee account look really really similar one thing you might find though is that uh there is a statistically significant difference between the two groups so the growth the the managers who are working at growth companies might have slightly different opinions from uh the the manager non-growth companies and statistics can pick this up so um if we had normally distributed data we' use a t test but they the data is very very skewed you can see okay most people have like a a number somewhere around here and then you got this right skew so a few extreme outliers on the right hand side of the plot so because it's really not normally distributed data we can use a Man U test this is also sometimes called the Willcox and rank some test and it's very confusing basically there were three statisticians man Whitney and wil Cox and all working on this sort of uh separately and so multiple names for the same thing all right so uh before we get to doing the test we've got to write some more SQL to pull out the data so uh to begin with uh we are going to uh select can't select select uh this uh column of Interest the question two row on this is the one about employee count and we're going to select it from the survey uh CSV so um we're not looking because we're writing SQL we're not getting it from a data frame again we're going to have to copy and paste our previous code uh on uh how to uh where are we uh on importing the data set so all the way back up here right at the start you got the read CSV auto scroll back down to where we are okay so we're reading the survey data again and this time we need an additional wear Clause because we don't want all the rows we just want the rows where um growth firm is zero let's run this you see before there were 120 rows now already got 58 um we're can do the same again for uh getting the the ones actually are growth firms oh just um double check in case it's uh different for you make sure that uh you've got the results of this is q21 non growth it's just written in the uh in the instructions there so now we're going to do the same again but this time finding the growth firms uh so easiest thing to do is we're going to copy and paste and then the thing we need to change we want the cases where growth firm is one so to find the growth companies just double check that you got the right value uh for uh what this being to what this being returned as you want a data frame Q2 uncore onecore growth I'm going to run this and we got 62 rows so we had 58 before plus 62 makes 120 that's the whole data set that's a good stand to check that we've actually got the right uh amount of data and then in order to perform the man Whitney U test we've got two separate data sets so we're going to call man Whitney you think that's got no typos I'm going to pass in these two variables so what this is going to do is determine uh whether there is a statistically significant difference between the uh average expectations of employee count growth between the two companies sorry between the two groups and the thing you want to look at here is the P value so we've got a P value of not. no uh sorry .88 so it's a pretty small number um so in that case it suggests that there's a very high probability that there is some kind of difference between the expectations of growth from the managers uh between the non-growth and the growth companies I've say this is a little unsatisfying now I included this task because this is uh one of the things they do in the paper but I got to this point I was like okay so we know there's a difference but we don't know how much that difference is is it important why do we care about this and so it's leaving me leaving me haging a bit I'm like well you know the uh what I want to do more um so this is something that again is something you might want to look at uh in more depth you might to know well how big is there how big is this difference between the two groups so for example you could calculate what is the average or uh either as a mean or as a median any might do some some more sort of sophisticated modeling you might want to do some kind of regression to see well um what affects the difference in perceptions uh do some the responses to some of the other questions contribute to this so there are all sorts of questions that you can ask and I make it clear that just running this hypothesis test it's not really an answer as such it's more something to annoy you into um asking more questions and then doing more analysis so um you're not done after one hypothesis test it's really just the start now hopefully I've annoyed you enough to make you want to continue working with this data set uh and go and uh continue uh doing some analysis all right so um I just want to give you a little taste of Statistics um if you're interested in this sort of thing uh the hypothesis testing in Python course does have a lot more of this sort of stuff so you can find out uh a bit more about like what you can do with uh these different tests all right uh from that uh we're going to go back to uh visualization this time we're going to look at some of the categorical responses so most of the questions in this survey they've got categorical responses you got five choices It Go strongly strongly disagree disagree neither agree nor disagree agree and then strongly agree so you got five points now the values are encoded as numbers let me scroll back to where the data set is shown and you can just see what they're like so uh for example uh all of these These are these um uh questions with the responses from strongly disagreed to agree so you can see their numbers from 1 2 3 4 five this is okay for some kinds of analyses but when we're drawing visualizations just showing 1 2 3 4 5 is not really helpful what we really want is to show strongly disagree and so on up to strongly agree so we're going to need to replace these uh numbers with the text labels uh there is in fact one more um tricky point with this is that sometimes you're going to get um questions where nobody said strongly disagree or some of the data is missing so uh you can have sort of uh some missing uh values to deal with and so uh we're going to see how to De that now one thing to note very useful piece of jargon if you're doing any kind of survey analysis or in fact anything sort of social sciences um these sort of questions where you've got um opinionated responses going from strongly disagree to disagree or whatever the categories are so you got ordered categories uh they're called lyer scales um and so it's just a bit of useful J to know if someone ever mentions that to you okay this is just about opinions um right so uh first thing we do is build up a SQL query to get the counts reach response type and then we're going to use that to draw a bar plot uh now I've got a file here called um agree disagree. CSV so this is going to be a lookup table that's going to take us from those uh numeric codes the 1 2 3 4 5 uh to the uh the text values that we need so first thing we're going to do is import that so we're going to do select star from again so this is uh standard formatting uh which means we can just use string with the file name uh oh I need to change uh the value this to lookup um I think you probably all going need to change this so you want to make sure the return value is a data frame available as lookup you can see we've got two columns we've got a CO column called code 1 2 3 4 five colon called response and those are the five different options all right next thing so uh um what we need to do is take this lookup table and we're going to do a left join to the survey data uh so first thing copy and paste the previous code and then we'll add in a left join uh oh actually I wanted the as look up here um so we going to do left join and one useful thing uh that read CSV Auto function that we used before in the from Clause it also works in the join Clause as well so I'm going to go back to where we had this before and I'm going to copy and paste that uh that reading code the read CSV auto code and so it's here in the in the left join code and I going to say this one is as survey now we need to tell um the SQL query which columns we're joining on uh so we're going to join on lookup. code is equal to survey dot uh oh uh we're look at question three row one there are a lot of these different questions but I thought uh let's start the first one and and that's about it so let's run this ah so we got lots and lots and lots of columns there's a bit much to worry about so let's rather than selecting everything let's select some specific columns so we're going to select lookup. response and we're going to select the survey dot uh question three row one let's try this again all right so we got 121 rows and um so actually let meble check I've done that right we should only have 120 rows really uh so look up all right let let's see if this works um so you can see wherever um you've got the value agree um the the original value was a four and if you got strongly agree the original value was a five whether it's wherever it's neither agree or disagree um you've got the value three and so on so we've got all the different uh uh responses uh it's just now we've got a text version as well as uh the original code all right uh so we're going to build on this again excuse me um and this time we want to get the count rather than just the individual values because we don't care about like uh the individual responses We Care like how many times it say agree how many times it say neither agree nor disagree so next thing we're going to copy and paste uh I think this is actually easier if we put each thing that we're pulling in on a separate line so one thing we have to change is rather than getting individual um responses we're now going to get the count of those so I'm going to wrap this survey. equation 3 Row one in count I'm going give it a meaningful name so this is the number of responses we call it n and then we need to group the results by um by the lookup response all right so here you can see for this question three part one we've got uh zero cases of strongly disagree so no one said that they strongly disagree with things um we've got uh 18 cases neither agree nor disagree we've got 67 cases of agree 29 strongly agree six disagree so this is great um there's a little problem in that these aren't in the right order ideally want things to go from strongly disagree all way through to strongly agree in in in the right order that's not there yet so we're going to have to include an order by Clause the other thing just bear with me a second the other thing is that in order to draw an easy uh to interpret plot we want to include a color scheme now for this sort of data where you've got a central value which in this case is neither agree nor disagree you want that to as a neutral color and then you want um bars that represent agreement to all be the same color and you want bars representing disagreement to all be the same color that's going to make it easy to understand and then generally the stronger the opinion the more intense the color you want so we're going to pick a color scheme that uh makes sense for that in order to get the data to make this work we need to add an agreement um statistic so what we want here is evaluate so where neither agree nor disagree is this is the neutral thing one that to be zero when you've got positive agreement you want have a positive number and the stronger the agreement the bigger the number likewise when you've got disagreement you want a negative number and the stronger the agreement the bigger the negative number so in this case we want to take the lookup code and subtract three and that's going to give us a an agreement statistic so just a little bit of data wrangling there all right so uh I am going to copy this and we're going to do two little tweaks uh and then we we've finally got the um the the data we need so um I took told you we had to or order these so we're going to add an order by clause and we're going to order these by the lookup code so this is like the values one to five so that's going to give us strongly um disagree is like first and then strongly agree is last and the other thing we need to do is add in um this uh this agreement value so we're in the select clause so this is the last thing we're selecting at the moment the count value is n I'm going to add a comma it's very important you got to put a comma between all all the things you're adding hit enter it's G me a new line so here um I'm going to calculate the lookup. code minus 3 and Call this agreement let's run this um okay ah okay so one other thing I need to do is add this to the group by Clause now so you need to in the group by um Clause you've got lookup response you want a comma and then add lookup code and the reason for that is because we're doing calculations that aren't included in this count aggregation let's try that again all right so we got the same same numbers as before just in a different order so here now you can see the results go from strongly disagree through to strongly agree so these are all in the right order and then we've got this agreement level so you got zero in the middle disagreement is negative agreement is positive now this is the data we want so we're going to save the results if you call it Q3 uncore 1core counts uh make sure it's the same just to uh just so uh the following code is going to match all right now we're finally ready to draw our plot so um in this case because we've got counts of categorical data we want to draw a bar plot and we're going to do some uh as I mentioned we're going to use um a color scale so uh it's called a diverging color scale it's got a neutral value in the middle and then it goes in two different directions so uh to draw a bar plot we're going to start off px. bar and we got a few um arguments to add to this so the first argument is the data set we're going to use so it's Q3 1core counts the second argument is what we're going to put on the xax which in this case uh is this response column on the y axis we want the counts and then we're going to color the Bars by agreement now one final actually let let me plot this as is and then we'll show you what happens when we give it a different color scale all right so here are the results you can see uh as we saw from the numbers no one strongly disagreed the most common thing was that well quite a lot of people just agreed and then there were a few people neither agree nor disagree and a few people strongly agree now this color scheme of of agree it it's a bit of a mild melter like the the sort of the rainbow color scheme going from purple to Yellow um this isn't very useful to us so the one thing we need to do to uh tweak this plot is to add in uh a diverging uh continuous color scale so in the cotents I've uh explained how to do this so we're going to use the the color continuous scale argument and then the options for this uh are all contained in px. colors. diverging do PX dot uh colors dot uh diverging and then you got all sorts of different color scales um feel free to try a few and see which ones you like I actually quite like the the Army Rose uh uncore arpa reversed for this so let me uh hit play and you can see what it looks like so in this case you've got a neutral beige color for the neutral response and the positive response is uh sort of green it's like an army green so this helps for like color blindness is like not quite the same as the um as the red like I'm I'm color blind red green color blinds and I can see the difference between these so then you got like red responses for um for disagreement and then the the most intense strong agreement or disagreements they have um a darker uh color scheme so this works really well for um this sort of uh opinion data set the like H scale uh qu categorical questions uh I'm pretty happy with how this looks now before we go to questions you have got one more option you're going to do the same again uh using uh a different categorical question so any of the uh questions in here with um uh from question three through to question six they've all got the same agree disagree scale so you're going to go through and take the previous SQL query put that together um with a new question just edit it for the new question and then you're going to draw a bar plot I'm G give you two minutes to do this um and then I'll go through this last answer and then we'll go to audience questions and actually reys uh yeah she got two minutes for that ree do we have many questions from the audience yet yep we got a couple uh so the first one that we've got is from Eric he says uh why is the column the same uh but it says change the column uh I think this was this was an earlier one uh back when we were doing the SQL bits uh so yeah why is the column the same but it says change the column I think that's when we had an error earlier uh that was almost certainly copy and paste error or a bad bad instruction from editing on my part um uh so next one is from uh Tom he says uh in which case can we use the man Whitney U test uh when comparing two numerical variables yeah so the idea is you've got two groups of uh variables you want to know on average are they the same or different uh so the short ter um the conditions generally I think it it works best when you've got slightly larger sample sizes you need at least I think 20 bits of data in each set is the sort of standard um and you because it's it's what you call a non-parametric test so it doesn't worry about the shape of the distribution um that means it's got less power than something like a test so it's less powerful for detecting differences but it does mean that you don't have the um the constraint that each of your distributions should be normally distributed in order for it to make sense so um this is useful when you've got two sets of weirdly distributed data and you want to know is there a difference between them got it cool um we got a question from Rich saying uh can you explain what a left joint is joints can be complicated oh man not in 10 minutes It's when um you tend to use them when you've got repeated values of um of rows and you want to like it works really well in this sort of count situation when you want to know how many of uh something have a have I got how many instances of something I've got that correspond to different categories um I would say I'm not going to do justice to An Answer here go and take the um the joining Daton SQL course actually yeah we got ton of joining course you can do the same again in Python and R as well um if you really want and it also gets taught in about like 12 other different courses in sort of um in in small chunks because it's something you need to know and it's it's probably the most difficult thing you do conceptually except for maybe statistics like in any of the data analyst or data scientist tracks uh it does take a while to get your head around it um if you've come across a vlookup in Excel is the same thing vlookups are left joints got it um yeah we'll we'll do a whole session on joining data at some point you practice the different things I would say you're going to get a better explanation if you take some other courses okay cool um I'll be I'll I'll find a link to the uh to the joining data uh course and put that in the chat shortly for one wonderful all right I'll go through the answers of this then and uh if you everyone in the audience if you want to think of more questions uh we've got a few minutes left to do that okay so um we're going to uh pick a different question we're going to copy and paste the previous SQL query and update it for the new question all right so I'm going back to the query I'm going to take a copy paste this in uh I'm going to do question three row 13 and it needs changing in two places it needs changing in this count and it also needs changing in the on part of the join Clause so changing in two places uh saving it as Q3 13 counts um if you picked a different question give it a meaningful name for you so got a different set of uh counts here and then I'm going to go back and again I'm going to copy and paste the uh the plotting code so just needs changing in one place here and we're going to run this this all right so this one's a lot more balanced it's like before we saw there was a lot of agreement here obiously most people don't have like there's a fairly sort of balanced thing between strong opinion like strongly agree and strongly disagree it's actually unusual so one thing you find in a lot of surveys is that when someone says a statement very few people will have strong opinions so you they tend to sort of do something through the middle and in general there's a bit of a towards agreeing like a lot of people are very agreeable like someone said the statement okay I'll agree with it so strongly disagree is actually like quite a rare thing in survey responses um I guess there are some people who would actually disagree well that they're going to respond like that but it's rare So when you're designing a survey you got to be very careful about how you phrase your questions and maybe sometimes you have to have questions phrased in opposite manners uh just to see do people um just account the sort of biases how people answer questions how people respond okay so that's that's pretty cool um maybe while we're doing this we'll we'll just try another uh another color scheme so you can see what it looks like uh we'll try Earth okay that's it you got kind of bluish on one side brownish on the a little bit muddy in some situations that's probably kind of cool I'm not sure it works that well for this sort of survey data um have a play around try some more color schemes see which one you like best okay uh with that uh I think I'm done coding along so I'm going to return to questions okay brilliant uh so we've got a couple of specific questions on workspace just a reminder for anyone uh that uh hasn't sent a question in that you want answered now is the time to do it uh and yeah we'll we'll get to them over the next 10 minutes so um we've got a couple of yeah workspace specific ones so we might we may need to jump back in uh so we've got this one from Tom he says uh is it always that easy to switch from an SQL cell to a python cell in a workspace um is there any preparation that you did for this specific session um so actually my python is slightly stronger than my SQL so I started writing all the data analysis code in Python and then I was like oh yeah we should probably should probably do this in SQL otherwise it's going to be a bit weird um so uh yeah it it is very straightforward um yeah the the natural sort of flow workspace designed to have this natural flow where you pull in data either from a database or CSV file using SQL and then SQL is very good for simple number crunching and then you can save the results data frame continue do more sophisticated things in in Python uh or D if you if that's what you prefer so yeah it's very very easy in workspace this exactly we just built it to be able to do this cool uh next one is from Anonymous LinkedIn user they ask um the way you you're directly referring to the CSV file name in the SQL query uh this method doesn't work outside of um workspace uh what do we do in in that instance um well I I obviously prefer use workspace but yeah uh so in a standard Jupiter notebook all this is based on using duck DB so duck DB is going to have um like the it's got the read CSV Auto function you just got to figure out how to integrate duck DB with your existing notebook I think there are some docu there is some documentation on that on the duck DB website um the other alternative is you just ignore SQL completely if you're wanting to use Python then pandas has all the sort of functionality buil you can do everything in Python if you prefer cool uh next question is from Yukan nice to see you again Yukon um is there any test that can be used for uh comparing two categorical variables um yeah it depends what you're comparing for I think I think you're probably looking for a Kai squar test but depends on the question you're going to answer I would say go and take the hypothesis testing courses because that gives you a pretty clear idea of like which test to use in which um situation I have to say after years and years of doing this I can never remember which test to use in which situation and I do always have to look it up before I actually start doing any analysis uh so yeah uh I defer to the course again in this situation cool um second of all we've got a question from sumano he asks uh could we have used a two sample unpaired T test for the hypothesis test you can so really T Test is sort of you're supposed to have vaguely normally distributed data it's going to give you a slightly different P value and uh I don't I would say run it and see what happens I don't think you get it much of a difference but in general when you've got this really skewed data um the non-parametric version the will Cox and um sorry the manw U test or V Cox and rank on test same thing um that's preferred cool um that's it for audience questions uh for the moment I've got one question of my own uh before we before we wrap things up so if someone's completed this code long uh without too many problems what should they be looking at learning next like what's what's the next sort of progression yeah absolutely so um one thing we didn't get a chance to look at today is how the responses to multiple questions interact so I did some homework tasks there at the end of the workspace uh so for example you've got two different um numeric uh response questions you can put those on a scatter plot and see what is the relationship between the responses in those two cases uh we have like lot because there were lots of categorical um response questions you can look at pairs of these and see uh you can draw them on a a heat map and see what's the relationship between those two distributions there's also all sort of really interesting questions like um which statements people most agree with or most disagree with so finding out some sort of sense of average agreement ordering them by question really really interesting you can also see like which ones made people like were most passionate about answering like what made them most mad like which questions have the most strong agreement and strong disagreements there's all sorts of stuff you can look at in this data set um and then in terms of like what what do you want to learn I would say please do go and take the analyzing survey data we got one in Python one in r as well uh take one of's courses I appre most people here go more python people so take uh that course cool and I've just put that in the chat for anyone that wants to have a look at those courses as well brilliant uh that's it for questions so I'll uh I'll leave you wrap up all right super before you all dash off we got three more webinars this week so tomorrow there is a session on uh data governance strategy so data quality data governance is just tremendously important for like actually having the right answer in any c in any your analysis that's going to be worth turning up to tomorrow got um some very senior Executives coming along to that on Thursday there is a careers based session so we're looking at how top universities teach data science so I'm interviewing two people who run uh Masters Pro uh Masters in data programs talking about like what skills do you need to know how does it lead to getting a job and then on Friday very cool session from some previous winners of uh data Camp competitions uh so uh we got four previous winners talking about uh what they did to win all right um with that uh I think we're at time so uh I hope to see you all again later in the week to see you again in future webinars thank you very much