Data Visualization and D3 by David Chouinard

Key Takeaways

[Music] I'm David Shard and this is D3 welcome we're going to learn about D3 today D3 is a JavaScript framework for building high quality interactive visualizations for the web things like what we're seeing in of me we're going to learn to make those things I mean kind of the basics of it but it's going to be cool let's get started making pretty pictures bit more demos of prospects available let's do it Act One Dom manipulation we're going to start right away making cool things first of all we have an on the left we have code on the right we have the result of our code let's go through it let's make a circle how does that sound SVG that append Circle we just made a circle you don't believe me right I mean it's not there so what we did right here is SVG is a scalable vector graphics this is the way we tell the browser to make vector graphics in the in the browser what we just did right now is added a circle to the Brower browser the problem is that the circle requires a bit of basic attributes before it can act we can actually see it we need to tell it it's X position it's y position it's radius we didn't tell any of that so we're not seeing it right now but let's let's Tell It stuff so let's um first of all you got to we got to give our Circle a name so let's call it how about Circle our Circle has a name now let's um and let's give it a few few attributes how about CX would be Center X so the center of the X position let's say 200 for 200 pixels let's give it a y of 200 pixels as well and an r a radius of how about 40 pixels let's see oops well can I not spell there you go we have a circle at position 200 pixels 200 pixel radius of 40 pixels it's kind of cool right we have a circle yeah so all this no need to follow along all these examples will be all the all the code I'm doing today will be provided online at the end for an in the form of inter interactive examples or checkpoints at every ax along let's do more stuff though this this black circle is really ugly um I'm sorry for that error message right there there we go um let's give it a color how's that I like steel blue well color change or Circle change color that's great let's give it let's make it semi-transparent too semi-transparent so these so these are attributes we're defining on the circle the first thing we did is we put a circle on the page and then we're defining a bunch of attributes some of these are required like CX c y and radius and others are optional there are a lot more attributes I mean there's there's a lot of them uh for example we could have a stroke as well stroke of red but let's remove that we're back to a circle blue Circle so let's make more circles how's that let's make another Circle this is exciting right so let's uh I just copy pasted what we had already let's call it Circle two and let's do the exact same thing and give it attributes give it an exposition of 300 yay we have two circles now and of course we can you know we could we could update these values I could put it at 400 and now it moves um and since it's annoying let's remove it so Circle two remove it's gone now so what we're doing right now is just very very this is very similar to what you might do in jQuery for example we're just manipulating the Dom it's called you might have heard that word before we're creating stuff setting attributes on stuff removing stuff now here's where it gets interesting so later later in the code we could still refer to the original Circle here so let's uh let's reset it attribute to um it's uh CX to say it's exposition to 400 and I'm going to transition that so it's obvious there we go so we we added a circle we set some attributes we added another Circle removed it and then we're modifying the original Circle but here's get where it gets a lot more interesting not only can we set attributes as just values we can say hey pay Circle go to position 200 we can also set them as functions so instead of giving 400 here we can make some calculation on the Fly for what we want that that attribute to be so this is how you'd Express that we say instead of 400 let me give you a function in instead and let and here inside this function we can make any crazy calculation we could you know take the time and look at some other thing and dynamically decide for the circle what value we want let's just how about we just give it a random a random Exposition so that's that so what that says is for every X run this function and what we're doing is we're calculating some random times the width and we're R turning that so every time you run that you get a circle that goes at a random place kind of cool I mean I feel like I could look at this for a little while um yeah we're starting to get to something interesting here let's make this data driven now there's no data here let's change that act two data driven documents okay so let's let's return to here and let's just get rid of circle two because we're just adding in a remove it so we don't really need it we need to be a lot more clever here let's let's say let's say we had let's say we had some data of some sort um something one moment let's say we had data of this form we had an array just a bunch of numbers we have seven numbers here whatever this represent banking you know amount in people's bank account how how much they weigh God knows what these are numbers and we want to use our circles to represent those those numbers somehow we want to tie our circles to those numbers um so what we do let's say we want a circle for every number we could do the old thing we were doing Circle a pen and circle two and then Circle three but this gets out of hand and there's a lot of repetive loog repeating logic so let's get let's let's get more clever with that instead of using this VAR Circle SVG pen that we were just using we're going to use this little block here um I don't want to go in depth into what all these parts do and it's kind of an advanced topic and I wish I could uh but the key thing to to recognize and you see this very often in D3 code this block of text basic creates as many circles as our data elements in this array right here sorry so this creates as many circles as are elements it's going to create us seven circles and it does a really really key key thing so let's run that let's remove our other Circle uh let's just comment this part out and run this again there you go so our Circle here is a lot darker because we have seven circles one on top of the other we just created seven circles one one each for each of these data elements but there's a key thing that happened with the snippet right here is that data was bound so every single one of those data elements the 1045 105 was bound to a particular Circle so these not only created a bunch of circles but tied those two things together and in the future if when we because we created the circles with this D3 function if I give you a circle you can give me the data associated with it so we can ask D3 hey D3 I have the circle what's the data that the circle has and D3 would tell us oh 10 or 4 five or 105 these things are bound that's a very very fundamental concept let's let's look at that so the way we ask E3 so this is how we I this is this is irrelevant for this but just trust me on it um this is how we ask D3 hey D3 give me the first Circle that you can find give me the first Circle you can find and then we could ask D3 well what's the data on that like this 10 so we just asked D3 D3 find me the first Circle you can find what's its data 10 that is indeed our first data element but ask it hey D3 find us our third circle 105 why is this really important so right here I mentioned that we could use functions and I mentioned that was a very powerful thing so not only can our functions do things like do some computation and return return a random do some computation for example return a random number it can also do things based on the data the this is what this is what data driven documents mean that's what D3 stands for so this Exposition instead of just saying hey all all the circles be at Exposition 200 we could give it a function and here we can make some calculation D and D here stands in place for the data so every time we have a circle basically D3 will create these seven circles and then for every Circle it's going to go hey Circle one what's your X position previously we just we were always answering 200 but now every time D3 asks us what's your exp position it's going to give us we have that Circle so we have the data it's going to give us the data and say what do you want the exposition to be based on that data let's just return the actual data so if we run this this gives us datadriven documents these circles are based in relation position as they're based as a function of the data so for the first Circle D3 puts the circles in and D3 asks hey where what do you on the exposition to be and we just say whatever the data is make the exposition 10 and then it asks what do you want the exposition to be for the the second Circle and we answer 45 and we of course can make some computation here I find that there those circles are kind of squished up so let's uh let's multiply by three multiply a data by three our circles just got expanded out right our value was was tripled uh the circle is really on the edge so let's let's maybe kind of offset it um let's say by 20 there you go this is a data visualization it's a very basic one but this gives us some insight into our data it tells us that for example we have a little cluster of elements and we have a a big outlier here this gives us some in information about the distribution if we were for example to change the data um to 150 here and and refresh our visualization has changed this document is data deriva so of course all these elements all these attributes here we can use a function not just the numbers not just the X and Y positions and things so we can use a function for the color um so we'll do the same we'll give it a function and let's say we could have conditionals in our function this function could be hundreds of lines long it could be very very complicated things so let's let's put a if statement here let's say if you know if our data is less than 50 that's some threshold that we're interested in for some reason let's make it green otherwise let's make it red how's that nice so our data visualization starting to convey more interesting information on on many channels so now we know a bit about the distribution and we know that there's some sort of cut off at 50 that we're interested in we know that there's two little points below that threshold and most of them Above So as a final step let's this data here is very rare to see this like that so let's just move it out to a variable because it's cleaner like this and then we use that variable here that's the exact same thing it's just a bit clean next up act three scale so one problem right here is if we we if we change our data in this this 200 value if we change it to 400 or something and refresh then this value just went offc screen so our our logic right here of how we do the times three and plus 20 to spread it out and then offset it a bit is really clunky I mean what those those numbers mean they're just hardcoded there and they're they're very much tied to the data we want we we want a data driven document we want a very flexible a very flexible document that given data adapts to it and represents it what we basically need is we have this range of numbers 10 45 105 and we want to map that out onto the width the full width here so we have this range of numbers going from Z to 200 and we have this canvas that goes from 20 to 700 in this case we kind of want to map that on we want to scale that up and then offset it a little bit it turns out that D3 has these it's called a scale so let's use it the way that works I'm going to type this out and then explain it this is a scale what it'll do is it'll map out values from 1 to 200 onto 20 to 600 we can we can check that we can see what that we can see that here so if I feed it one one moment well give me one second I must have maped it there you go I'm sorry about that so a scale what a scale will do is it'll take a value and then and then convert that expand that out so it fills the full range you're asking for so in this case if if we give it one one it's going to map that out on to 20 and if we give it 200 it's going to map that on to 600 and somewhere in between if we get a a 100 it's going to be somewhere in between 20 and 600 and of course now this is D this is this is what we need uh to remove those those hardcoded things we have right there so what we want to do is take the data that we're given that individual data element and pass it to to scale first so scale will scale it up um well oh we have a little error here missing a data there you go and that expands it out that gives us the same result we had before but instead of having those hardcoded constraints and if if the size of our canvas changes for example if we want to have this over 400 pixels then it squishes out we can have it over you know we can expand it or we can reduce this this left U this left margin to something less or more than 20 these numbers these harded numbers now make sense to us uh we could do a lot more interesting things as well so we can we can instead of having a linear scale we might want a log scale and that will give us a log scale so now our scale instead of just map just expanding out that range it's doing more sophisticated things um instead instead of having this this uh range hardcoded and instead of having that 600 we might want to just use the width so from 20 to the width minus 40 two times the margin on the other side and this makes a lot more sense to somebody who might look at the code interestingly scales get very very sophisticated as well they do a lot of interesting things so scales don't necessarily have to operate just with numbers let's make a color scale so our range could be our domain is one to 200 that's the input thing uh but we might want to map from green to Red for example and now every if we pass it one we're going to get green if we P if we give it 200 we'll get red and if we pass it something in between it's going to be some some mix of that somewhere on the gradient between green and red and instead of having this kind of clunky logic we have here with with the you know the conditional right there you could have something um linear a a linear scale between those so we'd pass we'd use a scale we just created which we called color and we give it D which is our data element and there we go we have a color scale so this is mapping so the far left is is completely green the far right is completely red and everything in between is a function of d we have an interesting visualization here but our data is kind of boring let's see what we could do if we had more interesting data act four working with data the first thing we'll want to do to make our visualization more interesting is to move the data somewhere else it's very clunky to have the data hardcoded here in generally we We'll be asking somebody else for the data we'll be you know maybe asking the government the Census Bureau hey what's your data and then plotting that or asking some thirdparty entity for some data and then building a visualization on that uh so the first thing we want to do is move that to somewhere else so I'm going to create a file here called data. Json Json is the data format you don't have to know much about that um and we're going to copy the little data we have there paste it in there verbatim go back to our uh our uh visualization code here and use this function right here you don't have to know the details but what this will do is we'll we'll find that file fetch it and then return it to us um so what this does is it goes and get the data. Json file and then all the code that's indented inside basically essentially all the code we have there will run only when we get the data back and then it's going to pass it uh run that code with the data we have great we have a visualization that queries the code queries for some code somewhere else which is usually where queries from data for some from somewhere else which is usually how visualizations work but I want to go back to the data so the data fundamentally in D3 for D3 D3 consumes data that's a list of things D3 expects the data to just be a list of things an array of things it doesn't matter what those things are so long as it's an array of them uh so here for example we could of course have floating Point values we could have negatives we could have a D3 doesn't care so long as it's a list of things um as interesting things we could have we could also have a list of strings like that so these are these are um headline the CRI Crimson headlines I picked I looked a few days ago and maybe you can find some interesting things about these headlines so again this is a list of things D3 doesn't care these happens to be string let's return we've changed our data let's return to our visualization now our visualization expects the input to be um numers so we're going to have to make a few changes uh so for example first of all we maybe we want to put these circles Along by the by the length of the headline the number of characters in the headline so what we'll do is of of every time we our function is called with a um with a string we'll find its length and then pass that the scale uh the color I'll return that to Steel blue and there we go we have a visualization of crimson headlines our scale is a bit off um let's assume that the longest headline is a 100 characters long so span it out a bit and we have a visualization so most most it seems that most headlines are pretty close together in terms of uh character length but one that really stands out um we could you know build some tools to explore that more um but let's when I was working on this I was curious whether in this data set headlines with a colon in them would be longer I I assume they would so let's find out let's use the color Channel like we did did before to encode some information about whether there's a colon or no so we'll use a conditional again you don't have to know the the detail of this but this is how we check a string for a particular character in JavaScript again not relevant but if we if we don't find a colon we'll return green and if we do we'll return Red so again headlines that have a c will be red this is what this means nice um so it seems like my hypoth this is debunked I mean there's only two we only have six data points and only two at col but it seems they're more on the on the Lauren in fact headlines of colon seems to generally be shorter at least in our data set interesting let's return that to Steel blue and let's see what we can make with with more interest even more interesting data so again I mentioned that data in D3 is a list of things we've seen numbers of many types we've seen strings but the things can also be objects they can be complicated things that include a lot of things um to say that more clearly in most cases we want to build every data point is more complicated than just one value if you'd imagine a database about students there might be a student name a student ID and a lot of things associated with a particular record not just a string or a number so let's look at that we might this is a this is one such of data set this is a data set about earthquakes so everything here in our in our list array of things contains many things itself so every data point has a magnitude and a coordinate and coordinates thes contain two things um so every data point is now a lot more complicated and a lot more interesting and contains much more interesting information let's see what we could build out of that we turning back to here um let's see if we can again using our kind of histogram Circle visualization we've built let's see if we can build a visualization of of a magnitude distribution in our data set so here it's the same concept but now D is contains more things D contains many data elements uh so we get DB back we get we you know D D3 gives us D and we respond by finding the magnitude of D and then passing that to scale and then we we need to change our scale of course so the magnitudes typically don't go much more than 10 actually there's never been a 10 magnitude earthquake but that's kind of our our upper end our upper Spectrum let's refresh nice we have a visualization um it's interesting to note so there's two data points that are almost exactly on top of each other in terms of mass magnitude you see this by the by the opacity we're using we have Geographic data now we have lat latitudes and longitude maybe we could do something a lot more interesting with that let's see let's find some more interesting way to visualize this more complicated data we have access to act five mapping so we want to put these fundamentally we want to put these on a map I mean this is where this is going we want to encode information about the position about the position of these earthquake readings as well as our magnitude because we have that now we we understand how to consume more complicated data um the first thing we'll do is is create a map a background map I'm going to go through this very quickly I don't this is this is tricky code it's another one of kind of those recipes you don't really have to understand fully for you to use um but this is code this code right here creates a map we're not going to go in detail but in gen superficially what it does is it queries this us. Json file which is a data file like the one we had before it's more complex of course but in this case everything every data point is a state and has a list of latitudes and longitude that Define the polygon that form that state um so what D3 will do is it's similar to what we did before is it'll request that and bind that to an element and then and there's a function that will map that element out based on the the latitudes and longitude you can read born that and I recommend it when this code the Links at the end with this code posted and there the code is commented and there are links for further reading on this I recommend you look it up but what we care about uh is this projection function I want to go through that first of all let me show you that yes we have a map maps are cool um so let's look at this projection function projection is very much like a scale scales again so what projection this projection function does is we could pass it longitude and latitudes in this case these these values here are the lat Longs of the building we're sitting in right now to projection and projection will convert that into X and Y pixel values so what projection is doing is very similar to our scale it's taking our latitude and longitude that represents a whole glob globe and Shrink and sizing that down to the square that we want that we've given it um in this case so we're passing these value and it's giving us well that on your screen means 64 640 pixels this whole screen is 700 pixels wide so that makes us about here and 154 pixels down which I would estimate is you know pretty much here so taking those lat Longs which represent something on the whole globe is pushing and moving that around to give us X and Y pixels values this is the first thing that's done in this mapping code and then the rest of the code consumes the data and then Maps those lat long onto uh onto something on your screen but we're going to use projection this projection function because it turns out we have lat Longs as well looking back at our data we have latitudes and longitude coordinates for every observation so let's use projection so looking at our exp position we want our Exposition we have a latitude and a longitude but we want pixel values and it turns out we have exactly what we want projection and very much like we were using scale right here we're now going to use projection and pass it coordinates so the first thing we're doing so we're getting d which is an individual data element of an individual earthquake reading the first thing we do is get the coordinates all right we have the coordinates the second thing we do is pass that on to projection projection converts those coordinates into pixel values X and y's and then the last thing we want to do is just get the X which in G in this case is the first one is the first of the two things that are returned by projection we'll do the same for y but instead we'll return the second element the Y you ready to refresh oh extra character here nice we have a data driven document that's consuming this Json file of objects making a map and and uh and changing the attribute in relation to the data to project it on a map this is really interesting this is cool let's let's take it up a not I mean we have we have two pieces of information with every data point I mean three we have the coordinates which is an X and A Y and we have the magnitude I mean I can't we need to encode magnitude somehow we have a lot of channels we could you know use color can use radius uh we could use opacity we could use many things in Coda we could any of these attributes and many more that are that are not listed there because they're optional we could use to encode this data uh the stroke and all these things we've mention I've mentioned let's do radius I think radius is the most intuitive so again we'll replace that hardcoded 40 and make some calculation we'll use our our favorite our scale again and we'll pass D but not D because we want the magnitude of d d is just a data point we'll pass the magnitude to scale let's try it again oh doesn't work why does it not work so remember what scale does let's look at scale again scale maps from 1 to 10 onto 20 to 600 more or less 600 is huge I mean this is this is why we're getting this so we want to change our scale to something more reasonable let's say we want to 0 to 60 I mean 60 is big but 10 Earth equates are incredibly rare in fact they've never happened uh so what this will do is it'll take our magnitude that goes from 1 to 10 and map it on to expand it out and map it on to 0 to 60 let's refresh nice we have a visualization this is great this is actual data you'll notice in my little toy example the biggest earthquake is right on top of us um but that's it we have a we have a data driven visualization that consumes the data and gives us really interesting information let's take let's um yeah let's add some interactivity to I mentioned now as the strong force of D3 so here for every element we're describing a bunch of attributes but we can also describe what we want to happen when interactivity elements for example we could describe describe what happens when we Mouse over and very similar to that that will take a function very similar to the attributes we had before where we do something to the data to the the the the data the U the element when we hover over it so the first thing we need to do is Select that element to find it basically in the browser um and then we could set and then we could set an attribute to it so what I'm doing here is when we hover over something we'll get that element and then set an set its opacity back to one to completely opaque let's see what that looks like um appears we have an extra extra semicolon here there you go so if we hover over here oh it gets it gets full but now of course it stays full because we we have to describe what happens when we when we remove our cursor uh so let's do exactly that on mil out as opposed to Mouse over and we'll reset it to what we had before 0.5 and now every time we hover we get a full circle that helps us see what we what we're selecting essentially and now let's make this really great let's connect this to real data so let's let's ask USGS about their data so the US Geological Survey has data about earthquakes um they have a public API that's a able to be consumed in Json format uh so let's let's do that so this is a bit of code that connects to this EUR this connects to the USGS API and there's a bit of processing on it this is not this is not relevant but simplifies it to a simple data format like the one we had before so I get rid of our call to our fake data. Json file and instead I'm calling um the USGS essentially that's refresh nice this is actual real life data from this week for earthquakes uh this is really interesting I mean apparently there's this is not surprising of course but there's a lot of earthquakes on the west coast in California but I I I thought it was very interesting that there were so many earthquakes in Alaska and apparently here in the Midwest I mean interesting and we're good that's the conclusion but fundamentally this this this is what D3 helps us do it helps us take data bind it to elements in the Dom and have those elements change as a function of the data how those attributes all the many attributes of the elements all be useful for channels to convey information D3 is an incredibly powerful library and amazingly well-run this is this is some power powerful stuff data visualization is an incredibly powerful tool for conveying to people deep insights that that gets to their core and helps them understanding in this profound and intuitive way how data works and how data changes your life