Jacob Tomlinson - What is my GPU doing? | JupyterCon 2020

Key Takeaways

hi everybody my name is jacob tomlinson i'm a software engineer at nvidia and i work on open source libraries including rapids and ask in this talk i'm going to be going through how to visualize and and monitor what your gpu is doing which is really important when you're getting to grips with with uh open source python libraries of the gpu accelerated you need to understand what your hardware is doing uh whilst you're running them so rapids is is a suite of open source python libraries which allow you to to use familiar python apis um but access the gpu and and get some acceleration through that so if you're a pandas user you can use qdf if you're a psychic learn user you can use qml and so on there's also like a bunch of other popular libraries that kind of fall under the rapids banner that are part of the open source community so you've got coupey which comes from the channel community you've got number you've also got all your deep learning libraries like pytorch and tensorflow all of these things are kind of uh popular within the pi data ecosystem um but sometimes when you're when you're working with cpus and then starting to accelerate your work with gpus it's not obvious how to really understand what's going on inside the machine that you're working on so we have a bunch of packages that you can install a lot of things come with with rapids so if you head to the rapids website um there are instructions here on on installing rapids you can install it with conda from the from the rapids and quandoforge channels you can grab the docker image and run that whatever you whatever you like to do but i've gone ahead and set myself up with an environment running the the standard condor install command from here i've also installed mv dashboard which is a python package um and jupiter lab extension which fall under under the rapids banner so it comes in like two components you have a server side extension which is used to like query the gpu and then we have a jupiter lab extension for viewing viewing plots and and things in in the web interface as you can go ahead and install that directly with pip and then the jupiter lab extension tools and we're also going to have a look at das integration with gpu monitoring as well so we have the das lab extension which uh similarly has like a python component and an adobe lab front-end component which allows you to understand what's going on inside your data cluster and das has some some knowledge of what is going on in the gpus that it is scheduling on so i've got all of this installed in an environment and we're going to jump straight over to jupiter lab now to try and figure out what my gpu is doing so i have i have a few plots open already on my screen here um and this is a combination of things from mv dashboard and from the dash lab extension so we can try and see what's going on i also have an example notebook here which is gonna mainly just stress the gpu so we can see what see what's happening um so at the beginning i'm gonna import a local cuda cluster from das cuda this is because the machine that i'm working on has two gpus in so i want to make use of both of those and das is great for scheduling onto onto multiple cores and in this case it can schedule on to multiple gpus in my machine i'm also using das qdf so qdf as i said before is this this gpu accelerated pandas api library so you can do all the kind of standard stuff you would do in pandas but it will run on the gpu by using das qdf this will be distributed across across my dash cluster so let's get these things imported and i'm going to create myself a cluster so this is going to inspect my machine see how many gpus i have and then it will start one dash cuda worker per gpu so we can see in our little visualization here we have two workers that have been created we can also see in our cluster map down in the in the bottom here that you can see the scheduler in the middle in our and our two workers and this is kind of to visualize how they're communicating with each other so i'm now going to read in some new york taxi data you know it's a nice open data set that's easy to get hold of so i'm going to read in some of this data i'm going to load it into gpu memory i'm going to calculate the length of it see how many records we have and then i'm going to do a describe of it uh just just something to make the gpu busy right and i'm going to call time it just kind of make it go around a few times if i run these cells we can see in the top right here the gpu utilization uh and the gpu memory of our two gpus in the machine you can also see the machine resources plot below that which is showing cpu memory disk bandwidth um network communication etc and at the very bottom we can see the task stream so these are all the function calls that das is making on our gpus and there's kind of two rows in this for our two gpus so this has given us like a nice overview of what's going on inside this machine we can understand um what's happening as as things are running but let's kind of pull this all the way back to basics i'm going to close out all of these these views and we're going to start kind of from scratch to see how this stuff works how we're getting this information and how we're displaying it so i'm going to close all of these panels out we'll come back and kind of open them as we as we go through later i'm also going to move this over to the side here just so that we've got a little bit more room i've got this notebook here what is my gpu doing and we're going to use some low-level libraries to get information about the gpus and then build up some of our own plots and dashboards so the first library i'm going to be using is called pi nvml nvml is is a way of querying the gpu for different statistics and metrics um about the devices you can install pi and email has like a driver side component which comes along with the nvidia driver and then you can install the python package uh from pips so i've already installed this we have pip install pyenvy mail to get the package i can import this and i have to init the package as well so it's kind of instantiating it creating communication uh with the the driver we call this once uh whenever we kind of uh want to use use pimp mount so often it's done immediately after import we can now query things about the system and about the individual devices so let's start by just saying what is the system driver version so if we call pi mbml.mbml system get driver version it's reported back i'm running driver version 440.100 which just happens to be you know the latest driver that i that i've installed um we can also query what is the mvml version um if you want to kind of dig into to what features are supported in mpml this is mbml version 10 that's come from driver for 40.100 now we can also query information about the devices so first let's say how many devices have we got device get count so i'm working on on a data science workstation it has two quadro gpus in it so device accounts come back to say we have two two devices available to us we can query each one of these devices independently for different information about what's going on how much is being utilized how much memory is being used etc we do this by first constructing a handle this is kind of like a pointer to the to the device and you do this by by passing in an index so i have two gpus here um so i there i will have two indices zero and one depending on the number of gpus you have in your system you know this is just a zero counted array so you create your handle uh using device get handled by index um and this will give us back this you know this this pointer object but we can then use that in in further calls to to specify which device we want to query so we can do uh get device name and pass it the handle and it should report back that this is a quadro rtx 8000 gpu that's kind of like a property of the device it is that is its name but we can also get metrics about what the device is doing right now so for instance i could do device get temperature and specify the handle i also have to specify which which temperature sensor i'm interested in and i think in this case there's only only one but we still have to specify the index uh the temperature sensor so i'm gonna i'm gonna specify zero here and i've wrapped this up in an f string just to add you know degree c to the end if i call this it's showing that that gpu is 33 degrees c um so this is really nice we can kind of go through we can query things like a whole bunch of different things that you can you can ask of a gpu to understand what it's doing when you can do this all within within python what i'm going to do next is i'm going to collect like a load of information about each gpu kind of in one go and then format that in a way that we can kind of explore a little bit more so i'm going to import pandas and date time and then i've got this function that i've written here called get data for gpu you pass it the gpu index just kind of like how we we had to pass the index to construct the handle but this function is going to manage making the handle for us and then querying a bunch of things about the device we're going to return like a dictionary of of different properties that we've requested so we'll set the timestamp um just so we know when we've created this because you know this is a snapshot in time we've said before what is the temperature that's the temperature right now so we'll record the time that we've requested this we'll record the index of the gpu that we're we're referring to and uh get its name um i've just decoded the byte string here into a regular string but then we're going to query some more information about the device so we can do device get utilization rates and get the the gpu utilization property here which is is a you know not to 100 how much is this gpu being used we can also query uh how much memory is being used and how much memory that we have total we can do this by calling the device get memory info uh function and then we've got a used and total property here for accessing that we can also get information about how the gpu is communicating so you know these gpus are connected to the cpu via pci express slot so we can ask you know what is the transmits um speed and what's the received speed at the moment throughputs um at the moment we can say you know what is the the pci express like max generation supported of this card and what's his bandwidth limit and things so we can ask that kind of stuff um if your gpus are connected by mv link you can get information about the you know the the link connection directly between the devices as well um and then i'm also querying the temperature so i've got this nice function you call it you give it the index and it gives you back a dictionary of information about your gpu so let's run it now for my first gpu we can see you know the utilization is still zero because i'm not doing anything with it but we can see we've got some memory being used from from reading in our taxi data before um you know pci express isn't doing anything because the gpu is idle but we've kind of got all this back in a nice dictionary some nice information about the gpu so what i'm going to do now is i'm going to take that function i want to call it for each device i have available in my machine so i have a list comprehension here it's just going to range over the device count and call get data for gpu for each each device so you know this will work if you're i have a machine with two gpus in you may have a workstation with more you may be running on the cloud where you might have one two four eight sixteen however many gpus this you know you can use this kind of approach to ask how many gpus there are and then get your metrics for each gpu this has given us back a list of these dictionaries and then we'll pass that to pandas to make as a data frame so if i call getdata here we now have this this nice pandas dataframe there are two quadro rtx 8000s that's in my machine index zero and one we can see they're using you know slightly different amounts of memory because they work through this dash graph um in order to you know work with the taxi data um and you can also see that the temperatures are slightly different here right 33 and 35 so i guess this is i don't know maybe an airflow thing in the in the case maybe one has got better airflow than the other slightly that's great we can go and grab all this information about our gpus but this is a snapshot in time this is what is happening in my gpu right this second um whereas what we really want to know is is you know what's happening over time how these how are these devices behaving so my next cells here i'm going to record some data over time so for 10 seconds i'm going to record 10 samples per second and i'm going to do that just by calling getdata and concatenating it onto the existing dataframe to just keep appending a couple more rows this is why we added that that timestamp before so we're kind of taking a snapshot um each time to create ourselves a time series of information and we're just going to sleep for 100 milliseconds each each time we go around so what i'm going to do let me just slide this back again so we can see both notebooks clearly i'm going to start this cell running which will record for 10 seconds and then on the right hand side here i'm going to run through the same the same thing again where i'm going to read the taxi data and put it into memory and do some things to it just to make the gpus busy so let's start that and then come over here and run those so while that's running we can just see we have 84 million rows here uh in the csv files this is just the yellow taxis for 2019 and it takes about a second to call describe on on the gpu for one of these columns on the left hand side it's finished now we've got this large data frame um we can kind of see if we already have a peak at the the utilization column here you know utilization starts out at zero and then as i called these different operations it's going up 26 25 29 etc um so now that we have this this time series of what's happened over those 10 seconds let's have a look at some plots to see what what really went on so we're going to use matplotlib and panda's built-in plotting here first we're going to do group by the gpu index so we can see a line for each gpu and we'll first have a look at the memory used so if i call this we can see um we started out hi because we'd already run that taxi notebook right so we already had this data loaded into memory but then i redefined it just to you know kind of make it do something so it unassigned that variable memory usage dropped and then it read in the csv file again and the memory usage grew again and we can do the same for utilization how busy is the gpu in terms of computation we can see early on um there's like a lot of usage and then it kind of goes down to a bit of like a spiky usage now i know that this happens because uh when we read in a csv file with qdf um gpus actually are very good at reading in csv files they're csv because it's a text-based format you have to do like a bunch of type inferencing on each kind of separation within the file and that parallelizes really well on a gpu so there's like a big intense burst of activity at the beginning as it's reading in this csv file and putting it all into memory and then we're asking it to do this describe over and over so you can see the gpu kind of bursting in between and you know the high points are when it's when it's doing the describe and the low point says when it's you know feeding that information back to time it or whatever so this is great we can like record what's happened and see how much our gpus have been utilized for different operations but what would be more useful is if we can see in real time what's my gpu doing uh right now so to do that we're going to use bokeh which is the same library that we use in the in the dashboard which we saw earlier on and we're going to create a plot in bokeh and then update it in real time so let's just import some something from bokeh i'm going to define a couple more functions here so i've defined get gpu names this is going to query both of our gpus calling that get data for gpu function from before we need to give each bar in our plot a unique name so we're going to we have a format string here which is going to take the name and the index and kind of pull it together into into a unique name for that device if i call this we should get a list of our two quadro rtx 8000 and quadro rtx 8001 i'm also going to write another function here called update plot data this is going to do the same thing it's going to uh for all of our gpus it's going to request that data dictionary from before but for now we're just interested in the utilization but in theory you know you could grab whatever information you want here we're just going to return a list of the the utilization rates for the the gpus so i call this uh here you can see they're idling at the moment so they're both a zero zero percent utilization but this list is is the same length as the the list of names and these are the two inputs that we need now to create a bokeh plot we'll create ourselves a column data source from uh the names and the utilizations uh we'll create ourselves a figure uh and just you know specify the height and the title and things like that um and then we're going to add a bar plot to this and say i want to display the gpu utilization as bars um here's the source data here's you know the fill colors and things like that and because utilization is showing a percentage of utilization we're going to set the y ranges to you know zero to 100. um and now we can show the plot in our notebook but i need to just capture the the target for that plot because we need to come back and update it in a minute so you can do that by assigning uh the output of show to a variable so we call this we get a plot here gpu utilization and we can see our two gpus neither of them are doing anything at the moment so there aren't yeah the bars are both at zero um but this is like a snapshot right we took that we recorded that information we've created this plot we put it on the screen but now we need to like continuously update this my last cell here is like while true update the the column data source with new uh information about how the gpus are performing um push that to the notebook to update this plot and then sleep for 100 milliseconds if i call this this cell is now going to start running this is blocking um and this gpu plot is updating now but we just the gpus aren't busy yet but now if i come back over to my my taxi notebook and run these same cells again we should see that plot spring to life as the utilization goes up great so you can see you know it's loading that csv now it's doing those describes both gpus are being used here and we've got like a good view on to on to what's going on um but we don't you know this isn't like a practical solution of you know you open this notebook and run it through and have this plot um this is why we've taken it this next step to create mb dashboard where we've taken code pretty similar to this wrapped it up in in okay server so it like runs as a standalone application i can actually i can do it from here so if i just uh interrupt this um run mv dashboard this is a python package that we installed but it exposes this mv dashboard bash command i call this it will start like a standalone okay web server and i can navigate to that in my browser so if i just head to the address and go here we can see we have like a bunch of bokeh applications running code similar to what we did before but these are plots that show memory usage utilization all of the gpu resources aggregated together all the machine resources so this is using psutil to get the cpu usage and the memory and things that we saw earlier on um there's also some plots for nb link and pcie throughput as well so you can see how the gpus are communicating with each other and with the cpu so you know if i open up gpu utilization um you know we can see that plot it's not doing much we have a look at gpu resources uh these are like time series line graphs um that are being updated as well and you kind of see a bunch of a bunch of gpu things uh in one go so this in itself is like a useful tool if you have a machine that's remote to you that you want to get you know view some dashboards to see what's going on you could install mb dashboard log into it run the mv dashboard command which will start the server and then you can open it in your web browser um but for us this is like the first piece to get in the jupyter lab extension uh working so as well as being able to run this from the command line you can install the jupiter lab front end extension which gives us this panel over here on the left of the little gpu icon we can see the list of these same plots um and this has a jupiter server extension so when you start jupyter it runs that bokeh server automatically and then in the front end you can see these buttons and when you click them it opens up a new a new panel which is basically like iframing in these these plots we looked at a second ago so if i open up the gpu utilization plot here um we can see that and the memory cgpu memory we still got some taxi data in memory let's open up all the machine resources as well that we could see before so this is great if you if you want to see you know i'm running some work it's a mix of gpu code and cpu code and i want to see how busy is the cpu how busy is the gpu where am i getting bottlenecked and that kind of thing so if i run my taxi data we should see these these plots busy doing things um you know they've kind of got some thresholds and things so you can see you know the bars go red if you're using a lot of gpu etc um you can also see the cpu was kind of busy there uh in in doing that describe this is uh mv dashboard it's great for seeing your resources on your machine either you know locally or remotely but because we use das to create our cluster because das can pull together um you know multiple gpus on multiple machines into a cluster we have some other plots that are useful for seeing what's going on so we have our task stream here let me pop this down the bottom just going to restructure my windows a little bit slide that over slide that over so we have a task stream um i'm also going to uh open the we had the cluster map plot as well before didn't we so let's open the cluster map and the task stream this cluster map is showing us a topology of our scheduler and workers in our das cluster if you're on on a larger cluster you would see more workers um and the task stream is showing us kind of how functions are being called uh over time on our two two workers here you get a rope thing so if i you know if i run this computer again we can see our read csv tasks going on here um all of our described here and you can see all the communication going on so these workers are communicating i think via pci in this case and under tasks are all being dished out so that's a great way of seeing what's going on um because das supports multi-node as well we can have a look at what uh if we if we ran das workers on multiple machines kind of aggregated them together into a das cluster dasc has got some built-in metrics for viewing what is the gpu utilization what is the gpu memory etc now the way that i've set things up here is a little funny um i actually have both workers running on the same machine so we're actually going to end up with double results here but i want to show anyway we have like a das plot for gpu utilization and gpu memory i'm going to run this again you actually see four rows because i've doubled things up with a couple of workers um but this is great if you have like a big cluster of machines you can get a good overview of your total gpu usage rather than what's just on your on your machine so that's it for for the demo i i hope this was useful you kind of saw these different plots that you have different things you have access to and how these things are built up from from the ground with with okay so future plans for this project um i showed that we can use das to view uh resources on multiple machines in the cluster but you're not necessarily going to be using das for everything and so while it's useful when you are using desk if you're using something else um we want to add support for multiple machines in in every dashboard so that you could you know maybe start mv dashboard on a few separate machines and then have another one which which aggregates the results into into like a you know a mega plot um we also need to make some improvements to the the dark integration as you saw there's a bit of doubling up they these two workers didn't realize that they're they're sharing each other's resources and things so there's a few things we want to do in terms of improvements but hopefully the tools are kind of uh ready so just to recap what we've gone through in this talk uh pi and vml is a python library for querying gpu metrics uh pi and vml can be used for building like time series and grabbing information uh and and you know then using matplotlib or whatever to draw graphs of your gpu usage but then to take that a step further mb dashboard uses bokeh to display in real time what's going on inside your gpu you can use gpu standalone as a bokeh server or you can use embi dashboard as a jupiter lab extension and view right inside youtube for the lab session that can also be used along with this lab extension to view what's going on in a cluster of machines so thank you very much for your time i really hope this has been useful um i wanted to share what we've been working on in terms of tools for visualizing what's going on inside gpus but i also hope that these open source libraries will inspire others to go and build other dashboards for other things that you care about that's happening in your workflow you know maybe you are training some models and you want to keep track of certain values as time goes on maybe you're writing those to a file or something you could build your own dashboards that are kind of taking in these metrics and displaying them jupyter plus bokeh plus whatever tools you want to use for gathering metrics are excellent for giving giving user feedback in jupyter lab so please do take those libraries and you know run with them and build anything you can thank you very much for your time and i hope you enjoy the rest of the conference