Maxime Liquet Interactive data exploration in a notebook with hvPlot | JupyterCon 2023 1

Key Takeaways

my name is maximilike and today I'm going to be I'm gonna talk faster now and I'm gonna talk about interactive data exploration in a notebook with AEG plot oh sorry I was a bit too far from the mic um so I'm working at Anaconda as a software engineer and I'm also part of the holidays open source project I'm going to start with the an introduction on holovies and HP plots the first thing I do when I discover a new project a company is that I jump onto their about sections and I want to know who they are what why they do what they do how many they are so this is what I'm I'm Gonna Give You Now I'm starting with holovies so all of this is an umbrella project of a few or actually now many different open source libraries that are dedicated to making data visualization easier better more powerful in Python it has a focus on on the user experience in the browser and that started a very long time ago about 10 years ago with notebook support and more recently with panel which is a dashboarding library with web apps and even more recently with the new parody jupyter light we have support for that except for datashader but I will come sometime in the future um Halloween integrates well with the by data ecosystem so many of these libraries they accept like objects from by data from pandas from x-ray panel can render a lot of a lot of different kind of plots from Altair from anything name it panel can probably render it and finally bokeh is not a project in in the Under the Umbrella of holovies but there are some tight relationships with bokeh bokeh for those who don't know is a plotting Library an interactive one and so yes there are some tight relationships with this project all of this has been a long time in the making so it started about 20 years ago with param para means not these libraries and foundational library that allows like reactivity and validating object attributes and that was started by Jim bedner who's not the founder of holovies let's say and his students when he was a professor at the University of Edinburgh and 10 years later he had a PhD student Johnny Stevens who was trying really hard not to do his PhD and he came up with Hollow views I was a way to display and analyze data they were running um they were running simulate simulations of like brain simulations doing computational neurosciences so they built this tool he started to build this tool to analyze the results of these simulations and he was soon after joined by Philip who helped him and started to contribute developing Hollow views so that was 10 years ago and and since then there's been many other new projects the funding model is I think it's interesting to know how these projects are all funded or not this except this one this first once before well what I should say is that this these three three people I mentioned the agenda Anaconda about eight years ago and then they started to work in a Consulting Group at anaconda and through Consulting projects finding organizations with which they share the same interest and the same vision in improving Python and data visualization in Python they've they've got contracts with public and private organizations to start these projects to maintain them to improve them so a panel was funded by an American public organization I don't remember its name now um Halloween has a growing number of contributors and and members I'll just share my example is that um I I was um a few years ago I was barely a python developer and um in my like I was describe discovering of about Python and I found out about panel and it just blew my mind because suddenly I became a web developer just knowing some python code some python stuff basically and I was like wow that's amazing and uh I started to become a contributor and later there was a job opportunity at Anaconda I was lucky to get it and so what happened to me happened a few times again to other members of of Halloween now so last year we had three new members and I think there's one there's one to come soon um the our group it's also interesting I think I think it's very common in Python but we have a background in science and engineering so I was a fried risk engineer before and I think that shapes the way we design our tools and we have a focus on I think this is General in Python but the focus on really helping users to solve their problems and and finally I think it's fresh from the oven so we we set up governments at the hall of his level before that it was just a bunch of people they knew how to work together they didn't really know they didn't need that but we decided that was a very good thing to do and we have a steering committee with members that are outside of our group and um I think we are I think we are we are about to be Affiliated to non-focus so this is great and this will open up some new opportunities we hope and now onto HP plot finally this is a high level library for data exploration and as you've seen we have many libraries in our ecosystem and we have found out that this is really the easiest way for you as users to enter this ecosystem when it comes to data exploration if you just want to build like some dashboards and they are not so much like data oriented or maybe you want to do it your way a panel would be your entry point but when it's data exploration there would be more HP plot and HP plot has three main features so it has dot HP plot which is like a versatile and a bit more powerful banners like Dot Plot API then it has that interactive for building interactive pipelines and then it has a new feature that is a low code data Explorer I'm I'm gonna give you more details about that I'm going to start with interactive plotting um H3 plot can do a lot of protein it's in the name but I really this presentation I'm going to focus on the interactive part um so the way each report works is that you have all these data structures from the by data ecosystem and so it supports all of them all of them that are listed there and then your interface as a user is the Dot plotter.hv Plot API that is the idea is that and I think it's a good idea is that the Dot Plot API for users who are not necessarily software developers is very good they don't have to deal with these complicated object so it's multiple calls now it's just one call and they create their plot and it's rest just right next to their data structure it's very handy and if you have you spend I bet you have used that plot already um so we believe it's a good model and we have built on that and then what H3 plot does is that it doesn't do a lot of hard work actually it delegates the hard work to all of you datashader Geo views that are really doing uh like messaging the data and and doing many complicated things and then this libraries themselves they don't do the protein hard work they delegate that to Volcan might broadly or partly there are support for these three and the default one is bokeh and is the one that has the best support [Music] um now a short introduction so what I'm doing now I'm just giving you a short introductions and and then I decided uh I really didn't want to do live coding but uh then I just did it so I I'm just gonna do that after um so an interactive data pipelines uh interestingly the way this was created is that there was an issue that was opened by uh I think Tom Nicholas is an x-ray car maintain a maintainer and he opened an issue to implement a feature in xra so that to create interactive pipelines to better to inject widgets to better analyzed to to make it easier for users really to interact with their pipelines so there was a long discussion in there with Philip and Jim and and what it ended up being implemented really in energy plots so there is support for a few different data structures and there's also so the way it works is that you are going to inject widgets in in your pipeline so it supports panel widgets of course but it also supports jupyter widgets and it works well with DOT h3pread and the last feature is this I call it low code data Explorer I don't know if it's a very good I don't know it's me trying to do some marketing if it's probably bad um and so this is very this is like the newest addition to HP plot so for now it only supports pandas but we plan to add student support for our x-ray it's top of the list and then geopenders and others uh what you will get with that is a panel based app that you can just display in your notebook so as I said it's pretty new it's not very it's not yet very pretty and you'll see that so there's always work to do and it's it is inspired by and I think we can draw more inspiration from other data viewers such as perspective of our purchase yourself superset if you know about them so that is [Music] um now onto the live demo so how long I have about 10 minutes I have to be fast so I'll start again with interactive protein and now you will get some more information about it so I'm just going to load some data and um just so you okay the regular pandas Dot Plot the way it works you call you have your data frame Dot Plot dot scatter and then you give some parameters and what you get is not interactive you cannot do much with it it's already great but it's it's a static plot um the way you use dot H3 plus is that you import a tripod.penders you import the data structure basically you're you're interested in and then you call a DOT HP plot there's a way also to call Dot Plot but we I think we recommend a way to call that HP plots it's just more explicit um and so here I have I have added the parameter a parameter that is not in pandas API that is by that is going to colorize the the points with respect to one column in this data frame and what you see first thing you can see is that you no longer get a matplotally plot but you get a bokeh plot this is how you recognize bokeh it has this little logo and so it's interactive you can pan you can zoom you can with how to find but actually you can zoom on the axis this is very very useful and you have hover enabled by default and you'll see there that by default you see that the columns you have specified in the in the call above another thing you can do is that you can mute some colors so in terms of exploration and interactivity it's already it's already great without much effort you can do more than that so however you can extend what you see in the hover just using hover calls by giving it a list of columns another another thing you can do if you set Surplus to True your plots instead of like instead of getting just one plot they are going to be exploded given that Dimension given this the species and another thing you can do is that instead of buy you can use Group by it's a the name is a bit similar it's unfortunate we haven't found yet a better name but what it is going to do is that you are going to specify here some other dimensions you want to explore but instead of like being really in the plot they are going to be rendered as as widgets which then allow you to um you know just drill really into your data and explore some subsets so you have like already with that this is like a small data up already um difference between uh between a tripod and Dot HP protein that part of pandas is that the re the objects returned by dot H3 plot are all of these objects so here this is a scatter element that has these dimensions and all of these objects you can alter them you can you can play around with them so by default you wouldn't really do that but in this case it's useful because all of these objects can be composed so here I'm gonna use I'm going to create two objects and using the the plus operator I'm just gonna lay them out next to each other and because they have the same X Dimension they are they are actually linked so again this is all about interactivity this is this is to help you to explore your data and you can also overlay two plots using the star operator um a more advanced feature that it hasn't found yet the place in HP plot but I thought it would it would make sense to show it to you now because it's all about interactivity it's linked selection or linked brushing so when you enable that you you have some more tool here on your toolbars and you can do selection over your data and you're gonna see like in any other your plots are connected so you're going to see that selection reflecting in any other data sets so it's useful for instance if we want to check like okay these higher values how are they how are they distributed over this other dimension and finally fight you plot there is support for large data and interacting with large data this is using data Shader so here I'm reading a data a file that has about a million rows it's not crazy but I think my bro my my laptop could display that but maybe um yeah it would take a while probably um and what you see so it was displayed very fast and also you can see if I zoom in the data is going to be re-rendered so really you can start with billion points and then you can zoom in and you're still watching like million points but it has no problem re-rendering it so you can interact really with large data which I think it's usually very very hard now on to interactive data Pipelines so I'm going to load a data set and now I'm going to use an xra it is a simple data set with one variable that is air it's the air temperature and three coordinates to enable interactive it's the same way for as for enabling a H3 plus you just Import azure.xr in this case and um what I mean since the beginning by pipeline because I don't know it doesn't mean merge pipeline really this is this kind of stuff you do that you have your object and then you're gonna call it you're going to transform it in multiple ways by calling methods or adding some stuff to your object um so this is a very simple pipeline I'm just selecting the first time stamp and uh and Computing the mean over the the 2D array and I get this data right with just one value really that is the mean temperature um and maybe I want to see like okay that's the first time stamp that's the second one and what it what you do usually in the notebook is that you change values and you re-execute stuff very often this example is a little bit contrived but you you get what I mean um I think there's a better way and so dot interactive is a better way um and that interactive um how to use it you start usually by creating widgets so instead of um I want to vary these I want to vary the time so I'm going to create the widgets whose value is going to be is gonna like emits integers really integer values because this is what I want to replace time with so I'm going to create this um simple int slider that is from panel and that has values that are allowed between 0 and 10. I to create a pipeline you just call Dot interactive on your object and you get this object that is an interactive object but actually it has all the um it is mirroring the API of the xray object or whatever object you're playing you're playing with so it feels like your original object it's it's not really it's not much different and here is really what the important part this is this call is very much like the static pipeline I had above instead of zero I am just injecting the slider and what happens is that the output is displayed but on top of it is the slider and when you change when you move yours I move the slider around the pipeline is recomputed and the output is rebranded this happens every time you change the value uh so this is really that interactive I could just stop there but I will show a little bit more I have to be fast so it supports plotting uh here it is not with that HP plot so it's it supports anything basically as an output really so it's just to show you it also supports plotting so this is here with um just Dot Plot of X array and I have changed the the widget to be another morph and widget of course that HP plot is supported so that works the same way to be faster um I showed a very simple pipeline but they can be you can have more complex pipelines and for instance in this one what I do is um I something interesting there is that the pipeline or the object really I I subtract it like it is a bit weird when you look at it a subtract the widget but it actually works so it allows me to change the temperatures from from Kelvin to Celsius um I am going to skip that because I really have no time uh and yes when you have these small data apps so really maybe they are for you to explore the data and when you're done you can just remove them even from you in from your notebook or you can keep them around but something or you can send that to your colleagues but something you can do is that you're really not far from creating a proper like small website you know like I was amazed about panel many years ago well this is the kind of feeling you get okay I have this small really small this small web app this small app in my notebook actually actually I can make a website out of it that doesn't look so bad suddenly your your web developer yeah I don't know I don't know maybe that doesn't really make you happy oh that was a big and finally so this local data Explorer so the vision with this this one with this feature is that um sometimes you really don't know about your data you don't know what you you have no clue and so you will just execute like you will you will just do like you will have a million sales after about 30 minutes in your notebook instead what you can do you can use the Explorer and you can you can play with your data you can create different kind of plots and change the parameters to create the plots and you get some sense about how your your about your data you build some insights and so here I'm going to show you how to just recreate the blood we had at the very beginning um I think there was something like that yes I'm gonna okay that worked so well here I knew what I wanted to do but um usually you don't really know what you want to do and you explore and you play around with this with this app here but it's it's okay you're there maybe you would just make a screenshot and then you go and you do some other thing but if maybe you want to capture that in some like meaningful way or you just want to get rid of this Explorer it's not the purpose of your notebook your notebook you're solving a problem you don't want this big thing to stay in there and so there are ways to get out of here the Explorer this object has two methods that are useful there are settings that is going to speed out the dictionary of settings you've just changed and you can give that directly back to um to HP plot so you can save that maybe in a file or in the Json file or somewhere and there is another one that I think is also quite nice it's broad code it's going to speed out a string that you can copy paste in your notebook and so I've just copy pasted before and yes it's going to recreate exactly the same plot so once you're there you just you can just get rid of the Explorer it's fine you have used it you're you're happy you're happy with your plot and you keep going with your line with your life and your analysis um I think that's it for me yes yes that's about it so thanks everyone foreign