Kunal Bhalla- A Notebook Style Guide| JupyterCon 2020

Key Takeaways

hi i'm kunal i'll be walking you through a style guide for jupiter notebooks i hope you find it useful for writing readable and maintainable notebooks particularly if you're new to jupiter if you're an old hand i'd love to hear what you thought about it and if you had any feedback before we dive in i should introduce myself i've been playing with notebooks for a fairly long time i've used them for everything from debugging android applications to actually diving into data that said i don't think i'm particularly qualified to write a style guide but i felt that one would be very valuable and i decided to take a stab at it anyways let's start by talking about why we need a style guide in the first place jupiter is extraordinarily flexible so having a style guide lets you structure your notebook in a way that you hopefully don't regret when you come back to your notebook it also lets you maintain consistency across your notebooks which can be really valuable in a team so it's very easy to pick up someone else's notebook and ramp up on it because you have certain expectations around how it works i'd recommend applying the style guide for any notebooks that you or someone else plans to use again in the future otherwise it's not worth wasting the time polishing it up and finally i've tried to keep the style guide as client agnostic as possible we are spoiled for choice these days and you can use notebooks using vs code emacs jupiter lab jupiter notebooks hydrogen and all these other lovely clients so it should work just about anywhere [Music] let's get started the most important point which is why i've put it first is that you should be very very careful managing state in your notebook i'm not the only one who worries a lot about this joel brought this up really well in his talk from 2018 where he talked about the tons of hidden and hard to reason about straight in notebooks the way i try to mitigate this is to only have global state that represents the sole purpose of that notebook for example if i'm doing a data analysis then my global state is going to be the data frame that contains everything needed for the analysis and nothing else if i'm building a notebook that is a tool then my global state is the input provided to the notebook and nothing else all the other functions are happening as pure functions that explicitly define their inputs and outputs that way i don't accidentally mutate the behavior of any other part of the notebook without knowing about it i'm going to try to make this a little more concrete with a dramatic reenactment which is also very very contrived so it might not work that well let's say i have a really slow function that fetches some data for me and i want to plot the square values of that data for some analysis i fetched the data i copy paste in some code from stack overflow to plot a graph and then i get a nice red line all right now i also want the cubes so i duplicate the cells i had i change the value of exp to 3 i get rid of the data fetch because i don't want to spend another hour waiting for it to download and then i run the cell then i get a nice cubic graph i publish these graphs and share them with my colleagues and they give me some feedback square and cubic look really similar i should change the color of at least one of the graphs to distinguish them so i come back i go back to my original cell i get rid of the line that fetches the data because i already have it in memory i add color equal to blue so that i plot a blue graph and then i run the cell again and i publish my notebook and go on my way unfortunately i was depending on exp which was still set to three and if i didn't remember the notebook and if i was being careless which i normally am i would have just not noticed this problem if i was following the style guide instead of having the duplicated code and the random exp value floating around in the ether i would have extracted an explore data function that explicitly defines that it depends on the data being passed in which is xs exp the value that is being raised to and a color that is optional that way i can come back modify this and play around with it i know exactly what's going into the function and i know what i'm going to get and it's much harder to break there are more advantages once you have small functions throughout your notebook you can have tests right next to those functions and be sure to debug the notebook very very quickly if anything breaks you can isolate it much faster than otherwise another advantage of having tests right next to your functions is that you can iterate on them with a very tiny test cell driven development kind of workflow i like to have the test right below the function definition and then i keep using ctrl enter to re-execute the function and the cell as i iterate on the function score once you have small functions that define your notebook you want to club them together into meaningful structure the way it adds structure to a notebook is to have lots and lots of headings if you have a good client then you'll also have a table of contents extension and that makes it much easier to jump around the notebook which is very good for long notebooks or tools you have a well-structured notebook that has lots of tiny tested functions now you should make sure it's not sloppy and that you can execute it from top to bottom that reduces a lot of cognitive load of running the notebook you don't have to worry about any special magical ways to run the notebook instead you can just press a run all and it will execute cleanly this is also the way i try to measure if a notebook is sloppy or not this does break the fact that you can't do pure literate programming anymore you have to define functions before you use them and you can't rely on some form of dangling to bring them back up but on the other hand it forces a slightly different style of programming which is programming bottom up now something paul graham describes really well and in it you're basically building increasing layers of abstractions towards what you need to accomplish and you can generally share those abstractions very well with util as utility functions now that you have a well-structured clean notebook that you can execute simply you want to make sure that you can execute the notebook in the future as well and that means you should document all the dependencies of your notebook notebooks have a lot of dependencies you want to make sure that you've documented the dependencies on code maybe you capture all the modules depending on with the requirements.txt you need to make sure that someone else running the notebook can fetch the data for it so document how you got the data or possibly include a dump of the data right next to the notebook does your notebook need a quantum computer to run on maybe it needs 20 gigs of ram you should at least call it out so that someone doesn't spend hours debugging why the notebook is failing does it need internet access is it relying on websites are those websites guaranteed to be up in the future do you need to give alternatives to fetch the data from those websites in some notebooks you might even want to go as far as controlling randomness if you want the results to be exactly the same and this might be particularly valuable for something like generative art now that you have a notebook that you know will run in the future it's time to polish it a little bit if you look at any english style guide the first thing they'll tell you is to remove any superfluous words and that also applies to the rest of the notebook if you have code you want to make sure that the code inside the notebook is towards the point of the notebook if there are a lot of utility functions they're just going to act as noise so you might as well extract them to an external python library for language of course just go and strip superfluous words and follow english style guides and also for the outputs a lot of libraries will throw a lot of logging information in their output cells you should decide if you actually need to show it to anyone else reading the notebook or not and set the log level accordingly you can even get rid of outputs completely by using the percentage percentage capture cell magic notebooks are both pros and code so you should make sure that you take care of both of them for pros you should follow really good english style guides i'm not going to waste your time trying to talk about english and instead i'll just point you to really good books on writing well is my favorite and of course there's always the classic elements of style and they'll have a much better return for your time investor similarly you should be very careful with the code you're in the notebook but that doesn't mean you forget all your software engineering principles so you should make sure you follow pep 8 which is why this is 0.8 all the lints i had some inline documentation follow naming conventions and all the other things you know from good programming practices be good with your abstractions i like to keep them minimal because in a notebook you're unlikely to reuse them apply kiss don't over abstract or use you ain't gonna need it notebooks also introduce additional structure because you're running them top to bottom and there's some space related relationship between cells so i try to keep cells that are close together highly cohesive and cells that are far apart lower coupling that way i can modify parts of the notebook without worrying about the rest of the notebook and if i'm looking at one section of the notebook all of that makes sense together again i'll try not to waste your time here and instead point you to a book that will give you a much better return for the effort you would invest and here i'd recommend go read being the pragmatic programmer which is an excellent book taking it from the top be very careful about how you manage global state only modify it with pure functions and transform it instead have tests throughout your notebook add headings and structure your notebook carefully make sure that the notebook can be executed from top to bottom make sure that the notebook can be executed in the future and document all your dependencies eliminate any clutter in the notebook and polish it be careful with the language in the notebook follow english style guides be careful with the code in the notebook and follow engineering best practices only take all of this effort when you plan to use the notebook again in the future there are several excellent notebooks online some of my favorites are by peter norvig you should check out his pie tues they're excellent both in terms of really good notebooks as well as really interesting things to read about fast dot ai's books are obviously amazing as well and they've taught so many people deep learning there were a bunch of references i checked out online before writing the style guide i particularly enjoyed reading the space telescope science institute style guide and of course clean code and jupyter notebooks was excellent as well i also wanted to thank my teammates for reviewing this presentation and giving me feedback thank you for your time if you have any questions comments or any advice please reach out to me on twitter or drop me an email this presentation started as a blog post a really really long time ago and i'm hoping that it evolves further based on what you tell me thank you you