David Schmudde - Dr. Schmood's Notebook of Python Calisthenics and Orthodontia| JupyterCon 2020

Key Takeaways

hello my name is david shmoody i am excited to be here at jupitercon this year even if it is virtual i'm really looking forward to seeing what people are working on what sort of ideas are floating around i'm a big proponent of computational notebooks myself i am a computational artist i am a programmer and i live in berlin now today what i'm interested in talking about is how we're going to or how we can leverage immutability to make jupiter notebooks that are easier to read and run now some of the older people in the audience are watching this at home might recognize the source with the inspiration for my title here it comes from dr dobb's notebook of or dr daub's journal of computer calisthenics and orthodontia this is a journal that was published by the people's computer company and its first editor jim warren had a cyber libertarian view of the future where he saw that maybe the computer was a way to disseminate information and information is power then information and computers were a way to empower individuals particularly he saw these journals as or sorry these machines as a way to organize and process information which is in a way what we're still really grappling with with jupiter notebooks because jupiter notebooks are certainly a way to organize information in a readable format and i think ideally for a lot of us are also an ideal way to process information from top to bottom when people read it now the state of the art in looked like this this is dr dobbs you can see that it's printed on some kind of inexpensive newspaper print in part because of course it was bootstrapped with not a lot of financing it also has this peculiar layout where sometimes it is um portraits or sorry landscape and sometimes it is portrait sort of depending on what sort of information it is trying to convey the magazine will switch its format so i want to get back to dr dobbs at the end of the talk we'll lead up to it again but right now i want to talk about the core problem that i want to solve or want to help solve which is the problem of hidden state in jupiter notebooks it is a common sort of irritant or complaint about the format that it's really easy to lose sight of state that say is created in one cell and then executed in another what i would argue is that it's actually a problem deep deeply rooted in python itself that jupiter notebooks inherit and as we explore the problem we'll see how perhaps overcoming these issues is not so difficult if we choose the right tools so starting from the beginning we have to start with assignment itself in python assignment is what's going to create states that's going to be executed and that's where our pro that's where the issues lie now in python we do assignments with the equal sign and this is already a little bit overloaded because equals originally meant when we were even before we knew how to program originally meant these three things it meant something that was had the qualities being reflexive transitive and symmetric if a thing a is equal to b well then b for example must be equal to the thing a that's equivalency and so this hidden state problem i think partially is a conceptual syntactical problem because of course we're munging sort of these ideas of equality and assignment and so now equality actually doesn't look like equality equality is a double equals an assignment is a single equals now in our notebooks we can manipulate data types like strings in this way where we set or assign the variable a to a string 10 we can assign variables to other variables using the equal sign and if we modify the one side one operand of the equal side then we get something that is clearly not equivalent a is clearly no longer equal to b and as we might expect actually we have a situation where the identity of a word exists in memory is not equal to the identity where existing memory for b a quick side note on this environment i'm working in most of this talk will be done in next journal where i'll be running cells in this notebook environment which is jupiter compatible it adds a couple nice wrinkles to the game where it is automatically version controlled so it's immutable from the get-go and of course and also when you share notebooks they become they come bundled with their execution environment which is really nice and those execution environments themselves are also immutable which is great if you share an environment in a notebook and someone changes the environment won't necessarily change the original now back to python see these are mutable data types strings um tuples and bytes so immutable data types do exist in python but problem problematically so do mutable data types like lists so running this cell i'm setting a equal to a list that contains a string 10 and then i set b to a i sign that and this does have the qualities of equivalency that we're used to where i change a on one side i did i did not update b and b is changed on the other side so these are actually indeed a balanced operation and indeed equivalent they are actually even in the same part of memory so python has several mutable data types and lists are just one of them and so are byte arrays [Music] so this syntax then is changing the state as we move as we as we go along and it looks kind of like a quality it kind of doesn't behave like a quality but sometimes it does and a computer scientist uh niklas vert did actually make a case for a different kind of assignment operator the colon equals he felt like that this equality assignment operator is a quality assignment operator which dates all the way back to fortran was a giant mistake he thought he basically argued that it overthrew a century's century-old tradition that led equals the sign denote a comparison for equality the way that we have it now of course the left operand which is a variable and the right operand which is an expression of course are not made really to be equal so in python x equals y does not mean the same thing as y equals x i really like the colon equals assignment operator that's championed here it is in atta 95 and we can see that we have a and b which are initiated as integers set to zero and then we have a test for equality of a equals zero a little bit later so i kind of like this because it helps us reason about state and so if i read the first line here i say set a equals to 42 or a equals 42 that's just naturally what i would say right but the problem is if it really was equality then if one side of the equation changes uh the state then the other side of the equation must change its state right but that's just not actually how it's working right and so what's more precise is of course a is assigned the value of 42. because 42 is immutable it's always going to be 42. it never changes from 42 and if a does change and is assigned to something different well then 42 is garbage collected or whatever whatever happens under the hood when that a assignment is changed so there is one other way to think about this too that's quite popular and it's an enclosure has has a way to think about assignment and that's through define and here we have a question or a setting so we're going to define c we're going to assign c but notice that there's nothing that it's actually being assigned to because it's not assignment see assignment is actually the relationship or the correspondence of the correlation between different entities whereas definition is about getting to the essential qualities of the thing and so c doesn't actually have to be assigned anything but because it's it's not an assignment in fact if i take a look and see what it is c is actually nil okay so it's nothing it's nothingness is actually its most salient property right so that's how it works in closure and enclosure the defined character the define function uh or macro the def macro works uh just the same as we might think of assignment in python a equals define a is 10 and then we have this to find b as a and when we mutate a then they're actually two different things to be clear in closure i'm actually not mutating a it's actually a whole new property but that's beyond the scope of this talk problem that the point is is it is it looks quite a bit like python like what we expect here so so the thing about the way we think about assignment is that it also is it matters when and where things are defined so basically it matters where the scope of the definition happens and so this is where referential transparency comes in and where that can be very valuable and so in this first example here any seasoned python programmer is going to know that this is this is not very good code in fact i have a global state here of x set equal to one and then when i run the function inc i'm going to update this global state and when i compare the functions of course they're not the same they're not equivalent because i run the function once i run the function twice and each time of course it's updating this x so the results of the function are different this is very poor programming style and for beginners it's just something that they have to learn that they shouldn't be doing this is a little better where i have a global state y and i have the ink function again and the ink function is modifying y but not the global y it's a local a locally scoped y so i'm defining it within a different context within this new context when i test for equivalency between the two ink statements aha it is true as we expect but really that's kind of hard to reason through and actually the best one best way to go about this is probably to define something that is unique in name as well you know jupiter notebooks are really written for human beings to read and only incidentally for computers to execute so we should pick good variable names of course and we should pick distinct variable names and here we have something that isn't globally defined at all so it's very clear that when i run these two functions and i pass in a literal they're referentially transparent meaning that the function and the value that it returns are essentially you can replace one with the other and of course their ink one is always going to be equal to inc 1 in this case and the problem really stems here from the fact that this whole business can seem very innocuous and so we have an i have a anonymous function here that is just a simple 1 plus 1 equals 2 and i have another anonymous function which is seemingly very innocuous 1 plus y equals 7. wait where did that y come from oh it came from up here okay oh this particular cell is difficult to read because this y is different than this y and i'm actually referencing this y and this cell down here this is this is very difficult to reason through right so we have to really think about how we use scope in the way that we use scope for example enclosure is we have these parentheses around a locally defined variable through a let binding and the interesting thing about the let binding is it has no global namespace it has no is never created in the global context it really is just in this closure spelled differently c l o u s c l s o u r e and so it's really defined just within these two parentheses now i do have a globally scoped a if you remember all the way back here in a different section and so again it's quite difficult to reason about where this a came from and so this cell in this function sorry this cell and this let binding is carrying all its context with it and these closures are kind of nice because they syntactically kind of wrap everything neatly into one bundle and are only as good for as long as that function is living and i think this sort of encapsulation is is a great inspiration for what how we can think about cells and jupiter notebooks if we can keep cells of course really ideally encapsulated and pure with very few side effects our notebooks will be much easier to read this next cell is just an example of how the scope works within closure where i have this eye which has never been created globally and if i actually try to reference some sort of global definition of it it's going to give me an error message when in fact the only eye that i can reference is within the let binding itself so those are some of the ideas that underpin the problem at hand but what i'm really interested in is how this can be applied practically and one of the places where this is immediately apparent is in is in pandas pandas actually makes an explicit statement about immutability in its documentation so how it handles state and in general they say we favor immutability we're sensible so the question is what is sensible so i'm going to bring in this data set of artists that are in the collection at the tate the tape gallery in the united kingdom and this is a set of this is global state that i've created and python or pandas tries to help you here because they're not going to make it easy to manipulate this state because of all these trappings so if i print sort if i print the column headings of this table i have id an ascension number an artist an artist role and artist id okay if i drop one of those columns and then print the headings again i actually get the same set of values even though i dropped the columns that's because it's immutable now pandas data frames can be mutated upon explicitly if i use this in place equals true parameter so in this case i print the column names again we see them here i drop a column name and then i print them again we see them here now the problem quickly presents itself if i run this twice well now i this cell no longer works because i've actually already dropped the column that i'm trying to drop so this is a problem with immutability with mutability but i would argue that this is not even a very big problem it's a it's a kind of an easy problem to catch compared to what we're going to look at i think what's more innocuous and more difficult to find is if we actually mutate or change data within a data set so here we have the data set the the table with all the headers and then all the data and here's the acquisition year and i'm going to mutate this act or look at this acquisition here for the first piece of art in the table if i do that the acquisition year of this robert blake painting is 1922. can increment that number pretty easily with this pure function and of course it's a pure function so it's always going to be the same however um i can actually mutate this so if i have this function where i increment the value in that cell and i test whether or not it is greater than or equal to 1922 sorry less than or equal to 1922 it's going to be true um then if i mutate the value and try it again [Music] well then 1924 is not greater than or equal to 1922. so the first time i did not mutate it i did the test and then i did mutate it and i did the test okay so this same piece of code is going to give us different outcomes depending on where we run it which is not so much a problem necessarily until you run it across a lot of data until you are taking individual values in a table and looking at them and poking them and changing them and it becomes quickly just as difficult to wrap your head around and to actually ascertain what is happening as an excel spreadsheet so we want to avoid this as much as possible while we're mutating values within a table and a way that i might recommend doing that is instead of working with a technology that says some immutability is is preferred you might consider a different tech or different library that actually guarantees immutability throughout the entire process so in this case i'm looking at static frame which is very similar to pandas it has some syntax and some tactical difference but adapting it adapting to it is not very hard if you're familiar with pandas and so i'm going to take this exact same exact same table i've been working with i'm going to then actually take a look at the same sort of transaction i did before and even if i increase again the acquisition year this no matter where it happens in my code this test to see if something is less than or equal to 1922 is always going to be true regardless of where it happens in the notebook now that might seem really limited or limiting to a programmer but in fact i would argue that there's a lot of power here it puts the burden on us to really think about to really think about what mutating data really means because we can essentially mutate it if we give it a new name so in this case this code is very similar to the code above it the only difference is i'm creating a new data frame with a new name that is pointing to the mutated data set so if something changes its essential qualities is it the same thing if not then what is the name we could give the new thing i don't know if df update is a very good name but for the purposes of our demonstration i think it's a fine name so i update the um i sorry update the data frame not in place but i actually increment a value in that data frame and the return value of that gets assigned to df updated that didn't change the original df df itself remains the same but the return value of df update df assign now that has changed and that is what's going to be assigned to df updated so we give something a new name when we mutate its qualities now back to dr dobbs so dr dobbs when they they shared information they they shared it in a format that looks kind of like this now this is still actually i would argue pretty readable code and so this code i'll bring it full screen here this code is actually an implementation of a for loop so you would type this code in and in this code you would once you execute it you would get a higher level function for loop that you could iterate over a data structure and um i would the there has a lot of literate programming sort of attributes it has a strong opening is heavily commented and so even today i can without knowing the assembler code very well or at all i can get some kind of sense about what's happening as we're poking stuff in memory and replacing stuff and putting stuff back in memory etc etc now i would argue that programming today hasn't actually changed much since it has since it was in the 1970s back then of course you had to type in programs but now if you are getting stuck or you're trying to learn or something's not working you go to stack overflow and cut and paste program programs what we really want are programs that come with their context and that can run instead of cutting and pasting or typing in we want runnable software that can be run sort of in in any point in the future so in order to get that we need we need immutable contexts we need cells that are referentially transparent and we need to make sure that cells that have side effects are very clearly very clearly and carefully attributed as such and that's one of the powers of these notebooks is the fact that you can actually give some kind of context to when you are actually mutating some kind of warning to when you're actually mutating values so we'll default to immutability wherever we can especially globally the syntax that we're going to use i think is more powerful if we are using imperative constructs like what we see in pandas and like what we see in static frame rather than declarative context like the for loops from that dr dobbs and finally we should publish all of this stuff together with our comp with our environments with our data so these notebooks can be run okay well thanks a lot for your time today it is a pleasure to talk about this stuff please feel free to contact me at any of these places to chat more about computational notebooks be a lot of fun okay ciao