Why data scientists love and hate notebooks: velocity and validation

Key Takeaways

totally so this is this is great now we're getting back to the three most important aspects um that determine the success of operationalizing and deploying ml velocity validation and versioning you make it very clear that I'm going to actually quote you um High Velocity um means creating many versions in other words having high velocity means drowning in a sea of versions of experiments right so I'm envisaging there some Pareto front where there's attention and trade-off between velocity and uh versioning but then you also mentioned there are synergies between velocity and validating early so if ideas can be invalidated in earlier stages of deployment and overall velocity um is increased one more thing you mentioned is creating similar development and production environments exposes attention between velocity and validating so the development Cycles are more experimental move faster than production Cycles however if the development environment is significantly different from prod it's hard to validate ideas early so we have this oh there's some sort of triangle with synergies as well so I've mentioned a few but can can you just speak a bit more to the relationships and correlations and causations between these three incredibly important things yeah uh uh um I think the the knife the anecdote there is on Jupiter notebooks um for a long time we had all been seeing each other some people so excited a bunch of different notebooks some people absolutely hate to Bruno books everybody has strong opinions everybody wants to give a monologue on their opinions on Jupiter notebooks and I think for me it's been like so many years of hearing this kind of over and over again why people love it or hate it and I wanted to know why it was so polarizing um and it was very satisfying to me to hear this uh or to kind of frame it as this um kind of where do people lie on the velocity and validating Spectrum some people want to move fast and break things in the Facebook speak and they're okay with that if they can fix it some people do not want to move fast they want to make sure that there's no buggy models they want to make sure that everybody can review each other's work yes that hinders velocity um and kind of like it's really hard right some some people want Jupiter notebooks because they can go fast some people don't want data scientists to have they don't want data scientists to go too fast because then maybe certain scientific principles are disregarded maybe things are irreproducible I don't really know but it was nice to frame it this way because there's no right answer right it's like where do you personally lie on the Spectrum and when you run a company or you run your team like what is the ethos that you want to create um or like where do you guys want to lie um amazing that was interesting to me this is incredibly useful I I mean this framework so what I'm hearing is in this framework of velocity validation um and and versioning we can look at people who uh prefer Jupiter notebooks and in this framework they're essentially prioritizing velocity whereas people who are strongly opinionated against Jupiter notebooks uh prioritizing validation and versioning or mostly validation or how do you how do you think mostly validation um I think of it as validation more because it's like how do you uh what like how do you make sure the development and production environments are as similar as possible um whenever they're there there's a discrepancy there is chance for books um so you can remove the need to validate a lot when promoting from Dev to prod if there is no real like environment change from Death to product one great example is like sometimes people will iterate locally and then deploy to the prod service in Cloud that it's a huge environment mismatch so you need to do some sort of big validation I don't even know I don't think people have solved this problem of like making sure there aren't bugs in business and as much of environments um but again right like why why is it so separate like a completely different Hardware not even in the same like Cloud um crazy I I think this also exposed to me that like I am now feeling somewhat opinionated it's not the corrective there's no correct opinion right it's like what is the opinion you want to hold and you want to prescribe for the team that you are running yeah absolutely and a question around notebooks for example is if they don't have affordances which we'd like them to have maybe as tall Builders we build them into them as well right it isn't yes of course how do we want to build the future as well right yeah yeah as you point out