Data Pipelines at Zymergen with Airflow, w/ Erin Shellman - #41

Key Takeaways

[Music] hello and welcome to another episode of twiml talk the podcast where I interview interesting people doing interesting things in machine learning and artificial intelligence I'm your host Sam charington this is the third and final show in our series of podcasts from the recent Wrangle conference as you might know a few weeks ago I was in San Francisco for Wrangle which is a great little conference brought to you by our friends over at Cloud ERA this is the second time I've attended a Wrangle and each year it brings an interesting and diverse community of data scientists to an intimate and informal setting for great talks on real data science issues and projects not to mention cowboy hats and barbecue if you haven't yet caught the first two episodes in our Wrangle series twiml Talk number 39 with Drew Conway and twiml talk number 40 with sherith Ral you'll want to be sure to check those out they're both great interviews and the intro to the first show in the series includes important announcements about the series as well as our latest ticket giveaway our online research paper discussion group and my email newsletter the show you're listening to Now features my interview with Aaron Shellman Aaron is a statistician and data science manager with zeren a company using robots and and machine learning to engineer better microbes if you're wondering what exactly that means and involves I was too and we talk about it in the interview our conversation focuses on zeren's use of Apache airf flow an open-source data management platform originating at Airbnb Ain and her team uses airflow to create reliable repeatable data pipelines for their machine learning applications and we explore all that in the interview a quick note before we dive in as is the case with my other field recordings there's a bit of unavoidable background noise in this interview sorry about that and now on to the show all right everyone I am here at the wrango conference and I am with Aaron Shellman who is a data science manager at zyren a company that's doing really interesting things that I'll let her tell you about but before we get to that Aon welcome to the podcast thanks thanks for having me I'm super excited to have you on the show why don't we start by having you tell us a little bit about your background sure so my background is sort of at the intersection of computer science statistics and biology so I went to graduate school at the University of Michigan I did my masters in biostatistics and then my PhD in bioinformatics so always kind of working at the yeah at at the intersection of of data and biology when I graduated sort of maybe counterintuitively I came out to the West Coast I live in Seattle and I worked at Nordstrom in Nordstrom technology interesting yeah and I was on a really cool team called the data lab there and built product recommendations for nordstrom.com okay and then did a stent at AWS and now I'm at zyren okay nice and what specifically do you do at zyren yeah so I was initially I was the the first data scientist at the at the company and I was a data scientist there for the better part of nearly two years and then recently I've transitioned into managing the the data science group there we're now eight people including myself so a lot of growth since I started okay so yeah so a friend of mine and a former a previous podcast guests Josh Bloom has said that the worst job in the world is to be a company's first data scientist do you agree or disagree that is a there's a lot of Truth to that it it really depends yeah it depends on what you like so what I like about so actually maybe surprisingly all of the jobs that I've had I've been the first onto that team okay and that wasn't really on purpose but what I like about it is that you really have the ability to kind of structure what what the goals and what the mission is who you hire and how you build out that team and I really enjoy that part of the job sort of the the higher the higher level maybe not so data Centric parts of the job and you definitely have that kind of stuff more available to you when you're the only person on the team how's it been like building out a team from having that experience as being the very first does it change your perspective on team composition and how you build it out yeah it definitely changes your perspective because you know when you're the only person you're quite resource constrainted and and so the uh the hiring matters in some sense more than when you have quite a bit more staff because every you know you double to two people that's still quite a not very many people and so it's really important that you get somebody who has skills that complement your own or some who can you a lot of things that you don't know to make everybody so it's definitely a different different experience than being in a big group with lots of people but I actually I think I like it more I prefer it actually okay awesome awesome now Zyer Jen I'm betting is a company that not a lot of people in our audience know about what does the company do yeah we're a little bit different than than you know the airbnbs and the the Facebooks and and and all of those type of those companies though we use a lot of airbnb's technology so kind of like airb be for microbes not really but yeah so at zurin what we're doing is we we partner with companies who use industrial fermentation to make materials and molecules and so what we do is we operate or we optimize strains microbial strains to be more efficient or more effective at producing molecules of interest to our customers through fermentation so often these are these are companies who are already using fermentation at scale to produce molecules so it turns out I'm thinking beer is it exactly is it are there other use cases here yeah so you know we we the common application right of of fermentation is to make alcohols alcoholic beverages but it turns out that you can use that process to create lots of different types of molecules and you can use that to make molecules that can be precursors for pretty complicated materials as well and so that's what we do we kind of do the same process but we're making all kinds of different types of molecules for different applications what are some examples of those applications so examples of applications in general not specific to zyren are well for example insulin is is sort of a a classical example of using fermentation in in Health Sciences to to produce insulin so that that was a huge revolution in terms of being able to create it because it was a very expensive and kind of gruesome way that we used to do it in the past which was largely through extracting it from pigs which is not pretty obviously it' be a it's a lot better if you can you know use microbes and and do it in Giant fermenters and you can produce a lot more at lower cost so it's kind of the same thing yeah and so is it a direct byproduct of fermentation or is it is is there a byproduct that's used in its creation yeah it kind of depends on the microbe and the molecule that you're producing but often what we're doing is sort of augmenting or kind of ramping up normal metabolic processes in the cell so these microbes will ingest you know sugars they'll metabolize things like that to create these molecules sort of as sometimes they're waste products it really depends and they excrete those into the the surrounding fluid inside the tank and then we Harvest that or we extract that oh wow yeah to to get those molecules wow so what was your talk about yeah so I was talking about some of the sort of the challenges that we face so I was talking a little bit about our mission actually the data science team's Mission and uh our goal is is to use our testing platform so the way that we do what we do at Z and stepping back for a second is that we rely on on robotic automation sort of combined with machine learning to to build this test platform that allows us to simultaneously measure the performance of lots of different strains in parallel okay and we use our goal on the data science team is to use all of that data that we're generating through this High throughput screening process use all of that to then make machine learning models or make predictive models to help us make better decisions about the experiments the strains that we design in the first place so basically to help the scientists design better strains so that we don't have to spend as much time experimenting if we could get to the solution or get to the answer faster that's really our goal so that's that's a that's an ambitious goal it's very it's not it's not easy to do and so part of what I was talking about was sort of the things that make it hard to accomplish that so what are sort of the Practical data data issues that we encounter and how we're solving those so that we get really clean in anal analysis ready or or modeling ready data for those complicated models okay and so what are some specific examples of the data sources and data types that feed your models yeah so by and large a lot of the data that we're consuming is really kind of measurements that represent concentrations so what we we're you know we're we've got these microbes they're metabolizing things and they're they're excreting these compounds or these molecules into the solution around them and then we measure the concentration of that so that we can tell whether the microbe has improved over its predecessor and then move that into make a decision based on that essentially okay and so for for the most part the DAT the data that we're working with are is some measurement some measure of concentration got it and what's the scale that this is happening at like is this concentrations in Vats of things or like micro Rays you or somewhere in between yeah that's a really great question so it's it's kind of all of those things so what we do practically in our testing platform is we we kind of do it more not not quite as dense as a microarray but typically it's something like 96 well plates so we have these these kind of plates that have 96 Wells each of those Wells contains fluid contains the microb and sort of the experimental input and that's the level that initially when we're doing our initial screens that we're that we're experimenting at once a strain for example demonstrates Improvement compared to its predecessor it'll go into another round of that testing we basically want to validate or replicate that performance again once it's done that we we actually validate those strains and ferment fermentors and so it's kind of a challenge to you know the size of of a fermentation tank is much larger than that of a a tiny well on a plate and so we want to we always want to validate those strains before we deliver them to a customer for example to make sure that the performance in the plate actually represents the performance that we expect in the fermenting the fermentation tanks yeah and so we do both at zyren and then we also like to partner with people and and run them at scale and in their tanks too when when possible okay interesting and so you started your talk talking a little bit about kind of the context and your mission and and then what oh and then yeah and so we were I was talking a little about some of the some of the challenges we face with our data some kind of practical challenges and how we're using using airflow to to build a pipeline that addresses some of those challenges okay what's airflow so airflow is it's a a patchy incubating project it's a python module basically that allows you to construct data processing workflows and you const you basically construct a dag of that workflow and allows you to do things like scheduling monitor the progress of those jobs and and and even a little bit of reporting yeah so it basically helps us orchestrate all of our sort of comp licated ETL steps okay where are you eting the data from where does it tend to live yeah so in mostly so we have we have what's called a limbs it's sort of a a biology information management system that's right yeah I've never met anyone who who was like not a biologist who knew what that was yes that that's exactly what it is so we have a we have a Limbs and we have a a corresponding frontend that the the scientists can upload and sort of download data from and then that gets persisted to a a sort of a source of TR like a data warehouse and that's for the most part where the the data scientists access the data from from a data warehouse okay so you use airf flow how long have you been using that we've been using it I guess almost about a year okay and that's you mentioned earlier you use Airbnb stuff air flow air flow is an Airbnb product yeah it was yeah exactly it was an Airbnb product that I think they open source and then it was picked up by a pachy and now it's kind of an incubating project which yeah and how does it compare to I'm trying to remember the name of the product that compliments like Google cloud has their data flow and there's a an open source it's also Apache it's not FL is like Uzi or yeah no it's comparable to that it's comparable to Uzi yeah okay but not it's it's agnostic to sort of I think Uzi is sort of a part of the Hadoop ecosystem airflow is not so it's it's pretty generic in that sense so you can you can really use it it's pretty powerful in that sense so it it doesn't have have any opinions really about the the platform and where your data come from okay and so do you you know what are the implications on the way you kind of craft and deploy the analytics that sit on top you know of the underlying you know the data the kind of the data engineering pieces like does airflow does the weight and you know any of the semantics of airflow have direct impact on the way you do the analytics or is it you know kind of separate concerns I I don't know I think maybe maybe they're largely separate concerns although I guess one of some one of the sort of use cases that I described was you know we we do a l a lot of experimentation adheer and lots of different types of experiments and our scientists use lots of different types of tools to work with data and the result of that is sometimes the data doesn't make it into our limbs or doesn't make it into the warehouse and one way that we've addressed that is that airf flow has all these nice sort of hooks or operators into third party things like Dropbox and so one thing that we've had success with is to be able to work with the scientists and and get them to make some standards around where they put their data in Dropbox and then we make really lightweight ingestion pipelines to grab that data and ingest it into our limbs for them and then we're using also a nbnb product called superet which is a sort of a dashboarding tool so we we've now hooked that up so that we can ingest data from Dropbox and then we can produce dashboards for the scientist to actually consume their own data that way and that's been kind of a success story making really lightweight stuff doesn't take very long to make it all and and can Surface the results right there pretty quickly okay and so what were some of the insights that you were sharing about using airflow yeah so I was sharing I I was kind of stepping through a couple use cases that we've well I was I was describing some of the our limit or the challenges that we face with our data and sort of the challenges we had when we were working on our own sort of homegrown ETL solution or platform and why we eventually sort of abandoned that and and adopted air flow and so what I imagine that a lot of people start there like what are some of those challenges yeah one so one of the one of the I think more difficult challenges was that I I mentioned that largely the a lot of the data that we're working with is concentrations of things so we're measuring how much of something there is in this in a certain volume of solution it turns out measuring concentration of something is not that straightforward so there are a lot of different ways that you can measure the concentration of a solution of something in a solution and depending on the group and whether like who they're working with and sort of the way that they choose to measure that that has implications for the data that we can expect but we need to basically process everything the same way regardless of the sort of the format or the type of experiment that they used right and so that was challenging to kind of articulate ourselves there's just a lot of overhead and sort of a lot of logic that we would have to encode to do that another challenge was describing the sort of complex dependencies in between our processing steps so you know I need this to happen and then that's going to kick off another job that does this and that's going to kick off something else but orchestrating sort of that communication and writing all the logic to for what do I do if the first thing fails or what if I what do I do if the second thing fails and doing all of that ourselves is challenging and we have all the we have data coming in at different velocities and that's also hard to orchestrate so some of our products they start processing data as soon as a scientist uploads data into the limbs or into the warehouse others can be scheduled and so it runs nightly or weekly and or doing all that orchestration ourselves was was very challenging okay and so I imagine the upshot Is That airf Flow kind of handles handles all of this for you it it you know does the impedance matching from you know the different different velocities of information coming in and things like that so airflow does a does a lot of things for us in terms of sort of handling different data inputs and and being sort of agnostic to that it's been huge for us for that so we've basically in our processing steps we have created sort of a generic interface so we have sort of three big processing nodes that need to happen and they all have very generic interfaces so they don't know anything about the data that they're going to receive and then basically we we contextualize I it at the time that we receive data and so and that that has made it very flexible and modular for us so that we can you know a new experimental platform comes online it'll be very easy for us to apply the same well I wouldn't say very easy it'll be much easier for us to apply the same set of processing steps with a new data set just because we've we've gone through the effort of making the interface generic so that and that was something it was harder to do yeah sorry are these three you said nodes are the three nodes are they is this an artifact of kind of the way you would Ideal World design your own processes or is this something that's kind of imposed on you by the way air flow does things oh no it's it's more just like sort of the the flow of the pipeline the processing Pipeline and it which so happens to have three steps yeah so like the the initial step one of the one of the things that we see you know I mentioned I think that we use part of the reason we're able to do what we do is that we rely heavily on robotic automation to do a lot of the heavy lifting in the lab but robot fail sometimes or weird things happen in the lab you know it's a it's a it's the experimentation is is challenging and so the result of that is we'll see sort of extreme values or like outlying values that you know typically indicate a process failure and so that first step is really just an outlier detection step so let's let's identify those process failures and filter those out of any Downstream processes the second step of that pipeline is something called normalization and that's really meant to address sort of another challenge that we face which are batch effects which is a very common phenomenon in high throughput screening environments so you you know you've got a bunch of high you got a chip with a lot of samples or a lot of you know probes or on it and you're asking a lot of questions in a very tight space and so these types of environments tend to have strong temporal effects so even if you meas you do the exact same experiment this week and next week they might look like they're coming from different distributions and that doesn't actually reflect meaningful biological variability it's actually just kind of a reflection of the process or of the temperature in the room at the time or yeah the person who ran it or all these other things that we don't actually care about they're kind of nuisance things right and so the second step of that processing pipeline is normalizing the the data to try to eliminate those process process related biases okay and then sort of the third step and sort of the third challenge that we face with our data is we have a motto at zyren that is any microbe any molecule and what that means is that we we've built a testing platform that is agnostic to our customer microb and to the molecule that they're making we think that we have a process that will allow us to optimize those strains regardless of of the actual application that's amazing from a sort of a business strategy point of view because we can work in lots of different Industries we can work with lots of different bugs and make lots of different types of molecules but it's it can be challenging from a data perspective because it can result in a proliferation of solutions so we don't always have agreement on what the right way or whether something that constitutes an improvement for one group might not be considered an improvement in another group and from a modeler's perspective we it's not always clear what the result of the experiment is to us as consumers of that test data in a way and so one way that we're addressing that is this sort of third piece of that processing pipeline where we actually do the matching up of the candidate strain with its reference strain and we do that testing and we like statistical hypothesis testing and we write that result so that regard sort of independent of the decisions that our scientists make we we have some indication of of what we think happened in the experiment and a sort of a consistent view of what Improvement looks like okay for our models interesting and is it from a a modeling and statistical perspective is it challenging to I imagine it to be challenging to kind of normalize results from you know comparing one molecule to another or one process biological process to another is that challenging so in general we don't we're not doing that so we don't we don't share data at all between sort of projects or like I guess what I understood you to say that you know you've built this General platform for testing the results of these these molecule production micro to molecule production processes yeah and then as a way to make sure that you understand their efficacy you know you're kind of taking that analysis all the way to you know well tell me what is the that end result that you're that you're driving for yeah so that that node or that last step in the pipeline is called often called Hit detection in sort of the biology the high throughput screening literature and that's really just the the process of identifying in your screen which one which which of those candidates that you were exploring seem to have the characteristics that you're looking for and so it's sort of a it's not as quite as because it's a screening scenario where our statistical criteria isn't quite as high as it would be if you were doing a single test like I want to know you know we're screening and so actually we care more about keeping false negatives low than we do about having high false positive rate and and you know retesting something that actually wasn't a very stra we would rather waste some resources testing something that wasn't good than lose the opportunity to test again something that actually was good mean you're screening for possibilities and the more possibilities you have the more opportunity you have to find a thing that actually works exactly yeah okay yeah so it's kind of a different like way of thinking about it than than you know often like in an AB test for example you want to know the right answer right right okay got it and so you talked about the challenges that led to deploying airf flow what about the challenges of deploying airflow and kind of building out this system did you encounter anything in particular there yeah so you know it's sort of probably challenges that are typical of adopting anything that's kind of an an early project right I would say one of one of our bigger challenges was really just finding non-trivial examples of how it's used in production so what I think some of at least I experienced this I feel like this was sort of the general experience of everybody on the team is that you know there's a certain way you're supposed to write these dags or these workflows in airfow every time I would write one I felt like it was the wrong way so I would either try to it felt like I was putting too little processing sort of in one node and kind of making too many nodes or it felt like I had one big node that did everything and so it was really hard to like get a sense for what the right way to construct what how what unit of work was appropriate and was sort of intended by the design of air flow okay that was challenging took some getting used to the team is using it a lot now so I feel like we're we've got that a pretty good handle on that now but that was certainly challenging sort of getting yeah getting familiar with it and and finding good good examples of non-trivial examples of of its use yeah that's we have it's interesting it's something that comes up a lot in my conversations with folks in different domains right in the you know both in you know take as an example architecting neuronet right or or this like there's you know the documentation and there's the research and the literature so much of adopting these new technologies like tribal knowledge or black art or you just practice that you know you often don't find good sources for how to do that stuff yeah and that makes it I don't know that that's kind of a friction on adopting this stuff like you you want to see some success stories of somebody having used it so that you can be sure like before abandoning right like our home our homegrown ETL system we want to be reasonably sure that the thing that we move to works like isn't going to suffer from the same problems and of course that the understanding that it's it is a new project and that there will be sort of foibles as a result of that but General we want to make sure it solves the bulk of the problems that we have with our own solution and that can be hard to demonstrate sometimes but in our case it worked out airf flow is is is a really powerful tool for our group has that changed at all since You' started to use it are you seeing more of these examples or more you know better documentation of these more subtle um know either use case examples like you described or kind of some of these more subtle design philosophies and decisions yeah I mean even today like I I was the speaker right before me talked a lot about airflow and then right after me it was a panel where they talked a bit about air flow so okay it seems like it is now becoming quite a popular tool and being used pretty pervasively so I I expect that we'll see more and more more and more of these kind of stories of how it's being used in production and and all it's a very flexible tool and has a lot of functionality and so we're using it in ways that we didn't actually expect initially and so I'm I'm excited to see how other people are using it I'm sure there are a lot of really creative things that are being developed on it nice is it common for the or is it intended do you think that the main consumer of the tool is a data science team as opposed to a data engineering team or some other variation on the term yeah I think that's a good question I'm not totally sure what the intentions were but it's definitely a tool that's very powerful for data scientists okay in terms of just like the way that we think and and being able to construct workflows that way it just it feels very natural okay I guess it's even a hard question to answer because data science and data scientist means you know so many different this is this actually came up in my last interview as well we talked about how this has evolved since you know 5 years ago when everything was considered data science right or data science was considered needing to know all of the different bits and pieces of you know moving the data around and doing the analytics and getting it to production yeah when I hear you describe the the tool I think Plumbing right I think kind of lowlevel stuff yeah and that was really the source of the question like is is you know do you think that that that is typical you know typical for data science to to kind of dive into that level or is or are data scientists typically supported by other groups that are kind of putting the plumbing in place yeah I so for us the way that we ended up you know kind of productionizing our airflow environment was was with a collaboration with data engineering so okay we have our group is pretty engineering heavy anyway all the data scientists are are fairly do a lot of engineering okay and so we we of course we had help from uh from actual data engineers and then and then members of the team who are who have that skill set as well okay are responsible for sort of doing all the configuration and the startup scripts for people because our our group is pretty mixed in terms of the the background and and interests and and the skills okay and so that's been great like they they spent a lot of time really developing developing structure around how to get it set up and so that all of the data scientists when they come online can can easily set it up and then start doing what they already know how to do which is construct the workflows in in Python so it's it's kind of two levels I think I think we did probably need help from data Engineers to actually get it up and and get it running consistently and sort of in a production level of an environment but then once it's there it's actually I think a very simple tool for you know the average data scientist to to quickly start making making processing workflows and and part of that is that it has a really cool out of thebox UI so like you can write your workflow in Python and then you can go to the UI you can run it there you can view all kinds of metrics about the pipeline there's even stuff about sort of which tasks in that pipeline are sort of the bottlenecks and what are the performance metrics of each of the individual tasks so it it allows you to kind of see get V visibility into those workflows that in the past I I haven't had actually really anywhere so it's great for that interesting is and is that user experience is it kind of analogous to a notebook or is it more like a job processing type of tool no it's more it's more like like a Bonafide application so it's got like sort of a a table of all of the D or all of the jobs that are around and you can kind of change the scheduling right there in the UI turn turn them on and off okay and then tab over to metrics and all kinds of other things you can view the logs from there so if something failed you know obviously you can you can log into the machine and view the logs there or you can just use the UI and view the logs directly see what happened with your job you'll get a bunch of Rich Diagnostics about which part of the the workflow failed and and all kinds of stuff right there in the UI so it's actually it's a pretty developed tool oh wow it's really helpful oh very nice very nice anything else you'd like to share with the audience or leave with the audience I guess just check it out like it's been it's been a really great tool for us like our so you know our the reason that we invested in it really is to support our our work in predictive strain design that it's sort of our mission is to to use machine learning so that we can construct better strains and help our scientists get to the solution faster and airflow has been incredible in helping us solidify that processing pipeline to support that work now we have clean and Analysis ready or modeling ready data that's consumable directly from this Pipeline and a lot of it a lot of the headache of you know what's sort of traditionally or maybe not associated with being a data scientist but what data scientists know it's actually about can be sort of addressed with air flow or at least can be ameliorated some so that it's not so that you know 90% of your time isn't actually spent cleaning data and munging it moving it around it can do it's very flexible can do all kinds of different types of tasks and that's been really helpful for getting rid of trying to eliminate sort of the the boring stuff so that we can do the cool the cool stuff okay awesome awesome if I can draw you back in that was kind of a great summary but we haven't really talked a lot about the the specific models that you use like we've talked about this tool that helps you get the data to the models yeah tell us a little bit about the you know the types of models that you're building the modeling techniques you're using things like that yeah I can't can't talk a ton about them but conceptually what we're doing is we're taking information about what we know the scientists are engineering into the strain so the type of change that they're making where it is so what What gene is it that that's being perturbed or being engineered and we're basically making models to predict combinations of those changes so the you know the typical microb has something between you know three and 5,000 genes in it and so if we were to you know perturb each of those individually and then start perturbing each pairwise combination you know that's an infinite gorical space to explore and so what we're trying to do instead is is take all of the information about the things that we know we've already done and then make predictions about how we think those those are going to perform when they're combined so a little bit of evolutionary genetic algorithm types of things or a lot of things actually so we we have a lot of different types of models and and and some work in better context than others so we have we have models that kind of address different things some work better when you have a bunch more information so after a project is fairly mature and we have a whole lot of test data that we can use to train on our models are different than sort of when we're at the cold start problem where really all we know maybe is the you know the metabolic structure of the microbe we haven't started doing anything yet so how do we how do we guide the experiment in that case when we don't know at all about really where to start and so that that technique or those those Suite of models are a little bit different and they rely more on structural information about metabolism than than experimental information which can happen later after we've collected a bunch of information from experiments okay all right very cool very cool well thank you so much ER for taking the time to jump on the podcast with me yeah thanks so much for having me it was a lot of fun thank you yeah all right everyone that's our show for today thanks so much for listening and for your continued support of this podcast for the notes for this episode or for any feedback or questions please leave a comment on the show notes page at twiml a.com talk sl41 thanks again to Cloud era our sponsor for the Wrangle conference series of podcasts to learn more about Cloud era and the company's data science workbench products visit them at cloud era.com and be sure to tweet at them using @cloud era to thank them for their support of this podcast if you're interested in joining the twiml online Meetup we will discuss research papers like Apple's recent paper on generative adversarial networks you can register for that at twiml ai.com Meetup and don't forget to sign up for the newsletter at twiml a.com newsletter thanks again for listening and catch you next time