Web Scale Engineering for Machine Learning with Sharath Rao - #40

Key Takeaways

[Music] hello and welcome to another episode of twiml talk the podcast where I interview interesting people doing interesting things in machine learning and artificial intelligence I'm your host Sam charington this is the second show in our series of podcasts from the recent Wrangle conference as you might know a few weeks ago I was in San Francisco for Wrangle which is a great little conference brought to you by our friends over at Cloud era Wrangle is such a fun event each year it brings an interesting and diverse community of data scientists to an intimate and informal setting for great talks on real data science projects and issues not to mention cowboy hats and barbecue if you haven't caught the first episode in our Wrangle series twimble Talk number 39 with Drew Conway you'll want to be sure to check that out it's a great interview you and the intro includes important announcements about this series as well as our latest ticket giveaway our online research paper discussion group and my email newsletter the show you're listening to Now features my interview with sharth Ral I reached out to sherith about being on the show and was blown away when he replied that not only had he heard about the show but that he was a fan and an avid listener my conversation with him digs into some of the Practical lessons and patterns he's learned by building produ uction ready web scale data products based on machine learning models including the search and recommendation systems at instacart a quick note before we dive in as is the case with my other field recordings there's a bit of unavoidable background noise in this interview sorry about that and now on to the show all right everyone I am here with s raal he's an engineering manager at instacart and we are on location at the Wrangle conference and shth has a talk later on and I'm fortunate enough to have him here to tell us a little bit about what he's going to tell the Wrangle audience about so shth welcome to the show it's great to have you great thanks for having me Sam I'm a big fan of the show from having heard earlier episodes nice nice well I really appreciate that as you know one of the places I like to get started then is to have folks introduce themselves tell us a little bit about what you're up to and how you got there yeah yeah definitely so I'm at instacart been here for a couple of years instacart is a um on demand grocery delivery service and my role here is I started off as the data scientist or machine learning engineer focused on search personalization and recommendations and now I do that but also leader team that is you know working on that effort so prior to this I've spent time at a couple of companies doing search advertising and auctions that sort of stuff over the last 10 years and even going back further grad school I worked on speech recognition and speech translation before it was you know practically useful in the field it's part of the reason why I started working on other things after after grad school so yeah so that's captures the range of things awesome awesome I hear a lot of stories from folks who you know I was working on this stuff in grad school and it just wasn't ready or we were in the middle of the AI winter and it kind of went into hibernation and you know now that we're all you know so focused on doing this stuff it's like time to dust off those skills work yeah yeah so in your description you you said data science slash machine learning engineering how how evolved is the distinction between the two of those at instacart and and do you consider yourself as spanning both yeah let me take the first one first I think at instaart well overall as industry I think we are moving better you know towards understanding these two roles and so at least we are sort of settling around maybe three roles actually data data analysts data scientists and data Engineers even if they're slightly called separately I think maybe there are companies like maybe LinkedIn rbnb where probably have data scientists and machine learning engineers and data Engineers but broadly the difference is around you know people working on data products Building Systems and algorithms essentially things that are consumed by others other algorithms and systems who might be like machine learning Engineers people working on things the output of which is used for you know decision making you know by the team that they serve or the executive leadership we call them data analysts at instacart but data scientists elsewhere and then finally data Engineers who are working on like platform infrastructure typically comes in you know maybe later in a company stage right right I think it's a lot of progress we've made from just a few years ago when you know data scientists was this unicorn that no one could really hire because we've defined it in such a way that you know it requires these disperate skills that you know aren't traditionally paired yeah I remember back in maybe 2012 2013 the the term of unic data scientist right was probably one that captures what he just described but I don't hear that too often there's more like specialization and people understand that over time most professions end up specializing although I'd imagine you it depends on the context of the companies I would imagine say startups if they if they're hiring their first data scientist they probably want somebody who can also do like the other two things at a reasonable level of competence overall in a macro view I think U the roles have sort of specialized over time yeah and I remember back in that same time period 2012 I think as a community we thought that it would take way longer for us to reach this level of maturity it was like you'd see these stats that say you know we're going to have you know we're going to be underresourced in terms of the number of data scientists until like the year 2050 or something like that and I think I think this this specialization has been part of the key for alleviating that stress although we're far from having enough you know people with data comp encies to to meet all the Demand right I think every speaker at an event like this gets up and says oh and by the way we're hiring right yeah so I've I've heard that and also more recently heard about and maybe even seen it in the field where there's you know with you know obviously the market has responded to at least a perceived lack of you know strong talent and so we have quick like smaller onee Masters or boot camps that have served the need so as a result I think in maybe in some part of the market they might maybe at like Optimum or even over supply of you know data scientists at arguably at starting level really now that's happening quickly maybe I mean just generally that I guess everything is moving faster and now even for example with newer areas around say deep learning in AI there is you know there there are more research institutes there are more again training you know workshops and so on that people respond faster to needs than maybe they used to M mhm interesting all right so your talk you've got a talk in an hour yeah what's your talk about sure so my talk is titled lessons from integrating data machine learning models into Data products so initially I had a two-part talk but you know today I'm talking about like the part one of that but really the the Genesis of that comes from having worked at first of all you know relatively a smaller company as opposed to some of the bigger companies that where in we have there is a product per se and there are machine learning models that are integrated into various parts of the product they're making different decisions for the customers so there's that and then there is aspects about like data scientists building models model prototypes and even productionizing them but also interfacing with product Engineers within a single team and the fact that what I call like all mod all model prototypes look familiar but every model and produ ction is unique in its own way so I'm trying to understand try to frame a conversation about what does it mean to have a model in production you know what are the questions to ask yourself and how do we communicate as data scientists with you know product managers and product Engineers about what this model does what are its requirements what are the constraints and so that's that's what I'll be talking about okay and so how do you how did you structure walking people through that yeah I try to keep it try to find you know going back to conversations within the team for example each time we we have one team that has you know three or four product Engineers you know a couple of data scientists and a designer and a PM right and largely most of the work happens you know within the team and each time we work on a an experiment or a feature that that data data scientists build we have this conversation about how will this go into production and typically the conversations you if you think about it like the of building a model thinking about a problem building a model prototype largely happens you know within the the domain of data science and maybe you know with some induction with the PM but the moment it goes into production it's now touching like so many different parts of the system and you know a couple of things come in like how much time do we have to make make a decision in terms of like the model you know running in production you know meaning if you're serving up Pages live to the web you know you've got a certain yeah serving latency for example yeah address right yeah so There's a constraint that very often getss dictated by the product itself and rightly so and then there's this question of you know for the model to be successful how much information does it need and how can it get that information by information I mean like for example let's say you're trying to recommend a product maybe you you definitely need to know if you want to you know it's a personalized recommendation you need to know the user you know user identity and their past data you probably need to know like what page they're on and so on but do you really need to know their recent searches and you know recent activity right that certainly is like short-term context how much context do do we need does the model need rather to operate successfully both of these sort of give you a way to think about can I cash my recommendations you know what can I cash versus what needs to be seed in real time MH if it's real time can I do a reasonably well if you know let's say let's say I'm making a decision about what product to show based on things that you've added to the cart recently so think of a model that is continuously producing a you know prediction of what what product is a good recommendation but it doesn't necess surface that it's always you know continuing to score and maintain a best estimate of a recommendation in the background but you know at any point it's available with a best effort response so this is an example where you don't you know you have a good response whenever whenever there's a need for it but you can wait until the last moment to actually serve that recommendation so it's things that of that happen in the background are you describing a system that is is doing like online learning or Active Learning which is where you're updating your model kind of in place well yeah so well the model could be static model need not be updated but the the model predictions could be happening continuously in the background ah based on based on whatever data it sees yeah it doesn't need to score and immediately report it can keep scoring as it gets more data and then always be ready to serve a recommendation when necessary okay yeah so that gives us like you know so we have this you know four quadrant because I'm considering latency and context sensitivity of the Model Behavior and we get these four quadrants where you know obviously you have like high latency is okay but you have contact sensitive and the other basically the other four possibilities other three possibilities mhm and so is part of what you've observed then you know are we at the point of maturity and thinking about this that you know for each of these quadrants we've got got like design patterns or best practices that teams you know either at instacart or in the industry are following certainly at instacart we try to place ourselves in terms of like where does this model sort of lie you know just mentally internally and that's sort of all the given that we've built a few experiences few models over the past couple of years we have patterns you we know we sort of know 90% of you know the way the model is integrated the rest of the product when we are thinking about the model we don't have to like start from scratch each time mhm so for example search search ranking you know we can we latency needs to be really low and contact sensitivity we can start off with you know low contact sensitivity which means that you let's say You're simply matching the search query and a bunch of products without like recent context we could start there but then as we the way to improve the model would be to well continue to have latency requirement being low but make it more context sensitive obviously it's a lot of work that might involve a model that is ranking reranking product based on you know recent activity so that's one example so with search we start I I call it we start from Quadrant 4 and go to Quadrant One sort of it'll be clear from the slides I guess but um with the recommendations it's it's it's almost always the case that you start with you know it stting quart three where you cach as much as possible you score them you know in batch mode you cash them and then then you serve so that that keeps that gives you you know you can use as much information as you want and you know gives you the latitude to use more data and have like recommendations the way to improve that would be to go from Quadrant 4 to well quadrant 3 to 3 to two I guess where you know you still you know latency is still not a consideration you're reranking your products your you know candid products and you know in the background be ready to serve then finally you might have like you know contextual recommendations wherein you have to recommend something you know immediately and so that becomes like a real-time recommendation which is you know a model that you know ranks a set of products that you've cached you know based on the context so the set of products you've cached may be personalized or maybe not personalized but then when you serve you rerank it based on user activity and make it personalized yeah so how much of this machine learning engineering is machine learning and how much is engineering meaning a lot of what we're talking about is you know system levels scalability like you know just hardcore software engineering that you know in a lot of ways is orthogonal to the machine learning well I guess the model stuff isn't but in a lot of ways it seems orthogonal there are principles at least that seem orthogonal to the thing that you're scaling right it's it's web scale engineering applied to ml so like like fine-tune that statement how much of it you know is spe is domain specific and where are those points that it gets super domain specific that's a really good question and I think the point at which it gets domain specific is that as a data scientist you understand the model its assumptions its requirements at what point you know it'll fail to perform what does it absolutely need what is stable Stakes versus you know what is a cherry on the top you understand that and you well you need to understand that and then articulate it to you know product Engineers well and to to others on the team obviously you don't want to do that right at the end when you've already built the model that you want to do it right in the beginning so that you can know up front if there are any constraints on the on the engineering side right so it is sort of good mix of machine learning you know and software engineering and even product design frankly you know I'd imagine one one of the things we found for example is we may have some latitude in how we change the product experience for a user based on some of the constraints on the engineering and machine learning side for example if you want to buy buy yourself time to be able to score what would that interface what will we do with the customer at that time how do we design the product experience uh yeah interesting I've been I've talked to a few people about this like I I think there's there's an evolving field of like I always call it intelligent design but that's like way overloaded but they there needs to be and I think there will be you know some you know thought best practice practices for lack of a better term like just a way of thinking about designing kind of in light of machine learning and intelligence and things like that that you know I don't see a lot of people talking about that kind of stuff yet and it's not necessarily exactly what you're describing here what you're describing is even you know kind of going back to the previous question it's it's really more the interaction between you know design and product engineering and like performance engineering and things like that like how does our how does the assist some how does a user experience change because of the limitations you know in you know performance and product engineering and stuff like that yeah I definitely think it's definitely controversial to to suggest that we have to change user experiences to work with the limitations of you know engineering or machine learning right and it may not be common to do that but I wish I had like more examples where it's possible to do maybe maybe it's not something that we generally talk about if if indeed it happens in the field and that's other thing which is like it's hard to talk like about machine learning models in production widely simply because there are a lot of details that are probably very specific to the product and you know one just assumes that it's not interesting to somebody else or that these details don't really generalize so there is no there are definitely basic principles you know of software engineering but beyond that are there other principles that are specific or that are more important when it comes to like software engineering with like machine learning systems or data products so I called out a couple of examples of some work in the community like recently maybe about early this year there was a a document from one of the machine learning engineers at Google about I think it's called rules of machine learning or so it talks about a really nice what I think is a required reading for data scientists building data products about what are the different things what are the different trade-offs what are the different stages of building a data product and talks a lot about the engineering aspects the training the data management metrics the interface between like metrics and uh model formulations and so on it's kind of like you know it's a four-page PDF with with such density of knowledge like at at instacart we had like a reading group session we spent like I let a group with like two hours of going through that and you're talking through like Salient ideas from that paper okay well I have to make sure I get a link for the that one and post it up in the show not definitely yeah there's one more paper that I like quickly which from Pinterest where they talk about the evolution of the recommendation systems over 3 years and this is it's a really good one I think it talks about incidentally it talks about how they started off with a simple system that was like caching recommendations of the quadrant 3 world and then from the how they progress from that to you know Quant one wherein they use what you get what you cash and then you rerank it with a model it's helpful to have this you know view of how this really massive you know at scale recommendation system evolved so that you know people who haven't done that before don't need to believe that we need to start with what Pinterest has in production after 3 years of work right right so sort of underappreciated part of literature I think yeah well this is maybe a little bit of a tangent but I don't know if you've heard on some recent podcasts I've mentioned that I'm going to be starting uh like an online paper reading group of podcast listeners I I mentioned it randomly in a podcast and like there were a bunch of people who expressed interest in it so I'm going to be getting that kicked off but since you mentioned that you are involved in one at instacart any tips for running a a paper reading group great so yeah let me see we have like two different sessions where all the data scientists come together okay you know sort of alternate weeks and one of them is or the one that I was talking about we call it lunch and learn we have we do it over lunch every Tuesday and the bar in a sense is that you know you ahead of time we we talk about like what paper we or somebody's going to lead a discussion that person doesn't need to have completely understood but it's it's helpful for them to be able to know enough to drive the discussion it's obviously optional for you know people who want to attend because there are you know we have we touch different domains in instacart so somebody may not be particularly you know keen on a paper and but then it helps to have like discussion you know if you come in make sure you've skimmed the paper and you know come in for discussion we talk about what the paper some of the you know what the take base on the paper and also like how that immediately might apply to what we working on you know in our regular work how long do they tend the run it's a one hour and we typically have anything from like 8 to 12 people in that you know group okay you know I would say maybe like four to five of them will have like looked at the paper a little bit yeah but I think these are opportunities for people to not just learn but also a leader discussion right and you know people get better at it as they go yeah yeah okay well I'm really looking forward to it and like I said a lot of people have a lot of other people seem to be looking forward to it as well so love to have you join us and walk us through a paper of your of your choice really yeah definitely yeah I think it's very common in like grad school and like research labs to be like doing this like reading the paper is is an art and understanding like what is the next well what are the assumptions what they achieve what what are the opportunities from here like how can we build on that what is relevant to us right how hard is it to reproduce you know in our environment right that you know what are the generalizable lessons so to speak right those are sort of the questions we are thinking about yeah yeah so kind of going back to your presentation what are the generalizable lessons from your presentation sure so one of them is to think deeply about how the model rates into your product have the discussion ahead of time with people outside you know your own group like of data scientist maybe you know product managers product Engineers like the kickoff meeting should probably talk about that and at what level are you having that conversation because there's integration from a perspective of data there's integration from the perspective of data sources apis yeah generally at the level of apis okay and it's also a way to sort of get a Buy in from product engineer build that relationship so that it you know things shouldn't look like you're just throwing things over the fence to the engineer and asking them to like productionize it I think for them to understand it in a way that is at a level that they can even provide an input to and you know contribute to improve the design like the technical implementation would be helpful and I I picked out like two different aspects of like latency and Contex sensitivity there might be others well I picked up two and I got a four quadrant H yeah we can we can pick look at you know more aspects of it that are generalizable across models and yeah this I hope it sort of evolves into a discussion about how we can you know we have ways to talk about model prototypes you know how uh problem is set up and how training data is generated you know at some point you know the model is fit and we persist the model we have you know abstractions about what happens you know in training phases but abstractions that help us understand how a model is you know being served or the model predictions are being served and you know how it's actually executed and implemented might be helpful mhm yeah I keep coming back to this idea of design patterns like you know you may be familiar with the gang of four design patterns book where there's you know they've cataloged I don't remember the year but you know cataloged a lot of objectoriented design principles like Are We There Yet with machine learning or great question I think I probably had a I tweeted this question a few maybe maybe year ago like what sort of abstractions do we have for you know serving machine learning models in production right I don't know if any that is that substantially Builds on like you know software engineering patterns like or I don't know if anything that the community has agreed on I'm sure there are a lot of things that in people's heads yeah and a little bit that I'm talking about today will be about like what I you know have seen and working in in our team but going back to the Google Doc I think that again although it talks about more than just serving I think we need more of that in in the community yeah to imagine that there isn't enough generalizable because things are just too coupled with the with the product is probably not not true yeah you made an interesting comment early in the conversation about how you have a lot of these machine learning models in production but they you know at some level they all seem like snowflakes I guess is what I took from that and that you know it's been it's been an effort and a challenge to kind of extract the general principles from across these different environments is that kind of what you were getting at yeah in so far as we talking about like implementation yes right yeah right they're solving different problems but there might be like in terms of like software engineering patterns they share like some of these aspects that they even in one of these quadrants are there common implementation SL architectural patterns that you know that are kind of assumptions for you guys that you know you're you're doing across all at least all new efforts like for example microservices are you doing microservices containers are you doing containers like does any of that where does the kind of evolution of the state of Art and you know software engineering for non-ml Stuff intersect with ML stuff good question and well your early Point yeah we have services that are well we can call them microservices I guess but yeah we have you know Engineers well product engineers and Miss Engineers agree on you know certain contracts and so that is like table Stakes there are questions about like what can we cash what can we not cash there are questions about well yeah how long can we cash something you know what sort of what sort of data stores we might use and what is configurable what is not what is yeah what is an experiment versus what is not most things are experiments how much do we expect to trate on this feature before you know we think we're done for a while yeah which is I think for a company like instacart where there a lot of lot of different places we can invest in it might be like quite a few months before we come back to you know wi2 of something that we built because you know we're still exploring the space of areas where machine learning can help and you know where helped the most I guess mhm yeah but to your to your point about like Beyond software engineering what is new yeah I don't think have an answer right now okay okay to follow up on your last comment on just the opportunity prioritization at instacart are you generally taking an approach of like trying to take on you know big moonshot types of problems and you know make a few small big bets to kind of you know ensure that your resources are focused on you know these things that could have outside impact or is it more you know we're going to try to you we're going to try to touch you know broadly you know in you know high impact but more concentrated ways to kind of spread that impact around the you know the systems and the various business teams and processes yeah I think there's probably the there's a top down and bottom up in terms of what set of projects get worked on and explored you know within within say data science for just to stay in data science for a moment so we have company goals about like what are we quarterly goals long-term goals the translat company goals which typically talk you know are aligned to some metrics and you know different teams should know how they can move certain metrics and you know there is and then we think about like what sort of a data science effort could serve that met so there's that what I call the bottom up is well you know in the end we still we are we are e-commerce but also Last Mile you know Logistics operation so naturally with that domain some problems just are there like if you have e-commerce operation you have a search engine if you have a search Eng you're probably working on like you know spell correction and autocomplete and you know search ranking matching basically understanding the queries better and Rec recommendation systems and so on so so we already know given the kind of data we have we and we collect what sort of problems have been addressed like what sort of product features have been useful in that sense so between the top down and bottom up I think we generally find you know tactically projects to work on you know given you know current goals yeah and often times you know we hit like diminishing utilities soon enough or Plateau okay soon enough maybe in the first version and we might we might wait for like a while before we come back to it okay yeah interesting interesting well I think you've got a presentation to get ready for but I really appreciate you taking the time to chat with us and I'm looking forward to your sure thank you Sam thanks sure all right everyone that's our show for today thanks so much for listening and for your continued support of this podcast for the notes for this episode to ask any questions or to let us know how you like the show leave a comment on the show notes page at twiml ai.com talk3 thanks again to our sponsor for the Wrangle conference series Cloud era to learn more about Cloud era and the company's data science workbench family of products visit them at cloud era.com and be sure to let them know how much you appreciate their support of the podcast by tweeting to them atcloud era if you're interested in joining our first twiml online Meetup where we'll discuss Apple's recent research paper on generative adversarial networks you can register for that at twiml ai.com Meetup and don't forget to sign up for our email newsletter at twim a.com newsletter thanks again for listening and catch you next time