Live from TWIMLcon! Use-Case Driven ML Platforms

Key Takeaways

all right so uh super excited to invite up our next guest uh fran bell fran runs a data science platform team at uber she's got over 100 data scientists working on building tools that are a platform that's at a higher level of abstraction than uber's already famous michelangelo platform fran welcome to twimlecon [Music] [Applause] so i kind of paraphrased your role a little bit why don't you tell us a little bit about your team and your charter yeah absolutely thank you so much for having me it's really a pleasure to be here at the inaugural twiml con about the charter of the team the vision really is to provide cutting-edge data science at the push of a button to anyone within the company so that basically means that we're aiming to transform anyone within uber into a data scientist an example of this is forecasting so forecasting obviously underpins a large number of use cases within uber and so the vision here is to provide the latest and greatest cutting edge forecasts to folks at a push of a button via ui for example as integrated into a bi stack or programmatically accessible through our api and so the only thing that our end users within uber need to provide is historic data whether it's for example in the form of a csv file or query and the forecast horizon so how far you want to forecast out and we do everything else automatically in the background we scan over a whole suite of forecasting algorithms either those that we have integrated off the shelf into the platform or also those that we have developed in-house proprietary and we've gone way beyond forecasting in terms of data science areas other areas that we're investing heavily in in platformization are anomaly detection experimentation more recently also conversational ai in natural language and then personally i'm very excited about our proof of concepts of also platformizing and semi-automating intelligent insights generation and data exploration so of those conversational ai seems like the odd man out so to speak how did you end up working on that one yeah so we see a lot of really great benefits we have a lot of conversational ai data at uber one example is our customer obsession ticket assistant which was one of our first use cases in the space so here for example we wanted to aid customer service representatives in solving customer support tickets that are coming in and as you can imagine given the size of our platform we get quite a few of those and so using natural language and deep learning approaches we were able to build recommendations for our customer service representatives of what is the topic that folks are writing in about uh what are potential actions that the customer service representative might want to take and then also providing guard lines or guard rails basically around what could be the best starting point to actually address the response to the end consumer of course the customer service representative always has the final call and say on this but we saw really great improvements um in our customer care experience as a result of having this ai basically assisting our customer service representatives okay cool so you've got these uh this portfolio of platforms essentially that you're building to support these different use cases how do you know when it's time to platform something yeah that's a really great question and we're looking at multiple different dimensions here and very deliberately see which of the data science areas we want to platformize it's obviously a heavy investment and so we look at three items the first one is can the platformization of this area really create step function improvements to our user experience and the business and so sticking with the example of forecasting if we can forecast highly accurately demand in a particular space and time we can create more magical user experiences the second thing is really um the the wealth of use cases that exist across the company of course building a platform you want to be able to tackle many different use cases and so with forecasting for example it does span the entire enterprise ranging from marketing to obviously marketplace supply and demand financial aspects operations as well as our hardware we still have a lot of hardware on premise and so accurately forecasting the hardware needs especially on high demand days and special days such as halloween on new year's eve um is really important um so so that's the second aspect and then thirdly is a special day at uber yes yes it is yes see lots of demand on that front um and then the third dimension really is um the reusability of the models and methodologies that we apply um and again with forecasting you know a common framework that is needed uh to build forecasting algorithms is a back testing framework um so understanding the accuracy of your forecasts and that really is needed um for for any step along the forecasting uh journey and so having a common central paralyzed um language extensible back testing framework is something that's really important okay so is your team out evangelizing the opportunity to platformize and looking for customers that are already working on things that meet these criteria or um are folks coming to you saying hey we've got these problems help us all them how does the relationship with your ultimate customer evolve yeah absolutely so we have a lot of the product teams coming to us with use cases at the same time because we are this horizontal team that spans across the entire company across all lines of business we have a very unique vantage point as well and so we can also gently nudge some of the product teams to come and join us in this journey as well okay and so when you identify a problem space that it makes sense to platformize how do you approach that do you just jump in and start building start coding or what does the methodology look like yeah that's a great question so the way we uh build platforms is in a use case driven manner so that basically means that with every use case that is strategically chosen we augment the platform and we reuse as much capabilities as possible from the platform and that really allows us to have wins very early on and learning from this we actually now have a three-phased approach to platformization so step one is really consulting so we have these deep domain experts in particular areas of data science on the team and so we embed them with particular areas of the business where we see opportunities of of having use cases in these areas and so that has a couple of advantages firstly the domain experts learn more about the business about the opportunities the pain points and really can bring back these learnings to then drive the best design for these platforms it also allows us to tackle these use cases early on and really show wins and gain the trust of our partners and leadership on that front that of course is not a scalable approach this is why we set out to do platforms in the first place but it's a very good starting point and so the second thing that we usually do is templatization so what i mean by this is we build recipes whether it's form of documentation example ipython notebooks uh providing talks and educational aspects and this really allows us now to have a one-to-many multiplicative effect throughout the organization mostly to other data scientists that are dedicated to these business areas and then over time as we're taking on more and more of these use cases we really expand our platform to become more and more self-service and work towards that vision of really providing it at the push of a button without domain expertise required of course including you know best practices and guardrails in the process so in introducing you i mentioned michelangelo uber's low-level machine learning infrastructure platform uber was one of the first companies to publish about what they were doing to automate machine learning if i interpret your linkedin profile correctly you were at uber doing applied machine learning platforms before uh at least before that article hit possibly before the michelangelo effort even started what's the relationship between these two teams yeah we have a fantastic working relationship with michelangelo as well as the ei organization engineering branch that we also work with very closely to platformize and we have three modes of interaction here the first one is as the head of platform data science i get pulled in into the strategic and vision setting when it comes to michelangelo working closely with their engineering and product lead so right from the start there's a really great collaborative relationship that we can build on and then we have two other modes that have evolved over time the first one is michelangelo was more nascent we deeply embedded folks from our teams um into the michelangelo group so for example with the custom obsession ticket assistant example that i mentioned earlier this was actually the first deep learning algorithm that ran on michelangelo and so as you can imagine a lot of the features required for doing deep learning were in a very nascent state at the time and so having data scientists who are working on this particular problem deeply embedded in the michelangelo group and working together with the engineers and product managers there to build our capabilities not only to solve the custom obsession ticket assistant case but also more generic aspects that then really benefited the community more at large to build deep learning um uh you know algorithms and frameworks um was really important here of course um as michelangelo has evolved over time and became more mature we are becoming more of an end consumer of the platform and it becomes more of a self-service component especially with the onset of pi ml pi ml has really provided step function improvements to what's pi ml so pi ml basically allows us to write python code and bring our own models that then basically via michelangelo get deployed in a sandbox environment at scale and so that really reduces the barrier to entry for data scientists to be less reliant on you know the native approaches that are already integrated in michelangelo or software engineers that would help with productionization for example and so this approach has become really prominent across uber um and uh and has really opened up new avenues uh for self-service on michelangelo okay uh and so with your team pre-existing uh some of that effort you've got uh platforms and uh use cases that you've stood up before they were mature and ready now that they're more mature and ready is it a dynamic relationship in the sense that you know there's a uh have you migrated any of those legacy uh models over to michelangelo or um you know if it's there and it's working you're gonna leave it alone yeah that's a really great question um so we have a couple of aspects here uh one is platforms that are more recent uh so for example a conversational ai platform here from the get-go we built on top of the michelangelo capabilities and are actively utilizing this but as you correctly pointed out you know some of the platforms were built well beyond before michelangelo existed or was very nascent and so we have our own independent stacks on this front but here it's really important to see the opportunities for integration and to see a timeline where some of these platforms are already or may be merging in the future and then the third part is platforms that are currently standalone but likely will never merge with michelangelo so for example our experimentation platform which has very different types of methodologies and workflows i wouldn't imagine would would be combined with michelangelo can you elaborate on that what uh about the methodologies and workflows makes them you know not good fits yeah absolutely so um here we use more statistical approaches so hypothesis testing multi-armed bandits etc okay versus the traditional machine learning approaches um and so for that reason we keep them separate got it got it and so for the what are some of the key technical challenges that you face for this portfolio of use cases yeah that's a that's a great question so each of the platforms is different we have different users different use cases and so therefore also different requirements on the technology side but i can go into two concrete examples here the first one is for real-time anomaly detection and this was actually the first platform that i've built at uber and so here the idea is that we wanted to detect system outages as quickly as possible so people not being able to sign in or sign up or perhaps trips being degraded etc and and here we basically saw that this was still an open research problem and we set out to build a new platform around it and also advance the space we have a couple of patents now in that space as well but here the key kind of requirement beyond the innovation component was that we needed to have extremely low latencies so um as you can imagine because it's a real-time problem we had basically those considerations to take care of on and extremely high qps because we have hundreds of millions of signals basically backend as well as aggregate mobile signals that we're tracking in order to understand whether there's a system outage going on another one example is forecasting as i mentioned earlier we integrated our forecasting algorithms also into our bi stack for easy access through uis and so there's now a really great path where you can query a metric you can visualize this metric using dashboard builder and internal tool that we've built and then also you have this little button that says i want to forecast this metric and so obviously we want to make sure that we have a good user experience here and that people don't have to wait you know minutes or hours here exactly right um and so having low latency for some of these forecasting algorithms is really important and so are some of these technical challenges ever a reason why you might build something from the ground up yourself so to speak as opposed to rely on what the michelangelo team offers or is that not a consideration typically yes it does flow into kind of our decision making process here in terms of what is already available how easily it is extensible um and often a timing kind of component comes in as we discussed earlier in terms of you know where was michelangelo when we started to build um and and how can we evolve that in the future uh can you talk a little bit about the uh the technology stack that your platforms tend to rely on do you have your own kind of not michelangelo but you know you've got these higher level platforms that are very application or use case focused do you have your own kind of intermediate level of abstraction or are you building kind of use cases uh more independently yeah so when we don't uh build on top of michelangelo which is quite a few of our platforms we build microservice architectures um we built them in go in java actually for performance reasons um databases are typically in my sequel then we we run our instances typically on-prem for efficiency reasons and then several of our use cases are batch and offline especially on the training side but for those that we discussed earlier where latency is is something that we want to focus on we use caching for optimization and do you rely heavily on open source in this area or publish open source in this area yeah both so we're definitely building on the shoulder of giants and using open source wherever possible i think uber wouldn't have evolved as quickly if it wasn't for open source and so um utilizing the methodologies uh that have been developed by the communities is very essential but we also are heavily invested in open sourcing ourselves as well and we have quite a few open source projects if we look at the data science domain there are quite a few examples here as well we have pyro that was developed by the ei organization which is a probabilistic programming language we have horovod that is a distributed deep learning framework on tensorflow that has gained a lot of popularity in the community there is ludwig that was also built by the er organization which allows for a deep learning framework where you actually don't have to write code anymore to deploy and train models and then more recently our org has worked together with the ai organization to develop plato this is a very flexible um and use case-rich platform that allows for conversational ai especially especially in the research and prototyping phase and then also our causal ml package is a python package that we recently launched in collaboration with our marketing team that basically covers uplift modeling use cases as well as causal inference in combination with machine learning so yeah quite a lot of efforts um in that direction and we're considering to do more and with with so much out and available in open source that you're incorporating into these systems that you're building uh reminds me a little bit of a podcast that we published not too long ago uh the a quote that the the guests mentioned that became the title of the podcast was machine learning or ai was a systems engineering problem i wonder how much of what you're doing is uh systems engineering connecting pieces that you know in many cases exist already versus kind of pure innovation and building new stuff yeah it's definitely a combination of both um especially because we want to be fast to market with a lot of these things so we leverage whatever is already there um but often more often than not actually we need to not only deploy the cutting edge but actually be and define the cutting edge as well and as i was hinting a little bit earlier one example is the real-time analog detection platform that i built when i joined uber about five years ago and so here it became very quickly clear that with the scale that we're operating at the real-time nature the signal-to-noise ratio because we actually would be sending pager-duty alerts that would wake people up potentially in the middle of the night if our algorithm thought there was a system outage going on and so we were able to break new ground very quickly in this space and one other thing that really helped develop these algorithms to the precision recall that we needed was i put my cell phone on call i was very customer obsessed sorry wearing the pager yes wearing the pager for six teams multiple consecutive weeks i can tell you i did not sleep much i was woken up a lot during this time but that also helped to improve the algorithms really quickly um make sure you want to get it right yes exactly nice nice um so how do you [Music] your team is kind of building very use case specific things you know very close to the end user doing low-level kind of infrastructure as well to support all of this how do you organize how do you build an organization to support all of this yeah so we are organized by data science expertise so we have a forecasting team a non-detection team experimentation team convvi team computer vision team etc basically and then as many other companies we are cross-functionally organized so we have data scientists and engineers product managers designing working together basically building all of these platforms when we zoom into the data scientists we tend to hire full stack data scientists for these roles exactly for the reason that you described we see a lot of advantages to having folks who can write production level code in addition to having this deep domain expertise in a particular data science area and and some of the advantages we see here already start in the design phase so having somebody who has deep understanding about the constraints of the infrastructure stack as we're developing a lot of these things at scale with latency constraints can be highly advantageous because things that might look good on paper might actually not work in the real world once we deploy it in these ecosystems um this is examples of that things that look good on paper that uh didn't actually work out yeah so uh coming back again to the real-time anomaly detection framework so at the beginning we were developing algorithms that would have extremely fine granular data points so we obviously want to have near real-time signal and so we wanted to have an understanding minute by minute or even in finer granularity what was going on and so originally we designed an algorithm that would require you know multiple weeks of data to train uh one one way of how we did this is to frame it as a forecasting problem and then have that mini granularity and that obviously would be an extremely large overhead onto our database systems on that front and so we basically designed the algorithm in such a way that we have lower course of granularity in more historic kind of aspects and then very fine granularities overlaid as a secondary step in order to still capture the variance on for example a minute by minute level so just having that kind of constraint in mind made sure that we didn't require unnecessarily high kind of overheads on our databases and unnecessary infrastructure cost as a result so that's one example okay on that front um example another example of where we see a lot of advantages for having full stack data scientists is in the productionization step so here we're trying to avoid handoffs um in terms of a data scientist writing a script or a white paper and then you know providing it to a software engineer we see a lot of opportunities for errors on that front logical errors in particular and so having data scientists to also write the production level code without such a hand off step is is really important to us and then finally also developer velocity so as we all know there is this prototyping step involved and we're trying to exceed some threshold criterion that we've set out in the beginning of the design phase and it's of course not known when are we going to exceed basically this threshold criterion and so that can lead then to lag times once you have actually found an algorithm you want to productionize that software engineer might be busy with other things during that time and so again having data scientists who can write production level code can really also help speed up that innovation cycle as well you mentioned developer velocity and that makes me think of kind of velocity in the sense of agile methodologies do you mean it that concretely and how do you is there a methodology that you've kind of evolved to or developed that works well in the context of these types of problems so the way we work is very closely in software engineering kind of principles both in terms of best practices in terms of our working structure we work on a daily basis hand in hand with software engineers so we have developed a lot of this but i think you bring up a really good point in terms of you know developer velocity and productivity and so a big goal of the platform teams more holistically is to really speed up that innovation cycle whilst increasing accuracy of the various different methodologies employed right and so the way we see it is we have these four major steps uh within the development cycle of a machine learning or broadly speaking data science problem you have exploratory data analytics then this iterative prototyping phase productionization and then rollout and monitoring and that closes the loop for kind of a new cycle to start off if there is you know a v2 that we want to progress in and so building abstraction layers higher level abstraction lays and this is where you know a lot of the work that my team and collaborators are coming in to play really helps to facilitate not us only for us ourselves kind of this cycle but really for the entirety of the company to really go faster and faster around that loop so you're primarily doing this by hiring full stack engineer full stack data scientists they're not easy to find and we throw that term around like it defines uh concretely a a specific set of skills um but not even not every full stack data scientist is going to have the same strengths how do you how do you kind of grow your data scientists or if you if you find folks that need support in one or more areas or how do you manage the kind of learning cycle for your team yeah absolutely so when we um hire for these roles we also hire complementary of course right so there would be some folks on the team who lean more towards the research side and others who lean more towards the engineering and you know software development side and folks that sit in between so that's one aspect and then there's a very strong learning and teaching culture at uber really continuously striving to improve oneself and so we have a lot of programs even within uber educational programs everything from introductory courses to machine learning all the way to domain experts basically who then give workshops and training sessions hands-on basically courses i've invested a lot in mentorship programs at uber as well building out a community across all of data science and analytics where we then partner folks and and sometimes we also do 20 uh projects uh similar to what google for example does uh where we have people then uh immersed into various different teams to get hand a hand experience basically in some of these domains so yeah i think continuous learning and teaching is something that's really really important okay and what are you excited for going forward what's the future of data science platforms that uber look like yeah absolutely so um i i see both at uber as well as in the industry a big push towards these higher and higher level abstractions for platforms to really kind of commoditize it to make it available to a broader audience beyond data science machine learning engineers and engineers more broadly speaking and so coming back to that four-step data science workflow or machine learning workflow that i described earlier you know one of the gaps that we saw is that we didn't have any semi-automation or automation around data exploration or insights generation as a whole and that's something that actually goes well beyond data science or machining workflows right the first step of the four-step cycle exactly the first step right where there is still a lot of human hours that need to go into that to dig into the data to understand and explore the data and also for business analysts right a lot of the questions that they would be getting is i have a important business kpi and it moved up or down right investigate you know all the various different slices and dices of the data on what might have happened right and so what we have been starting to work on is a proof of concept to actually have an algorithm automatically scan through our data and to surface a potentially interesting insights to folks whether it's machine learning experts or business analysts for example and they obviously would go and dig into some of these suggestions and understand them more deeply but here we see a really large opportunity not only to save people a lot of time but also to really open up new insights that might not have been discovered previously and this is some of the feedback that we're getting from our early adopters in this field that the machine was able to come up with interesting suggestions that they say wouldn't have come up themselves and so i think that really will if successful will revolutionize how we do data analytics at uber and i think more broadly in the industry awesome awesome well friend thanks so much for joining us here at tomokan thanks for having me great speaking with you thank you okay