Machine Learning is Going Real-Time

Key Takeaways

hi everyone my name is chip and today i'm going to talk about real-time machine learning a little bit on my background um i come from a very non-tech background i have actually published a few books before i switch to doing computer science and i start with product management and then i i work in machine learning at prima netflix i'm currently with snorkel ai as a machining engineer and open source lead i'm also teaching music system design at stanford um so as a motivation for this talk i just want to bring the attention tech talk so who here has heard of tech talk because i can't see anybody's hand so i'm just going to assume that everybody raised their hand um tick tock is probably one of the biggest one of the biggest uh social phenomenon in the last few years it totally eclipsed like what came before they buy snapchat one one objective that i keep hearing people mention when i talk about tik tok is that it's incredible incredibly addictive there are many reasons um why talk is so addictive but i think one of the reasons is the technology um some other companies that own tech talk bike dance has built an incredible infrastructure that allows them to do real-time machine learning it means that they are able to learn user preference in real time so if you watch something right now they can learn uh where you have this watch and choose like what to show you next uh and the recommended recommendations are really good giving users an incredible um scoring experience so take talk is one example to show how powerful real-time machine learning is and this is what we're talking about in this talk so uh there are two level of uh online machining one is like online predictions and another is online learning but before we start uh let's just take a step back and try to see like what does real-time means um it really hurts like a lot of philosophers saying there's no such thing as tom and if you talk to software engineers they're gonna tell you that like there's no such thing as real time because it doesn't matter how fast the system is there's always some latency which is like milliseconds to nanoseconds there's some some latency and some people that should be technically correct they could determine near real time manila and this term is like quite not very natural so far as it was this talk i'm using real time to address uh to encompass all of that um so the actual level of real-time machining first is level one online predictions when the system is able to make predictions in real time and in this case real time is defined to be like from milliseconds to a second and the level two is online learning the exhaust system is capable of incorporating new data and updating the model in real time so in this case real time is or it defines me in sort of minutes i also want to clarify what this means in model and system because both of them are terms that have like have a lot of meanings so model is machining model for example bird gbd2 and system is encompassing all of that including interface infrastructure data compute um so let's go to the first level online predictions um so if you have view systems for users you will notice one thing that latency matters a lot there have been many experiments conducted to show how important latency is there are two examples that i really like one is by google and it shows us like if you increase latency from a hundred to four milliseconds can reduce searches like actual point six percent and another ebook in the comment for thirty percent increase in latency can cause like qualifying conversion conversion rate and you know conversion rate is tied directly to revenue so part 5 conversion rate is actually can reduce information it can reduce a lot of revenue and like no matter how great the machine machining models are like if this take just milliseconds too long to to make predictions usually i'm gonna click on something else so you have noticed like in um in the last decade the machining research community have taken on this approach like bigger like better performance so at first like you think a million parameters was crazy but then they went to like a billion parameters and now we're talking about trillion parameter models so as mono was getting as models get bigger the performance become better but so the latency like the inference lens it takes longer for this model to make predictions so so one of the biggest uh problems for for using muslim machine learning this is large mole in production is that it takes too long for for bird lgbt to to make predictions so so i think one obvious non-solution is just not try to make predictions in real time like if you don't do online predictions there will be no online predictions latency so so a lot of companies are doing what they do is called batch predictions so we generate predictions offline and and then you store on this prediction somewhere like in a in a sql table and whenever users put a query you just like try to pull out pre-computed predictions uh from the table so see you don't have to do patients uh online and batch preparations have worked pretty well in many cases um so um even companies and netflix are steering batch predictions for their recommendation system so so whenever you go through netflix you will have a see on the recommendations for you and it's right this recommendation and not just real time but uh but update maybe like every few hours um depending on the on the specific uh specification from netflix but uh but if you refresh as well so those are the order of rows might change but the exact movie recommendations won't change so there's a couple of like problems with batch predictions uh the first is that you need to know exactly how many predictions you generate so so so input the input space has to be finite so in case of recommendation system for netflix so netflix knows exactly how many users there are actually to generate recommendations for and if there's a new user joining they just might jared some like um some like generic recommendations like maybe most watched movies or something like that um the second problem is that um it can't really adopt a changing interest um so so say like you might be on netflix and you and you you're having a lot of horror movies lately right and so when you go into the netflix home you will see a lot of recommendations for horror movies but you're like very happy today and you want to send some comedy so you go like search for comedies you go to comedy category you see and you could think that netflix should which are doctors or like interest now right but no like you won't get new recommendations until netflix uh gerry's the next batch of predictions so so far many companies uh to continue doing batch predictions they they they force suppression speech to be finite by make it discrete so first of all to advisor you go to advisors first they make you like choose a metropole metro area for example san francisco or new york and then you choose another category restaurant hotel if you try to do something like different like input some like free range query like high rating thai returns in hayes valley you will get something like results that's really not relevant at all whereas if you enter the same query into google you get like slightly better results but still not perfect i think and i would argue like one of the reasons that it makes the results better on google is just google does real-time predictions so one another benefit of online predictions is that you're able to use nozzle static features burn some dynamic features study features uh something static for so age gender neighborhood job income of course you know static static i think they never change but like they're less slower to change dynamic features are something that's happening right now for example what is what are you reading right now what are you watching right now what you're interested in right now um uh so and and and this can change because um and our vibration will allow you to use this um dynamic features so to do online predictions uh for you need shooting first you need fast influence you need a model that can make predictions in the order of little seconds and second is that you need a real-time pipeline which means the pipelines that allow you to like take in data process data input into model and return predictions in real time so so for the first part fast inference is something that people have been working for in the last few years and a lot of research on it so there are three ways to make a model do inference faster first is that it makes the models faster the second is it makes the model smaller and usually one more smaller it runs faster and tourism makes the hardware more powerful so macro faster you can make a model run faster by like for example using operations right specific kernels optimized for different hardware the gpu tpu um anyone example tensor rt is doing pretty well inference optimizations uh making more smaller also known as moral compressions and there's so many different techniques that people can use and one of the most popular techniques is quantization to reduce the precision of the of some more amount of precisions in in like performance but precision as in like a floating point uh uh hurling by bit so usually if you store a normal float as a weight so they take like 32 bits but if you reduce it by half then you you don't need like 16 bits and a lot some companies are doing integer so you're only eight bit so you can reduce a model size and for also like smaller bit numbers also require less energy computations faster as well and other techniques like knowledge distillations um it's a student teacher techniques when you train a small model she may make the behavior of a larger model um so there's so many different open source um projects for this and i think there's one example you can see in the fourier tomorrow compression open source project and if you're interested in how on how this model compression is being used in productions i would highly recommend you read this blog post by roblox by how they apply different techniques and they're able to reduce reduce the latencies of their birth model like 30 times metagram of power 4 is another very exciting area so people are trying to build hardware and make it fast for both training and doing inference on both on the cloud and on device here's just some of the example of setup has raised a lot of money recently and you can see that they are um scratch all over the world and not just in the bay area so the next component of how to do online production is slightly bit complicated so it's a real-time pipeline so to start let see let's see an example of a ride sharing service and you want to detect whether a transaction is forwarded or not so to make the predictions you need not just information on the specific transactions you will need to see any other like user's recent transaction for some of the last seven days uh if if the transaction use a credit card you might want to see like how when was that credit card added um the recent transactions because if the credit card is just added a few days ago and it has been like used maybe like 200 times probably because like people it's pretty fraught because people are trying to optimize the most use out of that like stolen credit card so also recently enough for us to see if there's some new recent trends and people are um some people come up with new ways to listen scan the system so a lot so this little question is like how do we quickly access these features because you don't want to like store on these recent features on on a permanent storage like s3 because like just reading things out from s3 back to the systems take a lot of time and slower other systems so you want to store one of these features in in memory and with coins of like in every storage so every time an interesting event happens like first of all users picks a location they book a trip they add credit cards they contact the driver they cancel a trip so all these events go into the in-memory storage and it stays there for as long as it is useful for example seven days and after seven days um this will either go to like permanent storage and s3 or it can be discarded because it's no longer useful and and what it's called like this uh and it's gonna stream storage and them are like several popular solutions for it because the most popular one is probably apache kafka amazon has also systems like penises and you can see if you go to stack share there are like 100 companies are using the technology so so if you have like owns all events into a stream storage rate and then how do you use how do you process them to influence or the model so so you so to to make uh predictions on weather transactions it um fraud you need not only like the features from the stream storage uh stream data you also need feature from static data so static data is something that has like existed before and stored in some format like csv or parquet um and static data is something that's about it like you know how how big it is and if you tell like hey process all the data in that file like if sorry that you know when the job is done but whereas trimming data is unbounded it keeps always coming you will never finish something that gives you static features like age and gender when a guy was created the user's writing they make features give you like more dynamic um they're gonna give you extremely give you more dynamic features like locations the last 10 minutes uh recent activities um in the last row i have this hand waving thing not because like are waving at you it's like uh it's it's like it means it's very handy with explaining things uh so you'll see instead it did that because you know exactly how big it is um uh so so you can use tools like um uh sql like map reduce to efficiently processing in batch whereas in uh streaming data you want to process things as soon as they come so it consumes streaming uh stream processing and some of the common tools for is the apache fling and some of them and if um and associations are two different ways of like dealing with data like with static data and stream data and we know that like in training currently a lot of us still train models offline so you have some massive map radar somewhere and you train it in batch excuse mesh processing for face of pipeline but whereas in productions you if you serve things uh online you ever every time like um an event comes like a data symbol coming once you migration on it so you're doing trim processing so you see i get one model but two different pipelines and one for batch and one for uh for stream data and it's one of them one is a common core source of arrows for productions so is it especially common when these two pipelines are maintained by two different teams for example you have the advice for like for the best pilots maintained by the machine learning engineering team who does on the training whereas the trim pipeline is maintained by by the dev ops team who does who bring model to productions so like one changes like changes in one palette are not correctly uh are not correctly um uh replicated in another pi pipe and can cause um because arrows like because they use different features now even i think in the extreme actually so it's very common you can see that in a lot of companies here is a slightly older version of the waypoint by since it's online pipeline and offline pilot um but recently rainbow has made a lot of effort like to use amateur flank to unify this shoe pipeline and there's a new unified and this has happened like once alibaba and uh uber and lyft so so we have seen the difference between stream processing and batch processing and the next stop i want to talk about the difference between um request driven and event driven and if this term mean nothing to you yet don't worry i was gonna go talk about it so at this point you might have heard about microservices microservices has taken the world by storm in the last six years and microservices go often go hands in hand with rest api um and rest apis are request driven so what i mean is that you have like for rest api you have the client and the server and the client send a request to the server to tell us exactly what to do by request i post and get and the server responds uh response should choose a request um so a server has to listen for the request to register so if the service dial the client would have to keep resending and resending and resending so i think let's take a sample of a very simple application with only three micro services so one microservice is to manage writer's demand another to manage driver and other manager price optimizations so the price demands on demand and availability so you would need uh so you would need to ping both services for for the rider demand and the driver availability and then as a driver management microservice we need to get the demand from the rider to see like how many more drivers need to mobilize and need to pay the price optimizations to see uh what price uh what search charge you can use to incentivize drivers and you see like it's very complex um so there's definitely a couple problems so first it's just like because every service does its own thing uh it's it's really hard to map data transformation to the entire system and if something goes down with one of the systems of price optimizations you can need to trace to like a and b to see like how it affects the performance of c and it's very very difficult to do with request driven micro services so so so what what if we we don't we get rid of all the inter service communications by by this model so only services can just publish own events in their service into a stream so uh and and owners of services who want information from there can just like subscribe to the stream and pick out what information it needs um so now all the data flows through this stream uh you can actually create dashboards to monitor your holiday transformations through the entire systems um and and also like and this is what is what what is uh kafka is doing and this model is called pops up it's not the only architecture for shrimp um but um but it's a pretty popular one so so you can see that a lot of companies are doing doing this uh but um i think there are a lot of companies are still not doing it um so there's several barriers to it so first companies don't see the benefits of stream processing yet so the system might not be as a scale where uh data transferring between surfaces can be problems or like best bridges have the scope 5 for them and they have never seen they have never done online predictions before they have no idea all operations can improve the performance and therefore they don't even try so there are several different other reasons like is high in initial investment on infrastructure so if the company has been doing best predictions and to want to switch to stream processing uh or online predictions we need to require a lot of investment in fresh russia not a lot of companies are willing to do it it also is another mental shift so you get people academia they only use your thing in batch analysis it would be streamlined it can be very disorienting like what it means to like for them to never finish every so um uh tools for for for online predictions uh like kafka and flink they are built on java and a lot of machining people are really comfortable with python and they might feel like a little bit like squishy if you like uh you have to switch to java uh so for the next part is online learning and this person will be quick uh mostly because are out of time so so i want you to like before we go inside to make a small distinction between online learning and online training so in by definitions online training is like to learn from each incoming data point and very few companies actually do it because it suffers from crash shopping for getting and so it can get very expensive as in like why use using a very big hardware to use like to learn from just one single sample so in practice i have seen this a lot of companies who do a lot of companies who do online loadings do it in micro batches so you learn from um so so you can wait uh for like incoming samples and you can get like maybe like um 100 200 500 and then learn from it and then you evaluate after a certain period of time for example like five to ten minutes and you need to there are two different types of evaluations so first off like evaluation so i should make sure that like this model is not doing something like is like really really really stupid and and then uh because all the point online learning is that you can adopt the system the changing environment like first of all changing preferences so it so you would need to evaluate on the changing environment as well so you can't just rely on a static test set so so coming to it by doing like uh by forming out you to actuate users and see how it affects like performance with actual users and it's a very complex system maybe testing is complex because you're not just you're just not testing between two models but we test among many many different versions of a model so online also that doesn't mean like you don't do offline learning so a lot of companies are still doing online learning in tandem with offline learning and they try so it can give you like a more stable version um here's an example of a weibo uh here's another talk from the same talk i referenced before and their uh iteration cycle right now is about like 10 minutes from like getting new online dynamics new features to like evaluating and deploying um so they are like so the biggest use case for online learning as i see right now is recommendation systems one reason is that regulation system is um has very natural label so if user click on something it's pretty a good predict a good predictions so you don't have to wait for like uh humans or domain experts to like label the data you can just get laid out like on the go um however not own recommendation system need online learning so for things that like have fossil preferences that like change slowly uh like houses cars flies hotels you pre don't want to learn continue learning because it'll be a waste of however uh preference uh for a lot of things can change very quickly like especially for online content like videos articles tweets um stuff like that and and for that you want to learn user preference continually and also because preference for online uh for online content change also means that like advertising system we need to also learn continually as well and um and you can see online content and advertising everywhere so you can see the uses for my learnings can be like everywhere um other cases for example like um rare events uh think about black friday for shopping uh because black friday happens only once a year there's no way you can get enough historical data to learn what the user is going to do on black friday this year so you will need to learn uh about user behaviors that day on the spot or oregon so have you deal with ghost star problem when you ship an app to an user uh you don't have information that you say yes you don't know what they like so you will need um so so if you wait until it's the next time or you learn to tune the system in and shift the new model it might be like too slow so so you want to continue learning sports to get the preference of the new users under similar with on device so on online learning um it's very challenging it has a lot of theoretical uh challenges um so first it flip a lot of things that you learned about machining on his head so first of all if uh people who have learned machine learning in school we probably have heard some sort of resist like similar version like change a model with a sufficient enough for sufficient number of epochs until conversions and evaluate on aesthetic basis right so so you know um online learning there's no epoch like um you you know you you will see each data point at most once i say levels you don't even see at all and there's no conversions because there's no static distribution to convert you like it's already kind of shifting and there's no state test set because you actually have to test the productions and and not you can't rely on a single test set uh to evaluate anymore if you are interested in learning more about the topic i'm actually taught teaching a course on machining system design at stanford and the materials can be online and you also get in touch with me via twitter or my email or website and i'm gonna be geometric q a sessions now thank you very much bye