Common ML Serving Architectures Explained

Key Takeaways

[Music] Rea are you already here tasting hello hello hello hello hello how you doing challenges here like having some challenges it's okay we'll overcome them is this it did we do it yes okay you can okay hi everybody I'm Rebecca Taylor and today okay no sorry Rebecca you go okay so you can share it full screen Yeah so basically yeah as you said I'm Rebecca and um I'm a I'm a tech lead at lle so like little e-commerce and um yeah like we have many models in production and many different architectures for them and different use cases so um I'm like specifically on the personalization side um so basically in general looking at how can we make the user experience better using data um so today I'm going to just speak about like some considerations when it comes to choosing a ml architecture and um yeah like some basic examples of what you'll currently see so you can go to the next slide okay so as we all know it's getting so much easier to like build actual models um I mean 10 years ago they weren't like probably 1% of the packages that are out there at the moment and there there's also so many like resources and boot camps everywhere and you know even on the data engineering side there's so much out there that's available but I think um there's maybe a bit of a gap in terms of what actually is there in industry and what's the the current state of industry and the kind of constraints that you run into I mean you can build these insane models but to actually have them living in production and signed off and you know making money for the company um you know what that actually looks like at least in Germany is not necessarily what you might have in mind or um be as flexible as what we would like them to be okay so um you can move to the next slide so basically some of the things that will really influence what you end up building um I'll talk through some of them but one of the biggest ones I've seen practically is the team structure so different companies have different structures and I I was in Consulting for about five years as like an emo engineer data science dat engineer consultant and um basically what I saw is that there's such a diverse way that companies set up their team structure one of the the common ones is that you have kind of like the data scientists all sitting together building their nice little MVPs and um kind of having these proof of Concepts and um also doing a lot of maybe analytics and things um but then they kind of hand over the their models to like an Ops Team or like um maybe ml Ops maybe devops it depends on the maturity of the company if they even have a ml Ops Team so sometimes their models are actually um not even um used as is and completely transformed by another team right so you get a notebook and you know they hand over a notebook and this model gets completely kind of redone and productionize and deployed in other cases um the data scientist is responsible for kind of giving a model artifact and maybe some config files or or whatever and then there's like slightly less done on the the operational side um in some cases your your model that you make will be used basically just like in a non-production environment for a while to test how well it works and then it gets changed into a totally different language and deployed so that's that's some of the things you really have to to consider I think the stack is becoming slightly more coherent U well slightly and I think one of the things that I've seen a lot more lately is kind of a product structure so you have kind of like a mixed team um where you'll have like um in one teams like data scientist data engineer ml engineer even some devs um business analysts and that they'll deliver like endtoend um systems so this is what we have for part of the business in Leo at the moment which is quite cool because um what ends up happening is we're responsible for um basically like parts of the front end all the way through backend data engineering MMO engineering feature stores well not like the infrastructure as much of it but like what ends up in the feature store um you know pieces of the data Lake that we own um and the whole Tutt so even you know the monitoring alerting so like as a team we're responsible to deliver stuff and um yeah that makes the collaboration quite interesting the challenge there is when you have a lot of models in production then who maintains them while you're the same you know people that are building the new model so um what often happens is you also have like a Ops Team that's like um going to be on standby and they'll also be briefed on how to handle these models and stuff but but basically if you don't have like those kind of mature setups um then you actually really have to consider carefully what you put into production and what your design is because if it's not going to be able to be maintained um properly and um if there isn't correct alerting and also like change like as your model if you want to retrain and do like significant changes um you might not get the budget to do that right because there just you know isn't that option so yeah and oftentimes like the ideal model never makes it into production just because of these type of Team constraints um and then another constraint that um we often like run into is um that there's like predefined tools and Tech that you're allowed to use right so you might be super experienced with a certain stack and you join a new company and you realize nope you know you're not allowed to use that there specific things that you you're only allowed to use um that can also be for example um from like a location point of view cloud services you might want to be building the coolest New Gen model or using it it in some way but um it's it's only available on like gcp or something or AWS and not Azure just for example um also from a like a like an Integrations point of view there's certain things that work well together and others that you know maybe don't work as well together and if you have bad luck in terms of the architecture decisions of your predecessors you might be forced into combinations of tools and Tech that actually are difficult to work with and then a lot of time is spent kind of um gluing things together right so um yeah I think that's something that also could be surprising if you if you're new to the industry and realizing or new to a different role I mean this is actually maybe a tip for if you're going to choose a new role to really ask questions in those interviews right so yes they're interviewing you but you can also interview them in terms of like L like the tech stack and ask detailed questions about that and how like I mean do they have a in-house ml platform that they're maintaining and building are they using something that's already like tried and tested existing like are they using like a more flexible type like for example like zml or they like locked into you know something else right like Azure ml for example um yeah th those decisions can significantly impact like your designs and what you end up being able to build um also the industry so and business requirements so that's super important like I mean in some cases yes you can build a you can build a nice um data science model that pulls features from like you know some cloud storage and you know um does a prediction and you know sends it back or whatever and that will work for you right but in other cases pulling from storage for example is just not going to cut it you're going to need um you know much more um you know low latency Solutions and you'll need to be putting caches in place and often um like on our side basically we have quite Advanced caching systems on our website as well which means that you know you'll be caching things on your side potentially in your model apis but there's also front-end caches that you have to consider and you also have to look at like what those kind of um systems do so um this is something that like can really constrain your design especially if you want um like near realtime features some of those features aren't available in the way that you think they are from certain systems so you know if you're in a in a situation where um the actual full design of how events are happening um in your front end this is just in like an e-commerce or banking maybe example um if you don't if that isn't set up um in a in quite an advanced fashion like maybe you don't have a proper cka solution or something going and you can't access the events in in time it can really constrain your design and you have to get quite creative of how to still deliver some good business value um even you know um with these constraints in mind so okay you can move to the next one so here's just some really basic patterns so um yeah this is the example of you you have a model you want to um perform inference on this model and um what are some of the the ways this will look right so the the most kind of easy obvious one is a batch scoring job so basically your input is a is like data multiple tables or whatever and your output is also a table right so that's the the easiest it's nice you can have an interface where you know you are just um putting put things into a table and downstream gets to consume that and serve it and do whatever they want with it or maybe it's used for some forecasting or some pricing models or predictions or something um you know reports I mean that's the one that like yeah it's easy and there's normally no issues then um in the case where it's also batch scoring again and often you'll see batch scoring in like Marketing Solutions right they're like here's all our clients or you know let's run some batch job and do some you know prediction or clustering or something like that same also in like um you know banking for clients they they might have some like um for example if you're looking for um patterns of fraud or things like that sometimes they'll do General screening batch jobs that will just run and then there can be analytics done on top of that so if you like in that space that's quite nice um but then the the one step up from that is normally like if you're also doing the serving side in terms of like the results so it can still be like a batch scoring job so the results are written to a database but you also expose those um that that database basically it's the information of that database is then accessible by an API so you'll basically just have some um like get request and you can then pull the the model's results um either for one like instance or one client or one whatever or um for even a batch of them potentially right but it's available by API and then that's easy to consume from other Services um and then the the slightly more um tricky side is when you need to have prediction kind of on the fly so the prediction part is real time not the features but the actual prediction happens real time and this is very common when there's any type of data science happening in webites right so the person will be clicking and doing things and based on what they've what the current state is of this person of this customer um you can you then want to make some prediction right and the nice case is where um everything that you need for this prediction so all the features not necessarily like um pre-processed features but at least the the raw information for the features if those are available um you know by the front end or in that layer that's calling the API then you can just have your model in the API and just you know run predict and get an answer right the the problem comes in and that's the easy this the nice one right but often this isn't the case and you have additional sort of historical or slower moving features all features that you have to get from somewhere else and that means that you get your some of the the features in in your payload but the rest you have to get from somewhere else and when you have to get it from somewhere else that's normally going to be um these data engineering jobs typically that will get it to the place you need to read it from you'll probably have something like Rus or some caching layer as well or I mean maybe you're using some like something like which also has a bit of caching built in this is where you have to also worry about cost a lot and um yeah basically tuning this um sort of online feature store basically right to to handle like what loads you need and um what latency you require basically so that's sort of the the harder one and it gets harder as um you know like you even move into as I put in the last Point like really hardcore streaming stuff so if you have like a full-on realtime transport that's um you know used in a prediction sense as well so that's not as common you know really like used in industry I think you get most of the value already by the the first points okay we can go to the next one wow this is really small I'm looking at it on my phone um so basically this is just um like a batch scoring example with a little bit more um detail so this is the simplest case basically where you'll often have like multiple data sources I mean I just showed two and then you have some Transformations and mapping jobs and things like that and then combination of the sources and then um like in our case we're using like Unity catalog for our like data layous thing um but yeah so you have some tables like or maybe one table there and then you have a a batch scoring job that basically has access to your MMO model um and it just runs predict basically over you know all of those entries and obviously these jobs all these blue jobs are data engineering jobs and they will then be you know optimized as well and um the Clusters that you're using to run them on um if you're using clusters will be tuned and um you know like customized for the actual code that they're running on to save costs and then CU often this is a lot of data that you're working with right and these jobs can you can if you write them badly they can take two hours if you write them well they can take 10 minutes or something you know or if if you optimize the cost there will be huge differences right um also using things like spot instances if you're in data bricks please you know do that so you know where you can if you're doing a patch scoring job for example um and then um yeah so once you're there then you typically will for um future analytics and for you know traceability in future you your write the these predictions to a historic table I mean you can also have different data modeling ways of looking at things right so you could also have one table but you know when you pull the data from it you can just pull the latest but often storage is cheap and computer is expensive so we kind of just have a historic table and then we have a a current state later stable and then we in our case we have a job that writes that to um and then you have a prediction collection in that you can look up and obviously there's all the partitioning things that can come into that it could be multiple tables you can handle it how how is needed to be handled but this is just an overview of what often is used okay and then next one so this is now the this the serving part of a solution like this so this is um basically an example where um this API will um have a model inside the API right but it will also um have data that is being pulled from predictions basically that have already been made right so this is also a common use case in something like um e-commerce or banking where you have customers and you'll have like old customers and new customers so in some cases you have people that you have data about lots of data about like historic data about purchase data or whatever and other cases um you don't you only have the the real-time features that from their current clicks or whatever or you know um maybe just some information about them right that's available from the front end um so you'll actually have like the the model in the previous slide making these patch scoring predictions but you'll have a different model um that hopefully does something very similar um just with less data um and it will then um make a prediction um for everyone where you don't have the historic features for so basically if it's existing user you you pull the the features that you um have already pre or not the features you you pull the the predictions that you've already made um directly in that case you can end the loop and return with with a lower latency um well not necessarily lower because yes it will be am I coming to an end yes yes okay okay I'm basically done okay I'm basically done okay um but in in the other case then you have to still pull the features and make the prediction and then you can return so just in closing maybe go to the last um bit um yeah I'll just summarize saying that basically um you know you have to really think about many things when you're doing your design keep things as simple as you can build on that make sure you're getting value and speak to people that have done stuff before wonderful Rebecca thank you very much for this I love these types of presentations to just give me like a sense of context and overview about like all the different things even just to make me feel less crazy for picking something that starts out a little bit more basic before going to just like a full or kind of like more heavyweight solution so thank you very much for that [Music]