How to Design Machine Learning Systems (Beginner to Pro)

Key Takeaways

Hello everyone. Today we'll be discussing chapter two from the book that I've been reading designing machine learning systems by chip yung. So yeah let's dive into the topic. So uh this presentation explains the overall content in uh from the chapter 2. It's better to read a book always but I'll just give a gist on how what are all the things that are covered in chapter 2 so that you can just go back and read it. You can understand it effectively. And uh the first thing reading a technical book is not a uh easy task like a fiction books. uh I kind of read the chapter once or twice as I'm a beginner I just got started with uh and I convert this entire thing into a podcast I listen it three to four times and then I uh try to understand uh that well that's not enough you need to try to see things in this particular point of view that what you read so I'm just trying uh let's see how it goes but yeah uh coming to the topic so chapter 2 is all about fundamentals of machine learning system designs so let's see what it actually is uh so this chapter actually uh helps you to uh move from from the regular student mindset to the production level mindset. The student as a student myself uh I generally think about uh the code plus data is equal to the model and that's all that matters. The main goal is the micro accuracy and the basic context is static data sets and then endpoint would be deployments in the finish line. But that's not it. But when it comes to the production data set uh it includes a number of things uh actually as I told in the previous chapter model is just like 10% of the actual system that you'll be working on. We have data pipelines, monitoring, business logics, serving layer, value extraction and many more. The main goal in the production level is to solve the business problem. It's not to get a good accuracy. Obviously, getting a good accuracy is one of the things. But the main target and goal should be solving a business problem for the client that you are working with. So you uh if you got very good accuracy in a particular model and if you are showing to the client uh he doesn't even understand what it exactly means. He just needs how does it matter to the business. That's the goal. So just think on the other point of view, the business point of view and then the context here is shifting data. You'll uh get a good understanding of what it is uh we are going on. Uh and then the end point is deployment into the start line. Uh sorry deployment is a start line. So uh deployment the ML life cycle here is not a one time thing. It's not like a base that has an end. It's a loop continuously going on. You will be getting uh initially the data you are you will be training the model on uh then you will be working on the data training the model deploying and then a new data comes up. It's not time not like a one time data new users comes up their pattern changes their uh behavior changes over time. So you need to train the model again on the new data. It's just a cycle. It never ends. So you need to keep an eye on it. Deployment is just the start line. It's not the finish line. And training a model is the easy part. Sorry just a second. This was right. Uh training a model is the easy part. Building a system that generates value is a good Okay. Yeah. So, uh what it says from the book to tagline kind of thing. So, training uh we are all we were always thinking training a model is the hardest part and it's the core component of the entire ML life cycle but it's not it's the easiest part kind of not completely but the main goal here is to build a system that generates a value is value sorry able to see it. Yeah. Uh the system that actually generates value that's the goal actually uh at end. So yeah, what are the things that we will be covering on? I'll just walk through the entire chapter uh fast. So what is the actual goal? I mean what are the areas that we will be covering in the chapter. So the first thing is uh understanding the business and ML objectives uh what is the golden rule of value and then the system requirements uh the four pillars of production uh when when you're working on the production level applications and then the iterative process training ML problems and then mind versus data which actually outperforms uh which drives the success basically yeah that's the key things that we'll be discussing on today uh yeah starting with the golden rule business objectives and the ML objectives which actually plays a key role Right. So, uh here the basic concept as a beginner uh yeah here the basic concept is as a beginner uh you obsess over ML metrics like Azure CF score or low latency. Right. But as a pro uh you need to understand and you basically realize this uh companies care about business metrics like revenue user retention and cost savings it's not completely on the accuracy part on the metrics mathematical technical terms you won't even understand it. So the major disconnect here is uh increasing model accuracy from 94% to 94.2% 2% is useless if it doesn't actually adds value to the business. So based on the business requirement, we need to understand things and work accordingly. So the major translation here would be you must uh translate the exha not the other way around. Uh um for example you can see if a business wants to increase ad revenue uh a business metric and then the model metrics might be the clickthrough rate. So uh the ads generally work on number of clicks right uh but here there might be a small trap that sometimes better model hurts the business. So if you are actually working on the clickthrough rate the example that they described in the book uh if there are some uh thumbnails that were interesting and the content inside that thumbnail is junk there will be a higher clickthrough rate to that particular thumbnails but eventually does it actually it uh increases the quantity of the clicks but actual quality of the content it completely ruins everything right that's how the traffic to the website decreases here the model that you gave is working perfectly fine it's working on the number of clicks but for that particular ad but eventually the content content that user is experiencing it's it doesn't have any value and the website goes down. So that's the major uh point that you need to understand accuracy doesn't always gives the uh best results to the business. So you have to understand that point. So uh before even writing the code the major uh think like a pro point you can consider from this thing uh before even writing the code uh if the model achieves 100% accuracy how does it actually help the company make money or save time. So that's the major question that we need to consider. If at all the model is giving very good accuracy, how does it actually help the business? Do you have any edge cases on the top of it? So if you can't answer this particular thing, you don't need to build the model at all. You don't uh question yourself after building the model. The entire you'll be wasting the time on the basics. Yeah, that's the thing. And the major four pillars of the system requirements. They are the standard pillars. You need to uh work on them. You will be working on them almost all the time. So they are reliability, scalability, maintainability and adaptability. I generally just kind of remember it. I remember them as Raselma kind of thing. So, okay. Yeah, that's how it is. Uh reliability, what reliability actually means? Uh as I told ML fails silently. Uh it it's not a software that crashes if there is a a different type of input or something. It uh basically gives you a bad output. That's it. It doesn't crash. So, you need to understand the uh how the things are, how reliable the model is and you need to keep an observation on on it all the time. And then comes the scalability, the resource management. It's not just about the traffic volume. It is about autoscaling GPUs and managing the artifacts over 100 different models. So basically when it comes to production level, you won't be working with a single model. So you uh you need to understand and scale uh the entire system accordingly and you need to automate this entire system. You won't be sitting at the computer at 2 a.m. or 3:00 a.m. in the morning uh changing on these uh resources when the uh data or requirement changes right you need to maintain the scalability and automate at the same time and maintainability. So what maintainability actually means? uh it is like how easy is to is for a different person to use that particular work that you have built. So you don't force one tool on everyone right uh ensure the code that you are writing code or data whatever you are working on are versioned so anyone uh who are coming up in the future to join your team can reproduce the same model exactly so that it won't break anything and uh yeah basically that's the main thing so that's the maintainability of the entire system that you are trying to build and adaptability once again as I told the data shifts it doesn't maintain the same patterns and structure all the time so uh the system that you are trying to build requires a manual retraining for every data change it is probably obsolete. So uh you need to keep an eye on the adaptability of the data, the change of data and uh retrain the models accordingly with the change of patterns. Yeah, that's those are the four pillars and coming to the iterative process basically uh as it uh yeah the ML life cycle is not a oneline thing. Uh it doesn't have a start and an end and you can't just uh move out of it once it is done. It's a never ending process. uh when once it's basically starting with the project scoping, data engineering, ML uh model deployment sorry ML model development and then comes the deployment and then comes the monitoring monitoring and continual learning and then uh the business analysis comes in and that's that's a loop it goes on and on as the business uh requirement changes or adapts you need eventually the business any business need to adapt adapt to the market over time right as that uh those adaptation changes and the new types of data is coming in the you need to work on the entire life cycle one more time so it's a never- ending process you need to understand that particular thing. And uh one more thing so thinking like a pro in this particular uh section would be like uh expect to go backward you will deploy realize the business metric doesn't move and you have to go back to the project scoping or data engineering and then the process is never finished. So that's the baseline that you can remember when you're actually working in the production learn and coming to framing the problem itself. So uh you need to basically uh yeah your uh generally uh let's consider uh a scenario where your boss gives you a vague problem like customer service is too slow. You must translate this particular uh problem into a mathematical problem with inputs outputs and objective functions. So uh generally considering the type of task it we might have a different uh types you can consider them as a classification problem or a regression problem. uh let's say in classification we have generally we'll be putting the things in different categories right it can be binary that is two classes multi- classes uh I mean you have many classes but only one is correct and then you have the multilel part where uh you have many classes and uh multiple can be correct at the same time and so on we have different classification and regression types you need to understand uh the which type of uh task is required for the given problem statement effectively and then you can work on it uh and yeah uh reframing this problem uh you can also consider it as sometimes you should change the task type to make it easier. Uh yeah in one one of the scenario from the bug. So uh they were yeah this uh example just yeah so uh the main thing is you need to understand so yeah uh uh designing uh begins actually by fitting the pro uh business problem into the correct mathematical frame. That's how uh you need to basically if the question or the problem is big you need to understand how better you will be fitting that into the mathematical formulation and then you can actually start working on it. Uh we'll just take one example uh scenario. So how to reframe a uh given requirement. So the scenario actually have this uh predicting which app a user will open next. So that's a basic problem statement that we got. uh generally as a beginner uh you might be considering you can actually classify this uh you can consider this problem as a classification and you can bucket the different uh different types of applications and predict according to that let's say you created a multiclass classification and the output is the property of every single app installed and uh if the user installs new app you have to retrain the architecture again so that's the problem that you will be facing when you are working on this classification part but when it comes to a pro framing problem framing you can actually do a regression problem where uh user uh uh input is the user plus app's features. Uh how what is the behavior of the user and the features of the app instead of the app itself so that you can understand uh you can actually get the output of the score for that particular app according to the user behavior as well on how likely they'll be clicking on that particular app. So that's the regression problem. So cons uh change consider the change. It's not like you need to consider only classification to these kind of problems and regression to these kind of problems. You need to change based on the behavior and requirements. So at this particular uh situation even though a new app is installed you don't need to retrain the model to understand which bucket it falls in. You can understand the basic features of the app and then uh the actual model works for that to understand uh to get the score of the particular app. That's it. That's uh how you need to reframe the entire problem statement according to the scenario according to the situations. So yeah that's the thing uh and then the objective function context. So yeah there is a situation uh uh site yeah the example I told earlier uh when you consider okay uh so yeah uh sometimes there is a situation where uh you might have a context between the multiple goals that you are working on uh like the goal one is to get the maximum clicks and goal two is to minimize the field bits the example that we discussed earlier. So in this kind of scenario these are complete opposites right the model can't work on two opposite things. So here the major solution that you can think of is to decouple the objectives itself completely. Build a model that works on one thing and a different model that works on different thing and then try to club their uh results together like uh instead of generally when it comes to beginner uh they'll be trying to create a complex loss function that works with the quality loss and the engagement loss as well. But uh at a pro level uh train two different models separately. Once the uh predicts I mean one predicts the quality and the other predicts the click clicks right and then combine their uh combine their scores after prediction and get the best one out of after combining those both scores. You know you are not basically combining the entire models itself. You are just comparing the scores and then allowing to tweak the balances between them without actually retraining the models completely. So that's the major difference between a beginner and a pro. Pro tries to complicate things uh building complex uh you know formulas to get effective results but uh thinking as a pro you can actually decouple them completely work on them separately and then just uh combine the scores to get better results that's how you need to think it's a complete different way of approach and then uh yeah the solution as I told uh earlier you need to uh just decouple both of them uh you just create a rank function that gets best out of both and then the final recommendation So here the major rule of thumb is to keep the conflicting objectives separately always. And then comes the mind versus data thing. So I'll just give a quick example of uh this. I'm not sure if uh everybody relate to that. Uh if you're actually trying to go to gym to build a muscle or get into shape, you are training your body like let's say 2 hours per day but you are eating junk the entire day. Uh will you be able to see the results? I'm pretty sure you won't be able to see the results and uh you uh know that why right so that's exactly how it works in the when it you when you are working with ML models as well doesn't matter how much training you are uh you are I mean how much time you are training the model it matter what you are feeding the model right if you have junk data the model doesn't work at all no matter how complex the model is so you need to first work on the data and then work on your complex algorithms or models so eventually what you understand the data always wins so uh our general methods that average computation uh that is processing massive data consistently beat clever human design rules. So uh generally considering the example it's not example it's a reality Google always wins and sits on the top of the layer because it have a lots of a lots of lots of data that it can train upon that's the thing but it is able to adapt to any new technology faster and not just the superior algorithms it doesn't have the algorithms it has the big brains in their companies according along with that they have the data so that's the key thing stop tweaking the architecture for one person gain and spend the time on cleaning the data uh Yeah, the entire complexity the entire results of the model relies on the data that you have. The cleaner the data uh better the results. That's it. So eventually the the system designer checklist would be value first. If sorry if it's it generated a different form but yeah uh the value first if it doesn't move the business metric don't build it. The business metric is the key. Uh that's the main thing and then comes the silent uh oh I mean silence the silent failures is the key. So you need to keep an eye on them. Build monitoring for silent failures, monitoring systems for silent failures and uh work on them constantly not uh yeah and then uh close the loops plan for continuous monitoring and retrain from day one. So yeah it's uh you need to understand basically that's a closed loop and you uh you keep on constant monitoring of retraining the data uh for the different types of data that is coming uh coming in and uh maintain entire system from day one for the end of the crashes as well and then decoupled separate business objective problems uh goals when they are conflicting and then comes the data host better data is always high prior to I mean yeah better data is always uh carries the higher priority than the complex neural models as well so that's the key role Uh I hope you understand it. Uh yeah, I'll try to explain things better and faster uh going on. But yeah, I hope it added some value. Let's see in the next session.