MLOps x Generative AI best practices

Key Takeaways

I'm Divya Jain. I lead some of the training infra teams here at Google including TFX. And today we will be talking about LM Ops, which is in a nutshell LM Ops for GenAI. Um but the way I want to structure this talk today is to first talk about why we need ML Ops, um why they're important. I do want to give a very very quick refresher of TFX. How many people over here have actually used TFX or they used TFX? Okay. Oh, all right. So then then I think the introduction will be important because um And then we'll go into some of the GenAI issues and limitations and how we can use some of the best practices when we take something uh or create a new application and product using GenAI. All right. So, let's start with what is ML Ops and why do I need it? All right. Like uh the It's It's really a set of best practices that we have done for software engineering and we want to bring it into the field of ML. But let's really think about two things when we are thinking about ML Ops. The change and the process that we need to put in place to to figure out how do we manage that change? How do we address things that are really changing around us? I like to take an example, which always helps. So, here's my example. By the way, this uh picture that you are seeing is generated by Bard. So, it was like, "Okay, can you create a nice little cartoon?" So, definitely we are seeing GenAI helping us in our day-to-day work. So, let's say that we start a new business of a ghost kitchen, you know, and we are trying we to give this awesome recipes and food that we create to multiple people. We are using ML models to optimize our inventory and the orders and everything is going well. We don't have backup orders. We don't have anything extra. Everything gets filled and things are moving fine. And then suddenly, there is a new study that kind of came up and then your super powerful grain is not as powerful as it was or there was something else that happened in some part of the world and suddenly you are thinking over a period of time that you have more inventory that is just staying there. Some of the orders are not getting full and nothing has really changed on your side, but your business is not doing as well as it used to be. So, what happened really? The fundamentally, what happens is that the world around us change, right? The data changes. Things even if we are not doing any changes to our business, the world is continuously changing and we need to adapt to those change. We need to make sure that the models and the uh software that we have written is really adapting and making sure that it's addressing the needs that we uh see around us happening. So, this is this is really what ML Ops is. So, along with everything that machine learning development needs, right? Like whether it is about labeling the data, whether it is about uh creating the best model, about handling the rare conditions, but we really cannot forget about the model development needs. We need scalability. We need extensibility. And I'm sure you would have seen this uh, slide uh even before because this is not something new, right? Like we have done software at scale for a large number of years and we know that how important it is. Can you imagine having a system which doesn't have monitoring, which doesn't have any kind of security and safety in place, which is going out to billions of users? So, to accommodate all of these requirements that are coming in, the that's when we launched TFX a few years back, right? Like TFX is really a ML ops solution for a production ML system. It is It is actually you will be surprised that it is being used by a number of Google products are and even outside of Google. Like many of the Google customers are also using TFX as a building block. So, if if you guys use Gmail today or YouTube, you would have used TFX in one of those things. So, the pipelines that we create, which can scale to billions of users and multiple models and all of them working in synchronous to be able to really respond to the query and your answers, TFX is one of the building blocks. So, what exactly is TFX? Again, this is a very simple and like I said, we are going to do a quick refresher around TFX. So, this is a small way of thinking about how a traditional ML pipeline looks like. So, the top portion is talking about the task that you normally do, uh, starting with data ingestion, validation, then you train a model and if you think that it is giving good quality, then you push it and you serve it. The orange boxes um, are the components that you get out of the box out um, from TFX that help you build these pipelines. So, it's very easy. Think of them as like LEGO boxes, right? Or LEGO blocks that you can then really put it into different combination and you can create the workflows and the pipelines that work for your particular use case. The reason there are a lot of different benefits that come with it other than just codifying your pipeline and workflow, but the two main ones that I want to highlight today is one is the metadata store. And basically the idea is when you run these pipelines, when you are executing these different stages, there is a lot of more data that is being created. There are artifacts that are created, there is the execution states that are being recorded, and you generate a lot more information that is useful for solving some of the different use cases that you're not thinking from the perspective of user, but are equally important like lineage, governance, the lost changes, how can you make sure that you're compliant with all of them. So, when you run a TFX pipeline, you do get artifacts that are stored, you get execution records, and you get a lineage across these executions which help you really figure out how your things are going. So, the versioning, making sure that when you're retraining the models, whether the quality is better or worse, like these particular artifacts and comparison between them can really help you make sure that your production system is doing well. The other thing that I want to really call out today is the custom component and the reason is that not all the standard components will always work for your own workflows, right? Like there will always be something special, something different that you want to incorporate. But having a custom component in the same way just helps you create more complicated and customized or think of it like an extension that will help you really solve the problem that you have at your hand. Um so now what about generative AI, right? Like we uh how does that work? What does it mean in the new world of generative AI? And um I I do I think we have all seen the talks in the morning. I was here for the Bard talk and we really saw that how it is solving a number of different problems. It is I would say that, you know, I've been in AI/ML for now 15 years or so and this is another shift that we are seeing with the large models, right? Like it's a very different way of solving problems and um whether it is about, you know, summarizing it, whether it's a conversational coach kind of a thing, whether it is for language or code or images, things are really, you know, changing and and it's it's really making a big shift. So it is great and we are So but here's the thing, right? Uh I was reading it somewhere that right now the conversations um or if you look at the number of searches or tutorials, about 80 to 90% of them will cover generative AI. But if you think about the applications that are being that it really using GenAI, that won't be, you know, that will be less than 3% or so, right? So why is that? Why is that? And I And the reason is because it has its limitations. It has a lot of potential, but it also comes with its challenges. And um again, right? Like I I think this is something that uh we all have talked about, that generative AI um can hallucinate and uh and the and and the reason why it does that is because it is a knowledge base and it is generating a response from the probabilistic distribution and making sure that what it is coming up is the best possible uh probability. So, it's it's basically think of it more of um linguistic expert, right? Like it has learned a lot of information, but if you give it questions around, you know, spatial stuff or things, then it may not do as well. You would have to do more stuff to make it into that domain specialist, right? I was joking with my kids I was we were trying to do some things um and it wasn't able to get those math questions right. I was like, this seems more like my English teacher rather than my math teacher. And and um so so but the thing is that that's where the generalization and understanding of languages and lot of text has happened, but to make it a domain specialist is something that we have to explicitly do. Other thing that we normally see is because of the generation of words and the possibilities of things that it can generate, you might not get the consistent answers. Now, think about it, right? Like in our example, if we were having a chatbot and it's supposed to give you recipes and every time for the same thing it creates a new recipe, I'm not sure if the person will be very happy, right? Or um the other other challenges come with like factual accuracy and attribution, right? My my mom would be very unhappy if we are putting her recipe out there and not giving her the credit that she needs. So, so the these are real problems in the real world and when we are trying to solve these problems, these are not something that we can take lightly. We have to think about it. We have to think what it means for our business and how should we really solve these things? So, I want to also give another example. This came from a research paper that was trying to do some benchmarking across how the models are doing because we see new larger models coming up, you know, very often. There's okay, 13 billion, now maybe 175 billion. But if you look at this example, you will see that the smallest one, yes, it gave you some answer. Maybe it is not very informative. The middle one seems more accurate, but the ones on the right hand side are actually not correct. And and the last one in fact is something that you don't want it to come because it's just kind of mimicking a very, you know, some kind of a superstition that that was present. So, we need to really understand if a bigger larger model is the answer or what does that mean? But these are the challenges that we have to think about. Um Okay, yeah. So, this is what happens to our little ghost if they are using GenAI and none of these things are being uh considered or solved for. So, the other thing is that I cannot like say it more, but how expensive they are. And especially even if you're using the APIs, even if you are using, you know, open source models, there is still a lot of um investment that goes into the uh generative AIs. You cannot train the models all over again for small things, right? Like that's just it's it will take years and it will take millions of dollars and that's not how we do it. Okay, so what do we do about it? This is where I think LM Ops come into the picture and it is very important to think about it as you are starting to build uh those solutions and products and use cases for your users. Um I've simplified it a lot. Okay, but first I wanted to kind of say that these um requirements or principles that we discussed still goes for GenAI um even more than what they were for machine learning. So, what is different? Um I've I've kind of done it in three uh slides or three portions. One is the first one is about training. The training the way we used to think about in a linear way from data to model has definitely changed and we need to think of it as a multi-stage multiple pipelines, multiple workflows coming together. The way I think about it is think you know, think of like how we do develop microservices, small small things so that we can manage them better, we can version them better, we can make sure that we can just change one of them without changing the whole thing. Similar to that, like in this case, like every phase that was there, like training the model, can be broken into multiples different smaller pipelines. Evaluation itself is a very complicated pipeline. It can be with the auto raters, it can be with the human raters. You might want to run multiple evaluation at the same time to see how you are getting things. Based on the use case, like I said, if it is about solving a factual problem, you will have a different evaluation. If you are trying to solve a more creative use case, the evaluation will be very different. Um one of the example is quality, right? Like what does good even mean, right? Um if if they're generating a joke, right? Uh when my husband tells me a joke, I might find it funny, but definitely not my kids, right? Like they're like, "This is the worst joke I've ever heard." So so for something like this, like it's very difficult to get to a place which is very definitive and we can say that these are good answers or these are that the model is ready to go into the field. So how do you do that? How do you bring these different perspective in one place, compare them, and then still make a decision that it is ready for solving this use case. So thinking of it in smaller pipelines, smaller phases, and then putting it together really helps. And this is where um TFX comes into the being, right? Like it helps you put these LEGO blocks uh together in different ways. In fact, we do have um TFX pipelines for Bard uh launches as well. The second thing to think about is a little bit on the serving side. And over here, this is an example uh of a constitutional change, but I think I don't know if you guys attended the earlier talk, that when you take something into production, you will realize that there are so many different use cases that comes into picture, right? It's In this whole picture, if you think about like when the user query comes in, you, you know, adapt it with the embedding and uh more prompt and then you send it to the adapted model. The model might give you a response which is pretty decent, but you still want to make sure from the constitutional laws like is it illegal? Is it harmful? Is it, you know, biased? And you might want to use another model to get a score and then see how it is doing across these different dimensions that are important to you. And then give the response back. But one of the things that I want to highlight in this particular um picture is actually two things. One is that the model is really a portion of just the query to response system. It's not the only thing, right? There are a lot of other critical pieces that you have to think uh ahead of time and make sure that you are building it in a way that it is easy for you to version and then change each of those blocks um as needed, right? There will be right now there are three, there might be four tomorrow, there might be even more. And this is some of the learnings that we did when we were actually going and doing the launches is that the amount of things that you have to consider every time, it's just increases. The third and not the last but but the the most important thing, at least in my and this is this is my uh view, maybe different people will have different thing, is the artifacts and making sure what all are we storing. Now, we discussed it that those artifacts were important because of um the lineage and governance. Over here, I think given how the field is evolving and there are so many unknowns and we're still trying to figure out what works or not. Storing some of this information becomes even more critical because it helps you even in the developer journey. It helps you debug things. It really when you're talking about these comparisons, side-by-side views, thinking where the pipelines are taking longer, where you can expedite, where the things are not really needed. All of this information is extremely critical. So, the way I look at it is that not just from the terms of lineage or governance, which you have to do anyways, you have to make sure that you're taking uh user privacy, uh your AI is responsible. The laws change, but how do you always comply to them? Those are all good things, but from the user perspective, even if we were to really try to understand what is going on, how is it learning, how is it improving, we need to have this data. So, that's pretty much the talk where LMOps is just a superset of MLOps. Uh you have to consider all the things you did over there, but little bit more and more from the very beginning as you start to put those solutions in place. So, I know I It was a little bit high-level, but there are more talks coming up in the session, which will go deeper into some of those concepts. But for me, like really this is this is what I wanted to cover, and I have included more um stuff where you can get more information, which it'll be hard to cover in 30 minutes. But um so, there are certain machine learning courses, uh there is, you know, TFX add-ons. A lot of community contributed tools and libraries are available, so please do check them out. We do have, you know, all these forums in community. So, you can always reach out for asking the questions or if there is anything else that we can help with. And there is also a GenAI learning path that has come up. So, thank you.