Deploying an application with Generative AI best practices

Key Takeaways

hey everyone I'm usually a pretty casual uh presenter and so I'll probably speak directly to a lot of folks uh it's not as interactive as I like to have it which is like a lot of questions but uh love to at least I might ask a few questions for people to raise hands to see if they experienced something uh so I will talk a little bit about how to deploy a production grade application in generative AI which is a pretty hot topic I think a lot of people want to know how can this be done uh a little bit about myself uh I have a lot of company names but most importantly I've been working a lot on ML platforms uh I've been working on it across three companies um uh Airbnb I was the product lead for their ml platform uh their ml infrastructure as well as their applied ml team and started their first labeling team at meta I was a PM manager I led a team of PMS for their ml infrastructure uh and a PL um an AI platform team and then now I founded a company uh with a few folks from meta and we're trying to build the next generation of what is a generative AI platform and so last mile AI is an AI developer platform for engineering teams what does that really mean right there's a lot of buzzwords there everyone's building some type of platform and so I'll go into that a little bit more uh it is developer tools for software Engineers not just ml researchers to productionize ji applications um one of the key things that we've noticed building these platforms over again is we build this so often for ML engineers and researchers what does the platform look like for developers we strongly believe that the next generation of people who build these things will be developers it's not meant to just be for ML engineers and researchers but broadly applicable for all developers and for developers what are their familiar developer tools uh this is last mile problem often times people do ask us like last mile AI was that Logistics was it like Amazon uh but essentially the last why we call this last Model is because we saw that Foundation models were becoming popular and there was a last mile problem from Foundation model to get it into a production application uh we view this as The Last Mile problem and there's multiple steps to it uh for those who are interested I'll quickly highlight it um maybe as I go through it I'll pause and if people are interested in hearing more I can elaborate more uh if not I'll just go through quickly but it starts with prompt engineering uh everybody's probably done this if they're using gbd4 or any other uh model then it goes into parameter tuning uh your ad hoc and offline evaluation currently ad hoc evaluation is much more popular what is that it's kind of when you do the eyeball test you run it against you bd4 you're like does that answer look right or not batch evaluation or offline evaluation is you have thousand maybe 10,000 different data points you pass it through you have some type of valuation metric does it actually pass the test or not afterwards most people figure out that the models are great but I need to add my own data in that is data preparation uh from there you can do two different styles one is retrieval augmentation I think Jerry talked before you went into a lot about retrieval augmentation there's also fine-tuning where you can actually fine-tune your your models afterwards model deployment how do I actually get this into production integrate it and then three different style approaches you can have all of them there which is a rule based engine so guard rails how do I make sure the output that comes out from this model is something that I expect and how do I prevent that Behavior Uh second is caching it's expensive uh how many people have run gp4 with high traffic how many people got scared by the bill it's pretty expensive right and so overall caching is is one thing that's being more popular and then audit logs uh this is AI this is ml it's a ongoing journey and so you have these audit logs and you could either review it manually label it and put it back into the system to make your models better so last mile AI a little bit about us uh we're a developer platform for software Engineers to leverage gvi reliably uh we are Developer first this is really really key for us which is how do we build tooling and sdks uh for software Engineers to enable them to integrate genbi into their production applications we are Enterprise gr uh we want to make sure that the biggest hurdle to get into production is is it good enough and is it good enough means you have need the Enterprise gr features which is security how do I make sure I have Source control all the things that you need to get into production and you feel comfortable bringing it to your business and inflexible and customizable I showed I don't know was it 12 13 boxes there as people are looking through it they're like well I kind of have this solved maybe I have some of the observability pieces I don't need all these pieces together we understand that and so we try to make it highly flexible and customizable so you can integrate into your own systems so a little about Last Mile AI you know mapping out from that developer workflow to what we have it starts with developers then you go into your AI workbooks it is Jupiter inspired it's like a notebook inspired environment where you can use generative AI That's where you do your model exploration your parameter toing your prompt engineering afterwards you do your model evaluation we have something called AI workflows batch compute you're able to run it in batch you're able to see the results and you can iterate those are both powered by something called AI config that is the data format that you can use with an SDK to actually deploy this where you need it uh for deployment you can deploy it to wherever you'd like uh we're not really opinion it out we want to make sure that you have the flexibility to deploy this in your own environments deploy it where you need it and all of this can be backed by GitHub or any other um source code repository so how does this map to developers because you know I've talked we're developers devel developers but in the end I've brought all the ml pieces into it it actually goes really well with build test and deploy it just happens that we use all the ml jargon when we talk about it but it is build which is you have a notebook like environment or code environment your IDE uh then you have your libraries that you work with it test which is either I run it locally or I run it in some batch compute to test it out and there's some flavors of the ml kick in and deploy uh often times people say inference uh when you're looking to deploy models deploy is have this binary run somewhere it's a combination of code data I want this to run and so we map this directly to the developer workflow because that's what we're designing it for so we'll go into each of the stages we'll start with uh build with Last Mile AI config uh the first thing that we've decided to do was uh one of the most important things that we feel is how do you actually Source control this um when you're putting anything into production you often Want To Source control it and the key thing is there's a lot of data that's being put into to these gener of AI applications part of it is the prompts the parameters what model is going to route to how do you actually Source control all of this reliably and easily have the debuggability and observability and so we created a library where we've parsed out part of the core ml logic put that into a config file Json object and we have the library to execute it um and so now prompts are act as config files you can Source control this quite easily whichever you need this also means you can swap models very easily which means if you find that gp4 is to expensive you can swap it to GPT 3.5 it means that you can add more Logic for routing between models because maybe you use AWS and AWS has um bedrock and Claw too maybe you want to use the Palm models maybe you want to use open eii you can actually swap mix and match different of the models so that's where the model agnosticism comes in uh custom middlewares the generative AI open- Source Community is vibrant I don't know how many people have been keeping up with all libraries coming out there are a lot of cool things out there and having an extens ability to integrate into some of these things that are really powerful make the flexibility there so you could customize your own solution and we are Developer friendly we're focused on node Python and we have clis regularly available so with this config file uh one of the key things that we realize is you actually need a great way a great kind of IDE to iterate on this uh so we support something called AI workbooks it's a customized notebook for gener of AI it supports multimodality uh you can chain different gener VII applications you can easily configure model input parameters uh you can use a wide selection of different models gbd4 Dolly 3 palm and you can share and reuse these workbooks okay I think we're going to show the demo perfect amazing so I'll talk through it uh I'll have to talk quickly because the demo I designed for pauses but I'm going to try to get through all of it this is a multi- LM consistency pause amazing so this is perfect it's going to pause soon amazing so this is a multi- LM consistency workbook uh what does this actually mean part of llms you get inconsistent results so one of the strategies released in white paper was a self-consistency uh workflow where you can actually prompt the single model multiple times and you aggregate the results in qu and say what's the best result this is an example where we actually did with three different models uh we ran in against GPT 4 GPT 3.5 as well as the pal models to get three different R rationale and reasoning results and then we aggregated together through an evaluator and a a quorum decider and so the question in the paper in the published white paper was um deciding the age and it's a little bit of a tricky question so the question was um you're six years old and then you your sister's half your age uh which means that she's threeyear olds you're 70 now what's her age right and so it's a little bit tricky because generally LMS at the beginning were like well it's half and so I'm going to divide by two age doesn't work like that if you're a threee difference then it'll continue to be like that uh so this is one of the examples that we've built uh if we roll it amazing uh and so we run against three different models uh for this chat GPT I choose GPT 3.5 wide selection of different models dolly3 stable diffusion it is multimodal uh then I run it against Palm I do F shot prompting so I put some examples to try to infuse the logic into it and then the last part is the majority evaluator where I can run against the three different models and I can say which is the consistent answer across it so it's implementing a little bit what's in that paper I download the AI config file which is back the logic backing it in this IPython notebook in collab I'm able to run the SDK and from the SDK I read the Json file and that Json file is able to execute that exact same logic and so now you're in the developer flow where from the data that provides it and models everything behind it you can actually execute it I click into the Json object you can see it specifies GPT 4 3.5 as well as the Palm models it has the parameters which are pass through and it has the logic that's encoded in uh we have handlebar syntax for parameters and so you can actually do more complex logic in between we are multimodal and so one thing I wanted to show in the workbook was stabil diffusion 1.5 I did not record it with Dolly 3 we do support Dolly 3 uh but for this example uh you can see uh we create some images with stable diffusion 1.5 for like a western cowboy style ask system amazing uh there we go so multimodal uh AI config backs it you can actually integrate it into your production applications using the SDK but would you feel comfortable doing this probably not so that's when you get to testing with Last Mile AI how do you test these you can do your ad hoc test where you run it once clearly within that IDE you can execute it once you can eyeball test is this a right answer how do you actually run it in batch uh we support something called ai workflows ai workflows you can actually test and evaluate your gener AI application at scale which means you can send in a uh multiple rows of results pass it in execute it and get the results back and you can evaluate whether your gener application works for your use case or not uh we provide that to be scheduled uh you can scale it for your organizations and we do management capabilities on top so you can pause execute or kill different uh workflow executions this is another demo video perfect so uh pause amazing nice uh so uh the last sell that I've introduced at the end was uh you are GPT 4 I want you to respond with yes or no uh and does the proposed answer kind of match the actual answer so it's the exact same workbook that you've seen below but I've added one extra step which was I want you to compare the ground true set that I'm about to pass in with the data that's being already generated and I want you to do just a yes or no response on whether it's correct or incorrect roll amazing okay and then pause perfect uh and so two different styles uh one is pass in Json and then you can also pass in a CSV uh and then from here you can actually limit to we have a limit right now because it's in beta a thousand rows but you can actually the uh run against any number of rows that you'd like roll amazing and you can copy over the headers oh pause it's might be hard to see uh but in this demo we only do it with three rows uh we did decide to get a couple of Middle School tricky math questions uh which is kind of what the original paper had and so uh the second question we added was I'm an odd number if you subtract 10 from me uh you get 15 what number am I and the second one is a family has five Sons each of them has a sister how many kids does a family have in total for us it should be trivial which is like it doesn't mean that each son has a equivalent daughter it means that one sister leads to all of the Suns you know it's a little bit tricky it's something to fun to test against uh gener models and kind of evaluate how it works across the board um and so one thing you can see is uh column C within this Google sheet I added the ground truth the first one is 67 years old the second one is 25 and the third is six kids it can be uh strings which means I can actually pass it anything because I'm asking um I'm asking a model to actually evaluate whether it's correct or not uh if I had a specific function that I want to do which that does exist where I want to compare the exact value it has to be numerical with what's returned then you can specify that yourself uh roll amazing so I kicked off a batch workflow run and that workflow run will kick off our individual run for each one of these um and so there is three different runs each for the different rows I'm just refreshing here to see when it gets ticked off it only takes a few seconds to actually run I think it would took about 14 seconds for each uh there we go I go into the first one to see what the results are oh pause amazing uh so chat GPT even responded with a little bit of a elaborate answer which was the question is a bit tricky because it seems like there should be six children in total the five sons and their sister however the statement says that each of them has a sister this one uh this implies that there's only one sister not five therefore the family has a total of six children and so it's a verbose response mostly in string uh you can actually use this and it actually has some of the elaboration of why it decided this reasoning uh Palm also has the answer six and so I'm able to run this across all of the different rows that are specified all the riddles uh and then I could evaluate the results roll amazing and then you can actually see that most of the results have I think they all had yes which is the correct answer and now I can do this at scale which means if I have a ground truth set and I have different test cases to run against I can run it all uh these are the parameters where I can actually inspect what was passed in for the specific run and see what the results are amazing last but not least uh the key thing is deploy with Last Mile AI um I mentioned before that you could deploy it wherever you want and that's a key thing for us which is overall we understand that there's a diverse amount of use cases and we want to make sure that you can use this and where you need it to be uh We've open sourced the AI config Library so that you can run this where you'd like to uh for the AI workbooks you've seen the hosted version but we actually are going to release a local version as well so that you can run this yourself and then AI workflows uh if you're looking to run this on your own batch compute you can do that as well we really want to make sure that we enable developers so that they can use gener of AI there's a lot of systems out there that are focused on how do you get the ml workflows to incorporate gener of AI um we want to go from the other side we want to really focus on developers and from developers how do they get into generative AI uh and that means looking at their workflows carefully from build test to deploy what tools are they currently using are they using VSS code are they using GitHub to Source control their things what's their cicd environment and how do you incorporate that into the general generative AI workflow and so uh for the inference path uh you can go with vertex AI you can do stage maker you can do local execution you can use stream you can kind of go wherever you want with it uh we provide the library for you