DevTools for Language Models: Unlocking the Future of AI-Driven Applications // Diego Oppenheimer

Key Takeaways

foreign [Music] great well hey everyone thanks for being here today super excited about about chatting uh today so I'm Jacob Oppenheimer I'm a managing partner at factory we're a venture fund that's specialized in AI Investments I also happen to be like help a couple of different llm based startups uh with product and so I've been pretty deep in in the ecosystem um one of the things that are really interesting is that like this tremendous popularity this kind of Cambrian explosion of use cases and interpretations of like how we can actually use uh you know build AI applications I think um really the kind of quote-unquote chat gbt revolution has become fascinating in terms of like the inspiration that it's given to developers hackers uh and uh you know the people who have uh you know been inspired to go build uh you know for truly what we can start thinking about like you know as the you know software's 2.0 or 3.0 if you want to call it in terms of you know how it gets powered by AI so in today's talk I'm going to really kind of do a little bit of a survey of the environment and kind of the tools that are out there um I'm not going to be I can't go in depth there's so much going on uh but I kind of want to like talk high level about you know how people are building these applications and what kind of tooling they're using let's figure it out great so let's start a little bit into you know uh because I think a lot of people still have a little bit of this question in terms of foundational models versus large language models and so the first thing I want to do is you know these foundational models are trained on like extremely large amounts of data um and they're you know if we think about kind of like foundational models as these broad general purpose models um that are really aimed to capture you know a lot of capabilities and knowledge um the kind of like large part comes usually uh with for because they're a billion plus parameters um and you know they're really kind of kind of design and architected so that you can be adopted and fine-tuned uh so that you know to specific data so you know examples like gpt4 and clip and Dali fall in this um in for you know so foundational models really think of it as the kind of like generic or superset and inside that you know there is obviously language and there's a image and there's multi-model models so large language models are really designed for specifically understanding of language so uh and you can see their births and their llamas and gpts in this category they're really trained on massive data sets and you know they're you know you can see the tasks that exist in terms of classification generalization summarization and more so I'm going to talk I'm going to go back between foundational models and large language models a little bit but this can kind of give you that framing of you know how to think about about them so let's go a little bit into um you know kind of a history so in terms of like you know how do we actually get here uh from a research perspective right so you know back in the day started with the bing bang very sad no models emptiness nothing you know we the first kind of iteration or not first but kind of like big push in terms of uh you know machine learning was in self-supervision we started seeing models that had you know 125 to 350 million parameters um you know it's capabilities where uh to be able to kind of uh replicate uh what it had seen and kind of like spit it out uh and the kind of like data sets that were actually used to kind of generate these were really around you know kind of like small web you know so small web components uh you know or book dumps um the trend ended up going to bigger models right and is when we started actually getting into the kind of like large models which are you know getting to that you know how do we get to that 100 billion parameters these you know the the more you know a lot of proof showed that like the bigger the models the more the capabilities that were coming out of it so we started seeing these one to 100 billion parameters uh really tax taskless text generation was really kind of like the first kind of um uh things that came out of this and you know the data was really kind of trained on like all the web so now we're getting to these instruction tuned uh you know uh and uh you know and massive uh uh models that they can actually follow tasks and instructions they are using actually heavily curated data sets um they're getting ginormous right the 10 to 200 billion parameters they can generalize to tasks they also can listen to feedback in the sense that you can actually work with these models in terms of instructing them to get to results and providing context we're going to talk about some of the tools that allow you to do that and the more important part is from a data perspective these are heavily curated data sets um and label data at web scale with human feedback and the cost and the scale at which this is being done is quite impressive and and you know really kind of like the the the newest models have been there's a lot of work going on into actually kind of being able to do that so as the size and data quality increases you get more generalized um you know uh of the uh more generalization um and in context behaviors but on a much much higher cost and so we'll talk a little bit about that what that looks like as well cool so you know one of the you know interesting things about early stage development of of platforms is you um you know when the first kind of kind of platform show up people start uh you know with a very kind of like basic uh rappers around that and there's actually um a parallel to this in in general software and Technology right so when the first uh you know microprocessors came out there was essentially rappers around single board computers when the first operating systems came out um the first applications were really like rappers around utilities if you think about kind of like Unix utilities and kind of like uh you know like Norton as one of the first kind of like applications was really just OS capabilities wrapped around uh you know wrappers around that um in the internet we got these Like Rappers around like Unix and network utilities and what we're seeing today in kind of like the generative AI world is really this Like Rappers around well not just llms but also got around foundational models in general so we're in this kind of like first phase which is pretty natural um where these thin wrappers around these foundational models and there's an explosion around of it and the core thing that's actually happened right is the capabilities got to that holy moment right where we're looking at you know it really feels like magic and so my personal experience I've been working with these foundational models in a bunch of different contexts you know I like the best translation I have to it is it feels like you're Neo in The Matrix right like you're just accessing content you're being able to do things at a speed in a productivity level and so that holy moment of using Chachi BT for the first time you know has really driven to you know tons of Discord servers and hackers and Market maps are being made in an unbelievable amount of people are building uh you know what is today the basic kind of rappers around these foundational models uh but that is kind of like if we if a history provides true this is kind of the start of a whole industry that will be building deeper and deeper applications as we as the models get better as our tools get better and as we understand how to build these applications in a better way so the ecosystem is completely thriving and you know I cheated here a little bit and uh you know I stole this from uh from the folks over at at first Mark so I want to give them credit but like the data landscape and the AI and mlasky this is just growing and what's actually happened especially in that kind of mlops category is we're seeing actually now llm specific tools right that are just coming you know that are going everything from the most basic convenience wrappers around open AI apis to actually complex tools for orchestrating and multi-step reasoning it's a very simple databases for prompt templates and so what we're going to see here is that as developers continue to experiment with llms a thriving ecosystem is going to be cropping up to support that work and these tools are designed to enable developers to iterate quickly build amazing features on top of these llms uh around that and so we're going to dive here a little bit into kind of like a couple categories um of of those two tools that are emerging so you kind of understand like you know what we're seeing so first of all I think let's start with what does it take to build a llm based application and to give a little bit of that workflow so we just generally understand um is today fairly easy grab an OPI around an llm you know plug in your experimentation prompt tooling you know potentially if you need it and I'll explain when when you don't need it Vector database and data integration and you gotta be one product and to be clear like most of the stuff that we're seeing today all right and most of those explosions really around these four first boxes which is we're getting to that first version of the product and and we're still in that kind of like holy moment around that first version of the product so you know as we continue experimenting with these llms then this uh you know kind of like the thriving ecosystem kind of keeps going up you know we're going to see these steps go through the first is experimentation and prompting we go into knowledge retrieval and fine-tuning and experiment Haitian phase you know developers are really tinkering with the prompts um these llms have a really interesting API which is natural language and it has some specific ways of working with you know with that with that API API um that require uh certain tooling or at least not required but like we want an abstraction on some tooling to make it a little bit easier um and this might involve actually chaining prompts uh you know through that and the next is knowledge retrieval which involves providing like kind of like relevant context to the model so it can actually improve its accuracy so these generalized models if we can actually reduce and provide context into them um you know allow you to uh you know improve that accuracy and also it behaves better and run cheaper uh from that perspective and so finally fine-tuning is really where you know when we're going to go in like into the second version where we can actually get really really what I call kind of like snipe data sets so highly curated specific data sets to show examples to these models so that you can actually fine-tune on them to improve your model accuracy and actually reduce your inference latency uh which really what matters when you're actually thinking about production use cases so let's um jump in a little bit into what these uh you know what these uh you know kind of places are so as I mentioned like we're kind of you know version one is really where all the action is today uh and obviously there's a ton of people working on much much deeper stuff so I don't want to like take that away but like if you go look at and I was looking at this uh you know kind of last night uh one of my favorite newsletters uh no relationship to them but like Ben's bites like you know aggregator of a bunch of uh links and new tooling right so uh you know hundreds and hundreds of applications like a week are popping up essentially uh building in this kind of like V1 world uh which are really rappers around this mlms so you know you know for effect purposes you know that that moment where we that blows our mind like getting to the next step is actually doesn't blow our mind as much even though there's a lot of complexity getting into that V2 uh of um uh you know that version two of applications so let's start jumping into uh you know some of the actual like tooling here so the first one is really what I would call kind of this like you know experimentation and prompting so to get actual desired output uh you know from llm developers often need to experiment with different prompts and chains of prompts and this can actually be quite complex and time consuming so fortunately what we have right now is kind of emerging tool sets like land chain and Lama index that have helped to jump start and manage this experimentation process connecting to you know they're because these apis are natural language you know that that mastering requires that you know experimentation between single and chain prompts and so these tools have really come out to like help us connect the data sources right provide contacts and grab a bunch of data provide indices to actually be able to run through that you know you know through that data coordinate chains of calls and provide other uh core abstractions like agents that'll allow you to build kind of the applications of the future and so this is really again like the the core of this tool set is around being able to very quickly and iterate through that experimentation uh and orchestration process uh and Abstract the way some of those uh you know things and you know some of these tools have gotten extremely popular um and if you just Track by kind of like the the GitHub Stars which you know it doesn't really mean how much they're being implemented just really shows interest um it's really fascinating to see how this they've inspired you know I think it's fair to say that they've inspired a whole revolution of hackers to go build applications off of these uh large language models so then let's talk about knowledge retrieval and you know kind of vector databases so these um you know I one of my favorite descriptions around uh you know large language model is like it's like the it's the smartest goldfish you've ever met right so you know in what that means is like they don't really have memory right at this point and to be able to actually have like you know a memory or understand what you're doing over time you have to kind of provide a context and kind of like guide it through what you want and so the best way of actually guiding the context right is to actually pass it in relevant content so it can actually frame it in a way that it you know it understands uh around it and so um one of the ways of really um you know Finding relevant contacts and actually doing it in a very cheap and efficient way especially because the way embeddings work uh you know is actually using Vector databases and so there's been a you know the vector database is a very popular right now there's tons of fun being going around them but the real reason around this is that they're actually really really efficient for Vector similarity searches which is your retrieval they're really effective at storing uh billions of embeddings which is what you want to do to provide you know if you convert documents and different contexts into those embeddings to be able to feed them into these llms and they're really efficient have really efficient indexing capabilities so when you start thinking about you know retrieving similarly uh similar documents really Vector databases are this core component of giving memory and then hoping to improve uh output quality so if you're actually building an application and what you're trying to do is that to have you know as I said you know smartest goldfish um you want to provide memory to the the application and understanding over time these kind of vector databases tend to be a core component and so learning how to use them and actually Implement them uh becomes particularly important so then let's go into kind of like what these version two so we talked about the kind of like product you know the the the you know the the building and like you know the the kind of thin wrapper layer around the traditional model but that's actually not you can do today there's actually a lot of things you can do in terms of making these models more accurate um and also faster and cheaper to run and that usually kind of steps down to this fine-tuning step and to be able to do fine tuning you really need super high quality data and so I think it's really interesting and you know there's I you know you'll probably see a couple of talks on this but uh my good friend Alex who I'm sure will be uh we'll doubly clicking on this but like this is really kind of playing out in terms of the data Centric AI movement um that started you know a couple of years ago where the core quality of the data and the kind of like manicuring of data sets um is proven now to be the actual most efficient ways of you know reducing latency and increasing accuracy uh through this fine-tuning process uh for these llm based applications and so while they generalize really well um the effect of actually trying to get them like really really like can you customize or at least working in a better way is about getting data sets um that are uh you know very very curated uh and specific to task um and then providing fine-tuning uh you know apis to be able to do that and so in this process you know you can do uh you know the you know there's a couple different patterns um around that so the fine-tuning one is the one that allows you really to improve performance and and when accuracy is critical um and then what you the other cases what you do is distilling these models to smaller versions of the model that run in a cheaper way but that don't lose that accuracy uh but you just don't need a lot of the other um properties that these large models potentially have so you can think about this distillation process where I can get a smaller still large language model uh you know that has the same accuracy for those tasks but is no longer uh actually um you know you know it's much cheaper to run and much faster to run so as these use cases come up or latency matters where accuracy matters where cost and efficiency in the unit economics of running these models uh you know is an important factor you have to start thinking about these techniques uh you know to be able to kind of boil those down so finally we get into kind of like monitoring observability and testing and so the first thing is like let's be clear like the we're still working with deterministic workflows right so let's say probabilistic workflows so while most and and it's really interesting because as we look at the general population of software developers and people who are interacting with these models for a lot it's the first time that they are uh encountering probabilistic workflows so people who work in data science and machine learning like are very comfortable they understand you know that you know like that there's confidence scores and like how these models produce results uh but for the general population like you know we're used to like when I do the exact same thing you know over and over again like I should get the exact same results and that's just not necessarily true in a probabilistic world and on top of that when we're actually trying to assess performance for large language models um you know we're really have to assess quality via the user interactions right we have to understand what the user is doing what they're actually like complementing around that and so the um you know how do we measure performance is a really interesting question inside the uh you know large language model good and how do we understand good what how the good generated content is um we talk about this problem with uh you know hallucinations and overconfidence in the results but you know as we start thinking about monitoring observability and testing you know we're going to really have to think about okay why you know how do we observe the user interactions and what they do after it and what their results so being able to provide that feedback in to assess performance um we're gonna really have to double down on building a b testing around the full workflows for product analytics Because the actual result in the chain it's not just about one model and testing the model but that's to the entire workflow and they be testing that entire workflow to understand uh you know the you know if they're working or not um as these models you know larger models get larger and more and more there's a giant um uh swell around open source models so we're seeing a lot of these generalized models uh coming out how do we compare on them how do we understand just the eye test of being able to like run the same prompt against a couple of different uh you know models isn't really going to do the trick so open source efforts like how I'm over at Stanford are really providing comparison Frameworks on a task by task basis so there might be room here for actually thinking about local what does Frameworks for kind of like full workflows uh and testing those against each other and some of these tools are starting to pop up in things like honey Hive where they're really giving you like the ability to kind of test and iterate uh through through those workflows and finally you know what's the performance impact we need to be able to look at you know uh you know the the performance he directly impacts the ux so if we think about um you know how fast or slow these large language models you know the very very large ones might take a second a second and a half to produce a response if we're doing hundreds and thousands of these inferences every day not only is the cost really big but the actual cost of the ux and the experience that the user has is really affected so how do we actually measure the full user experience of the application and understand hey when should we be using a cheaper faster lower latency model because we're affecting the user experience in in that way so getting to you know finally like one of the probably the most interesting pieces uh right of working with these llms is you know they can really generate plausible but incorrect information which actually really poses a threat to what I call low affordability use cases like medical Diagnostics and final financial decisions so if we think about um what high affordability versus low affordability looks like so high affordability to be wrong right that's the framing that I use to kind of think about some of these use cases so if you think about your GitHub go pilot if you think about your suggestions in your email if you think about uh you know generating an image like these are all high affordability use cases for being wrong like you know you might it's not going to really affect anybody like you know sure you can ignore the suggestion this is why we see so many like this explosion of co-pilots type um you know workflows because they're just enhancing there's a human there verifying it like it's really about getting you to faster but once we start going into low affordability use cases things like medical diagnosis financial decisions things that where this technology should be applied to because it has the opportunity of truly revolutionizing uh you know those Industries uh the the uh you know that's how we actually can start thinking about you know okay what do we need to do to do that so we need to start Guard railing you know these uh you know these these models so how do we ensure safety accuracy and reliability um teams that Microsoft you know the people you know companies like Microsoft have full productives actually doing this um and it's really interesting because it's really kind of going to be at the core of how we explore the future of these um you know how to explore the future of using these llms so tools that start emerging that uh you know Define rules schemas and heuristics for llm outputs are going to be crucial to build trust in those systems so that we can actually start building and trusting these kind of what I call low affordability uh to be wrong scenarios so finally to kind of wrap up here I'm going to leave with um a couple of predictions um around this and so the first one is that like there's a ton of developer tools and you know usually iteration Cycles are what defines the winning developer experiences so you know if we think about what we originally started when we started working with uh you know deep learning there was a launch of you know libraries and kind of Frameworks like Cafe and tensorflow and as we iterated through it new ones emerge like pytorch that adopted quick you know uh very very quick popularity so it's really interesting to see and you know I think it's too early right now to think about okay what which one of these uh you know Frameworks that are become extremely popular today the good thing around this is I think like theater we're going to see the iteration Cycles really quickly um and these new libraries are actually enabling that iteration cycle um second one is and again this is probably not a not too crazy easy to think which is like you know if we start about what traditional amount something I spent a bunch of of time in and you know kind of like what would call previous generation of ml Ops um every single step of building an ml application was hard right so I had to build a data warehouse and capture the data I had to hire and train an ml expert label that data I had to go build a model and I had to build an inference point and then I had a version one of the product that level of difficulty the way I compare it to you is like originally you know before AWS and before cloud like you know just to build a web application you have to go stand up servers the whole year there's a lot of manual stuff to go do this is kind of out of the box now I can start right it's extremely easy to use Alm API it's literally interacting with it in English or any language right prompt design is actually fairly straightforward then you can get to complexity but it's actually really easy and then you know obviously there's complexity around dealing with the orchestration and the data integration but you know we can go very very quickly to these very one products so I think we're going to continue to see even deeper explosion of these really interesting version one products as these large models uh become bigger and have more capabilities and they're going to really continue to inspire this revolution around this future of software and I'm wrapping up here Dimensions it's the last one I promise it's all good man keep it going it's awesome yeah the final the final prediction and I stole this from Alex so I'll give him credit you know I think we're gonna see this uh you know gptu and what this means is like hi you know the the you know the the this destination of of Open Source models that are going to be highly highly contextualized to your organization so the ability to grab these uh you know models that are based you know trained and fine-tuned on your data on your organization's data that are specific to your business I think are really going to explode and what we're going to see is I'm a lobster tooling that is going to help achieve that right um and and that the core principle here is really is that you know data is the most durable mode um the last mile is always where the value is generated in these workflows and so at the core of what we're going to be seeing here is okay what is the next version of these that are really going to help you tune and specialize these these uh you know these these large language models specifically to your organization's data to your personal data um and and we'll and do that so that's kind of like the concept of that gptu so this has been super fun uh hopefully it was uh useful um happy to chat with anybody later but uh thanks Demetrius dude killer and since I ended up burning a lot of time at the beginning we were supposed to do a whole q a session for you right now but if anyone wants to talk with Diego and keep the conversation going ask him any questions feel free to jump into the slack link that I added into the chat and that's the ml Ops Community slack you can at Diego Oppenheimer and ask him questions we set up a conference channel that you can hit up and that is uh that's that I have one question though for you Diego and it's not about llm Ops if we're going to call it that it is more about you've been in the game for a while I mean you've been doing this I'm not going to say you're an old hat but you you've been doing it for a while right what's different about this time around do you feel like there's actually some legs here or is it are we gonna find ourselves in another winter soon so I I think no I'm I'm I'm so bullish about what's going on right now like I think like we are I was actually thinking about this last night and I was like I can't imagine a more exciting time to be involved in kind of uh you know ML and right now like the It Feels Like You know the for some people it's almost like dizzying like just keeping up but like the level of innovation that is happening at the speed I don't think we've ever seen it in the history of software uh and it is so exciting and an entire generation of developers and hackers and kind of like software artists are being inspired to go build and they're just going to build and people are just building and like the speed at which they're building is just awesome like I I I I legitimately I could not be more excited I think this is uh we will look back in history as this being a crucial time uh in in our existence as Humanity oh dude killer this is I'm excited I'm very excited and you just talking to you listening to you makes me even more excited some people were commenting on your visuals and I think it's worth noting that I think you made all these visuals with every kind of yeah I I tried to be almost fully powered for the um for the entire car talk so I I generated the outline I created the speaker notes I generated the visuals and I thought all using four or five different foundational models from uh different sources good job