NeurIPS Hacker Cup AI: DSPy for code generation

Key Takeaways

welcome all to another NPS hicker cup AI lecture I'm Agata and I'll be your host today in this lecture I'm joined by Christa AI PhD student in Stanford welcome Christa could you introduce yourself and give us a brief intro to this session and how it will help folks with a competition yeah sure so hello everyone uh my name is Christa I'm a secondy year PhD candidate at Stanford who's been working on DSP along with Omar and many other amazing collaborators um and the highle sort of background on dspi is that it is a framework that came out of Stanford about a year and a half ago um that it's specifically uh sort of targeted around being able to build and optimize language model pipelines um so pipelines composed of modular language model calls um to yeah uh essentially perform tasks well and here of course uh we are focused on dspi for code generation so that's what we'll we'll focus on for today's talk um and Al sort of go over these four different areas today uh so this is just an overview of what we'll be talking through so first I want to touch on exactly what dspi is uh how it works I'll then go into an overview of optimizers in dspi which um are essentially for optimizing prompts or weights in your language model pipelines uh and then we'll hopefully have time uh towards the end for a tutorial so I put together a mini tutorial on uh using dspi with the hacker cup data set just for folks to get started and then hopefully we'll also have lots of time for questions at the end um but we'll we'll I think address some of them also as we go through the talk all right so I'm sure folks here are pretty familiar with this um but language model systems or pipelines that are composed of multiple modules are helping to advance U many complex tasks and um one task in particular here that we see this a lot is with code generation so you all have probably seen these papers come out uh over the past you know years and and months um that looks something like this so there are these architecture diagrams this one is from alpha codium uh which is titled from prompt engineering to flow engineering um and so instead of using a single language model call to perform uh the code generation many of these Works look at using these sort of composed pipelines of multiple calls to achieve some final result so here what we have is given uh you know the input problem or description um we can reflect on the problem uh generate possible solutions rank them generate additional tests um and then go through this Loop of iterating on the code with our tests to produce uh a a final result and I think it's no shocker nowadays that this performs quite a bit better than you know just a direct prompt to gp4 um but we see uh pretty impressive results with some of these Pipelines um another recent example is this paper agent list um which has another similar architecture um that is used for sort of longer range coding tasks with swe bench uh and again we see these great results and I think one thing that is uh you know important and interesting to mention here with these pipelines is that the logic itself and the actual architecture that you choose as a practitioner here makes a big difference in the end performance so again it's not just the model that you use um these sort of higher level architecture decisions um have a big impact and what's cool about these sorts of architectures is that um not only are they uh you know able to uh help us perform much more complex tasks uh reliably they're also highly modular in principle so we see you know these beautiful diagrams with these nice modules and how they're all composed together however uh in practice uh as you all are also probably familiar with constructing these sorts of pipelines Al often involves um quite an extensive amount of uh manual prompt engineering and this is because as we know LMS at the end of the day are pretty sensitive to the actual prompt um that is being used and you know the specific F shot examples and all of that and so what you uh start to see uh or end up with are these get repos that look something like this uh and this is uh a personal example from a pre-sp repo of mine uh where you see you know 50 plus files containing all different tweaks to your prompt or different handcrafted few shot examples and you can imagine needing to do this for every uh module or LM call in your broader pipeline to sort of coers all of them to work well together um and not only is this just like from a software engineering standpoint um not incredibly elegant it's also very time consuming um and brittle so if the model that you're using in the backend changes like the weights update or let's say you want to experiment uh with a new sort of logic uh for your pipeline you might have to go through this whole process all over again um and so yeah this is not the most effective way to go about things so you might be asking uh how can we go about building and optimizing these systems in a more systematic modular way that is exactly what dspi aims to do with the broad idea that we want to program rather than prompt language models and uh in the following slides I'll sort of break down exactly what I mean by this so rather than tweaking these sorts of prompts um which you know can achieve pretty good scores uh when we uh you know do a decent amount of manual work here we can instead write a short language model program um in Python using dspi and then allow the program to learn how to use each of these LM calls uh effectively and here we can end up with even better results than the sort of manual approach which is pretty cool to see so the way that dspi goes about this or makes this possible is with two sort of uh big picture ideas so the first is that we want to separate out the program logic so uh sort of the control flow of those nice architecture diagrams that I showed earlier from How The Language models are actually being Tau uh to perform each step in between and then we want in uh the back end to automatically optimize how these LMS are being uh taught to perform each step through prompting or fine-tuning so just to take an example let's say we wanted to build this uh multihop retrieval pipeline in dspi so just at a high level the way that this works is given our question um we might want to generate a query to help us retrieve uh you know useful passages to better answer this question so we generate our query with a first LM call um we then use this to uh to retrieve some passages and maybe we do this in two hops so uh we generate a query a second time to get additional passages that might be even more useful to answering our question or if it uh is a two-part answer and then finally uh we'll return the answer that we generate in a last LM call here where we input our question and all these uh hopefully very useful passages that we've retrieved so uh if we wanted to build this program in dspi uh this is the code we would essentially use to do so and I'll walk through exactly sort of uh the different components of this program and and how it's built so we first have this initialization function where we essentially Define the language model calls that we want to use so you can see here we have generate query module and then uh generate answer module which we'll we'll then use uh here in our forward pass uh in which we Define the actual program logic so how these different components work together to produce the final output so here you'll see this is basically just specifying the logic that I walked through earlier so um given our question um we'll initialize some uh context of passages we'll go through in two hops generate our query using the generate query module um use this to retrieve some helpful passages do this twice and then finally um generate an answer using these passages in our question uh which we then return and uh so the way that these modules are uh these LM calls are constructed um are is using two components so a module and a signature and I'll walk through what these mean so here the signature essentially defines the inputs and outputs that we expect from our LM call in a declarative way so here you can see um we're basically saying given the inputs context so the passages we've retrieved uh in question uh return a query and then this module basically defines the strategy that we're going to use to express this signature so uh I'm sure you all are familiar with the many different prompting techniques uh that exist today so you know Chain of Thought react uh Etc here we're choosing to use Chain of Thought um as a way of expressing uh this the signature and and getting our final output and then the nice thing is um all we've had to do here is really specify the highlevel uh sort of architecture decisions and logic that we want to use for our pipeline uh we haven't had to do any sort of manual prompt tuning um instead these are all translated into prompts in the back end by dspi um and so I'll walk through quickly uh sort of uh high level view of how this works and then we'll dig into some more details in a bit so the first step here is um essentially taking these modules and compiling them into basic prompts using templates so the template is essentially just a a Json template that will take this module this signature and translate it into a prompt string to use um so here this is a basic one where um given the fields context and question respond with the field query and then it says f follow the following format and you can see we've uh inserted this reasoning let's think step by step to dot dot dot string uh to implement Chain of Thought um and there are different templates uh I'll I'll mention for different types of models as well based on sort of what works best as default and then the nice thing here is that prompts can be further optimized so uh and we do this jointly along with all the other prompts in your program so given this uh basic prompt uh template here we can and maybe it's you know performing okay at the end of the day we can uh further optimize it to sort of recover or uh discover this uh this more optimal prompt that allows our program to perform even better um so here we can kind of look in uh this is a real example uh from one of our past uh papers a task that we look to optimize so here uh We've discovered this um optimized instruction string so carefully read the provided context and question your task is to formulate a concise and relevant query etc etc um and then we have these F shot examples as well that were generated for us um and you'll see this is especially nice because uh the the reasoning string here for Chain of Thought has also been generated so there's no need to do this all by hand um and this is all possible thanks to the many optimizers that DSP offers uh which tune both the prompts and weights um of programs so um I won't walk through this full list but uh there are optimizers that are targeted towards prompt optimization so optimizing uh both the instructions and few shot examples I showed um or one or the other we have also weight optimization uh and then uh my colleague delara is working on a Optimizer that actually uh co- optimizes prompts and weights uh called better together so more there uh coming very soon and if folks are interested in uh check this out more here's a a QR code that links to um some of the documentation on optimizers and I can also walk through some more of this uh later in the talk so maybe I'll pause here um just in case there are any questions that have come up on the broader sort of what is ypy broadly how does this work and then we can dive a bit more into the optimizers themselves and sort of what's going on uh under the hood uh thanks Christa I think we're good to go unless thas do you have any questions for your um no we don't have any questions on the chat and no keep going this is super interesting awesome okay thanks all so uh now uh we'll start with sort of the 10-foot view of how these optimizers work and then kind of dive into some more of the details for specific optimizers so the high level view is that as input U will uh provide dspi with uh this a training set this is generally somewhere between 50 to a few hundred examples maybe more if you're doing full uh fine-tuning the uh language model program that P that you want to optimize and then some Metric to optimize for and um I will also mention so for code generation this will likely be uh you know does the the code that is generated by this program actually compile um and run and pass these tests effectively though you can also Imagine using other metrics like um having another language model program as a judge for example uh if if you don't have access to you know a hard evaluation metric but in this case uh we we generally should uh which is nice um and so our goal here is of course um to find this um optimize program P Prime which is what our optimizers will look to do um and this will consist of optimize either prompts or weights or both depending on the optimizer that we use and here um I will mention I sort of touched on this before but um we assume that prompts are composed of two main components so the first is um this free form instruction string that sort of describes um how to go about a task um and then these F shot examples which show or demonstrate how to perform the task so now we'll start walking through some of these optimizers in a bit more detail um so I'll I'll share about the optimizers and then um after we're done with that I'll I'll jump into the tutorial so the first Optimizer I want to touch on is sort of a classic of dspi U that's been around for the longest and is uh quite powerful so um this is bootstrap F shot with random search uh which focuses specifically on optimizing those fot examples that I showed you so the sort of rough procedure here um is that we want to take our language model program our training set and our metric and we're just simply going to um randomly sample uh an example from our train and basically input this into our program and we'll run our program over this particular example and this will generate a trace of how our program is going about solving for this uh for this example and uh we'll then evaluate this output with our metric and uh let's say we got the final answer uh or you know in this case the the code that was generated was incorrect we'll throw this away as an example uh because we're assuming that the traces in valid uh didn't really solve for anything but if the uh end answer is correct or if the in code in this case was correct uh that was generated at the end of the day then we'll assume that sort of each intermediate step that the program took to generating this code was valid and we can use this full Trace um as a few shot examples for each step in our program so yeah we'll take this now as a f shot um candidate and then we can essentially bootstrap lots and lots of these examples and um evaluate sets of them uh and basically return whichever works best so uh this is basically a random search procedure that we're using here so that's how we can go about generating these F shot examples um Meo is another more recent Optimizer that we worked on um that essentially co- optimizes both instructions impr prompts as well as the few shot examples so Meo at a high level Works in three steps um we first use the same bootstrapping procedure that I just talked about um to generate the few shot examples then to generate instructions um we actually use another language model program um as a proposer and I'll walk through exactly how that works um and then finally once we have all these peot examples all these instructions we can co-optimize them um using basion learning and I'll also talk about uh this step in more detail so here for step two now that we've done uh step one of creating these few shot examples how do we uh create these instruction candidates so the key idea here is that um we want to inform our uh proposer language model of sort of the Dynamics of the task and specifically the subtask that each module that we're trying to generate an instruction for in the program is actually responsible for uh so that we can generate really nice grounded uh instructions so here what we do is is essentially bootstrap and summarize lots of information about your task in order to inform the generation of better instructions so one way that we do this is um our LM proposer program is program aware so we actually use a reflexive view of your code um to uh generate a summary um of this view um and so this we'll sort of summarize the actual control flow of your program so here it says um the program code appears to be uh designed to answer complex questions by retrieving and processing information from multiple sources uh the module self. generate query in this program is responsible for generating a search query um based on the context and question provided and you can imagine how this would be helpful uh to sort of create instructions that uh are actually useful for this self. generate query module all right so next our uh program is also data aware um so uh we can actually l through the data set itself um and generate um a summary of of how the S the different properties of the data set so here it says the data set contains trivia style questions from a wide range of topics like music film uh history and literature we'll also use some of the F shot demonstrations that we created in the previous step uh to sort of demonstrate what the specific inputs and outputs should be um to give a sense for you know what the instruction is actually respons POS for producing and then finally we also include this tip for prompting which is randomly selected uh and this helps sort of encourage uh exploration of uh what might be helpful areas of the feature space so here we have a tip you know don't be afraid to be creative or provide the llm with a Persona so with all of this uh bootstrapped and summarize information about the task uh and the specific subtask that we're trying to generate an instruction for in our pipeline uh we'll take this uh and input it into a final language model call to create um this set of uh new candidate instructions so I won't read all these out but you can get a flavor of sort of what these look like all right and then the final step step three is to jointly optimize uh with basian learning um and here the idea is that we want to find uh now that we have all of these options for instructions or F shot examples that we could use uh for each module in our program we want to find the sort of optimal set of these that we should actually use to parameterize our program with um and the reason that we chose to use beian learning here is that um it is classically very good at optimizing uh functions that are expensive to evaluate in an efficient manner um that have lots of latent variables that influence in performance uh and this is exactly sort of the setup that we have here so the way that this works um is given given our set of instructions and fot uh candidates for each module will allow our ban Optimizer to essentially choose how to parameterize um the program um and here we'll uh basically run a series of Trials where our ban Optimizer is going to choose which instructions and fot examples to use based on its prior knowledge from what it's learned in previous iterations uh and then we'll score this and use this to again update our evasion Optimizer um and then there's Al an option to run this on many batches of data which is helpful if you have a large training set um to help your Optimizer learn more efficiently so here you know we're going to run uh various trials with different sets of instructions and fot examples we'll update our basan Optimizer um and then if we're using uh the mini batching every end trials will evaluate on our full training set just to get a sense of how we're doing um and we'll sort of record this score down and we'll repeat this process Gathering more and more scores as we're trying different things out and um as our basan Optimizer is learning how to explore and exploit all these uh variables appropriately and then at the end um we'll go ahead and return the language model program um that uh scored most highly on our full training set all right so um that was a little bit about um two representative optimizers um that uh dspi offers for prompt optimization in particular um I also want to quickly touch on bootstrap fine tune which is the optimizer for optimizing weights in DSP and this is actually uh one of the the most simple uh optimizers it essentially Works in two steps the first is again uh bootstrapping these task demonstrations you'll see this is a pattern that comes up again and again as a it's a very helpful primitive um for us um so we'll bootstrap these uh these demos and then we'll actually use them um as training examples to F tune with so essentially if you remember uh this this Trace uh for when we talked through the the bootstrapping procedure um these are the inputs and outputs that are actually going to be used as training examples um for each predictor uh in in our in our program all right so I want to quickly touch on um a study that we ran uh for prompt optimization in particular just comparing some of these different uh optimizers uh to give you a sense of some of the lessons and learnings that we had from studying these and then uh with that I'll dive into the tutorial so we evaluated um some of our prompt optimizers here using Ling probe which is this language model program Benchmark uh that consists of six diverse tasks and uh so these numbers in particular are using um gbg4 as the proposer language model and then llama 3 8B as the task uh LM um and I'll walk through some of the the insights that we we had from the study so the first is that um when optimizing just instructions in the zero shot case um we see that this can help deliver gains over Baseline signatures um so those Baseline templates that I showed you earlier um and in particular we find that that that method of grounding that we described earlier where we bootstrap and summarize different information about the task um seems to help to produce uh sort of better results we also see uh across many tasks that um optimizing these bootstrapped uh demonstration as uh demonstrations as fuchsia examples is really often key to the best performance I will sneak peek for the tutorial I was not finding that this was the case for hacker cup which was uh maybe one of the first times that I've seen this honestly so I think this is interesting uh to dive into to why it didn't work as well um is it due to the model that I was using is it the task itself U I think these are all interesting questions but um in general this is something uh that I I see across many tasks uh where these fucha examples that are bootstrapped are very helpful and one interesting thing here is that it seems that the um actual optimization process is quite important so the F shot examples that you choose really matters a lot it's not just about having um you know three F shot examples in your prompt it's really the ones that you're using um so you can see the variation between um the different examples and the performance resulting from them uh there's quite a bit so something we're interested in in the future is kind of understanding what actually uh from a first principal standpoint makes a good few shot example and this could uh help us build better optimizers going forward um and then finally we see that optimizing both the instructions and uh the f shot examples using me proo is generally the most effective approach though all caveat um this you sort of need enough uh budget to do this if you have very limited budget sometimes just optimizing the F shot examples is uh more bang for your buck all right so now that we've talked through sort of broadly how these optimizers work um and uh gone through a few of at least my own insights um from having worked on this um I want to talk through some uh just tactical recommendations uh for how to go about building and optimizing uh an LM pipeline in dspi so of course you'll need to start by defining what data set you're going to be using and um your metric so how are you actually evaluating this program at the end of the day uh and this might seem you know simple and straightforward but um this is definitely an important part to actually getting this whole system um to work quite well second uh I would recommend starting with actually iterating on the the logic of the pipeline itself um and so uh I say this because the pipeline as I kind of mentioned earlier and the actual architecture that you're using can make a huge difference in performance at the end of the day oftentimes even more than the like prompt optimization itself can um and so I would start with sort of seeing what you can do here uh and getting to an architecture that feels pretty good and then kind of optimizing on top of that and part of the reason I also recommend this is um because in order to get the optimization uh to work well at least with the optimizers that we have today you'll notice all of them rely on bootstrapping F shot examples so we need to be at least successful or um have correct results for at least some of our um existing uh data set in order to have examples that then inform creating you know a better uh even more performant pipeline so if you already have something that kind of works at least a little bit well um this allows you to gather more and more uh examples uh that are currently successful all right so then um once we've created our pipeline we can move on to optimization so here I would uh recommend potentially starting off with um just a few rounds of U optimizing with bootstrap F shot with random search that first method that I described um and maybe using a small data set um of somewhere between 20 to 50 examples uh uh in to sort of get a feel for how well um this this performs on your task and if these sh shot examples help out um before you know spending a ton of budget on uh optimizing and then if this does work then I would graduate um to or at least starts to show some promise um I would graduate with to optimizing uh with more rounds so about 30 maybe um this would depend also on how many modules in your program you have um if you have more modules you'll want more rounds of optimization um and I would then start to use um Meo V2 uh to do full optimization of both the fot examples and the instructions um I'll also note if you want to do just zero shot optimization um then you can use Meo V2 um in a zero shot setting as well there's a flag for that Al righty uh so now I think might be a good time to dive into the tutorial um so I'm going to quickly pull up my vs code uh window so just bear with me as I do this uh and then we can walk through um a actual Hands-On example a so sorry my window I need to reopen it so just give me a second here um maybe I'll pause for if there any questions that have come up while I'm opening this up uh thanks Krista so I think there's no questions in the chat um Thomas was there anything that you found interesting um I'm back yeah no everything I'm I'm I'm going to ask after like you show some examples but I'm I'm curious about this optimizing and yeah later let me sorry um the vs code window okay it's up let Meen full screen sh yes here we go all right can everyone see and I might zoom in a little bit more here that's actually too much okay how does this look to folks uh online or you all who are here all right can we maybe some a tiny bit more yes let me um yes let me do that okay okay don't forget to use your full screen um folks if you if you want to see it a little better yes and then I'll also mention if folks want to follow along um this should be I don't know if it's actually on the main branch right now but um this should be on my Branch ch o um on dspi and so uh if you want to walk through as well or go and find this later um it's under the folder dspi within the dsy repo um examples coding and then the main file here is hacker cup. py Perfect all right so I will quickly walk through uh sort of like the high level code here and then we can dig into um some more of um the the me of things so uh Thomas I a lot of your code actually um from uh your starter kit uh the uh submit first solution file here just to get started with uh with running the actual code so you'll see some utils that I modified a little bit just for my own workflow um that are in here so we have a function to actually extract the code uh that was generated uh to run it uh Etc so uh feel free to check this uh utils file out yeah thank you your code came in handy I appreciate it um and then hacker cup py here is the file that contains um the majority of the DSP code and sort of uh our our main flow um for how to readin the data um and actually create this program and optimize it so that is what I will walk through you there um so our first step here is of course to load and prepare our data um so here we just load the hacker cup data set um from hugging face um I go through and Shuffle the data set um I'll also say I haven't reviewed this code uh with all the folks on this call here so maybe there are ways that uh this is a basic example but uh maybe there are ways to uh improve upon uh sort of how I'm doing this but um this should at least give folks a way to get started um yeah so I'll go through and I shuffled the data um I then format this into a data set that I can use um in dspi so this just creates um a uh set of examples here that um are formatted in way that we can read into our program so I basically just take in the program or the problem description itself so um The Prompt of sort of what we're trying to generate code uh what problem we're trying to solve um and then this sample input and output and right now I have a um data set uh for just like the sample set that contains some of the easier problems and then I have a separate one for this the full data set here um and so then I essentially construct a the training set using our um sample data set just because again it contains some of the easier problems and I figured this would be useful for bootstrapping to get more like successful F shot examples that we can then uh then uh use as demonstrations uh and then I also uh sample some of the examples from our full data set here and then for the test Set uh you can play around with the numbers here for the tutorial I've kept it really low just to 20 examples just so we can run these things quickly um and yeah that's pretty much how I'm reading in the data set um then uh to sort of configure what model we want to use here we just uh set our LM um using the syntax um right now I'm using GPT 40 mini uh I experimented with gp4 as well though honestly for this for my particular setup I didn't see a big difference between uh mini and uh just normal gbt 40 so I am mini because it's cheaper um and then here we configure our dsy settings to use this as our default LM so now for any LM call that we make going forward this will uh default to gbg 40 mini I also will mention that I configured this to uh the experimental settings because this allows us to use uh a uh different template that's an a new template that someone has um created for chat models specifically so I found that this worked better for uh gbt 40 all right um so now that we've set the language model that we want to use we'll create our evaluation function um I think the important thing to mention here is the actual metric that we have created so um if we go up we basically have this test code function um which is a wrapper around this uh metric uh function which is how you essentially create a metric in DSP and so this takes in um an example which is basically our ground truth data um and then a prediction that was uh created by our program and then we basically check to see uh if the solution code that was generated by uh our program uh runs correctly and if it does we return one um otherwise we return zero I'll mention in this particular case I'm keeping things simple so I'm only using the sample input and Sample outputs to evaluate uh but of course you might want to use the full um the full input output uh pairs that were given um but this just makes it quicker to run all right um so now that we've defined our evaluation function um which is using the test set by default here um and our test code metric um with a timeout of 5 Seconds um we can then talk through sort of how we actually set up these programs in dspi for this task so for this tutorial I set up two basic pipelines uh this first one um is very simple just like a a one stage pipeline uh to demonstrate how that works and then we have this more advanced program generate code uh which is a multi-stage pipeline so if we go up here to see uh how this simple generate code uh uh program works essentially what we do is we Define um this one module generate code um which is defined as having this signature and I wanted to include an example of how to um Define a signature in a more Hands-On way in case folks want to do this um because I I think I showed you the syntax uh before where we have you know like input Arrow output um but if you have something more complicated that you want to describe in more detail um you can also Define a signature class so here I have uh generate code signature and then um this doc string becomes the instruction that is used so you can write whatever you want here um and then we have um our inputs which we Define so the problem description expected Behavior which is essentially a string that I create using the example inputs and outputs to show what the expected behavior of the um the code should be and then here we have an output that basically describes initial solution um so a plan for how to go about solving this particular problem um and then the Python program itself um and so once we've defined the signature we can construct our module um with it in our initialization function um and then in our forward pass uh we can use this module to basically generate the code that is uh using the variables that are inputed here in our forward pass so problem description uh sample input sample output basically the variables uh that we have in our uh example that we we created for our data set all right so this then generates some python code uh which we then return um so this is pretty simple um we can go down here and actually like run this and see how it works the um spoiler is that it doesn't work very well at all um which is uh again sort of enforces the point of why it's important to have uh these pipelines really matter so let me just this going really quickly I like that you have a dspi clean environment yeah I have uh I have mini now as you can imagine at least it wasn't DSP clean V10 or something all right so this gets uh two examples out of 20 correct um so at least it wasn't zero but you know we can probably do better so this brings us to um our more advanced pipeline here that we created um called generate code which I'll go up and uh show you how this works okay so okay here we go um so I went about defining this particular program in maybe the more classic way where we have um you know these declarative signatures um here so this is pretty simple but um it expresses maybe a more uh a complex logic so here uh basically the high Lev flow of what I wanted to do is create an ensemble of um some like number of different generated code options that we start off with and then I want to go through each option um and for up to a certain number of tries basically try debugging it um until we the uh code to run on our uh input and output test cases uh that were provided so this is essentially what this uh program aims to do so we can Define um basically Max tries being the number of debugging iterations um that we uh will go through for each particular uh code option that was generated um numb ensembles is basically how many options we want to generate UPF front and then we can Define um our layers here for uh basically generating the code um and then fixing it um so here self. generate code uh we're using a Chain of Thought module um and this basically takes in the problem description um the behavior that we expect um and then we return a Python program with this um and we set n to be the number of Ensemble so whenever we call generate code this will generate basically n options um using a different temperature for each generation so they're all different um and then the second layer fix code um basically takes in the problem description again um the current code that we have um the expected Behavior again um and then what we're getting incorrect um and it returns fixed code and so then here in our forward pass we again just Define how all these things come together to produce um our final uh output so um we generate our Solutions using generate code uh we extract out all the actual python code from each and then we go through each of these options and for up to Max tries we will run the code um we'll then get um either hopefully a great result or like the error and stack Trace uh that was associated with running it um and then we go and basically like start our debugging process so maybe this is more um like cases than you actually need at the end of the day but um basically I checked to see if the code led to an exception then we'll input the stack Trace into the like what we're getting incorrect uh into that particular field um if nothing was returned I'll just say nothing was returned um to make sure that something is returned next time um if the wrong type was returned I'll say the wrong type was returned and then finally if it's some other um issue then I'll just um oh in particular if the um output got some cases wrong then I will show the mistakes um that uh that uh occurred um and use that as feedback and then otherwise uh if hopefully it's right we'll say that we found the correct solution at this P particular uh depth and breadth um of our program and return it all right can I ask something about like when you say we show and like how do you you bu this there's so many trajectories and the like the retry Loop and yeah yeah totally I mean yeah it's a good question I just put in print statements essentially uh you can also uh we can like experiment also with putting in like a a break point I'll show you how you can kind of look to see um what the actual like prompt was to your uh language model and what it returns because that could be useful for some debugging here um but yeah just print statements and break points essentially um is how I went about uh debugging this but again maybe there are more cases here than we're actually needed and I could just say it got this wrong um and and fix it without the more detailed messages cool all right um so let me actually run this and maybe I'll set a break point too at the end just to show you uh like how you can inspect the history of um your your language model call um for debugging okay I'll comment this out I come from an experiment Tracking Company so I would I would say like you should use a tool for these but no yeah that's a [Music] good yeah I know we have we have weave integration with the Spy maybe yeah nice I really find that like I I know I work for Wes but like these pipelines and when you go to retry Loops it gets super deep and like totally interactions with the llm provider and maybe like even you are using tenacity and retries on the LM provider and then like checking then running code on a separate thread and you want to see and out there and as you said like you get an exception you want to re inject that and maybe you want to see the prom formatted with the exception correctly no the debugging is essential like a huge part of the process here for sure um so any tools that help with that uh are are very exciting um all right so as you can see this is running um we've already gotten more correct um than we did in our initial run here so uh I'm guessing we're probably tracking it somewhere around like 25% um or so which again you know we haven't solved the whole thing uh so more left for everyone here to do um but you can kind of see this is a demonstration of how um creating how we can go about building these pipelines and how um you know more advanced logic like this can be uh definitely helpful helpful over a single LM call um and then yeah just quickly to show how you can kind of look at um what was actually generated here um so I can go and call inspect history and this will Bas basically show me um everything in green was generated um by my model call and this is looking at n equals 1 so this is the the very last call that was done um so it looks like this was a debugging call where I uh basically had um let's see okay yeah so here you can see this is the instruction that was given very simple um because it was the basic template uh given the Fields LA return the field fixed code um and to follow the following format and so we have our problem description um and the expected behavior um uh the current incorrect results that we're giving uh and then it says provide um the output Fields uh reasoning and then fix C so here we do our Chain of Thought reasoning um to think step byep through how to fix the code and then we actually get into the the fix code here so this can be helpful just to kind of make sure that everything is working as you roughly intend it to and kind of look a little bit more under the hood all right so that is um a basic overview about how uh you would go about building these um pipelines now to talk about optimization the unfortunate thing is I tried to optimize on this particular data set and they seem to do not much at all um so um I can't say I recommend running them um right now for like large budgets uh if you're budget conscious here I would say if you get something to work let me know because I'm particularly intrigued this is the first time I've seen like bootstrapping fot examples not actually help very much um but I will walk through kind of how you can um how you can do this um in case you want to play around some more um and I have seen this work on human eval uh but again it might just be that this data set in particular is much harder uh or you know has different Dynamics uh which I think is um is interesting so this is the code for how um you can optimize with Meo um I'm doing this zero shot so just instructions so you can kind of see the impact of um optimizing with instructions um versus um optimizing oh this should be uh with F shot um versus doing the F shot examples so here uh in our optimize with mepro uh function you can see we basically instantiate um our Optimizer uh so mepro V2 we say what we're using as our prompt model to generate the prompts um and then our task model so like what's actually performing the program uh at the end of the day um and our metric um numb candidates is basically basically how many different candidates for each instruction or fot example we want to generate um temperature this controls the prompt generation temperature so I keep this relatively High um and then this log directory uh it'll it will um place the optimized program here and some other information um that can be helpful for um just tracking experiments um and then here we will actually go through and um this is where we compile uh or actually go through the optimization steps with our program so we pass in our current program um our training set um the number of bootstrap demonstrations we want to use so I set this to zero because um in this case we are just doing instruction optimization and then uh the number of batches so this is basically how many trials we want to optimize with um and for now I'm turning off mini batching because uh our training set is relatively small already that we're using um and then I essentially save our optimized program here so you can access it later um so I can at least show you how uh the the optimization itself um will take uh you probably about like 45 minutes or so so it it takes a little while um so we don't have time up for it on the call unfortunately but I will run and it depends of course on how many uh trials you want to use but if um if you want to do like full optimization it can take longer um so we can essentially run it using oops by uncommenting this and this will just give you a sense for what it looks like and then I can show uh a program that was optimized um that we saved as well and here we just uh asked for user input uh to make sure they want to run because sometimes these can be more expensive if you're evaluating over um large training sets especially without the mini batching all right so this might be a little bit slower but essentially what this is going to do is uh the process that I described earlier so right now it will bootstrap F shot examples that are actually used as input to creating the instructions so sort of informing what the inputs and outputs for each step should look like and then we create lots of instructions that we can then try out in our program so I can show you uh what like an optimized example looks like here uh so this is okay yeah this is one that I did recently which ended up getting essentially pretty much the same score again as before so um it's unclear if this made as much of a a difference but here you can see that uh like for for this fixed code uh string we have analyzed the provided problem and the code um identify any discrepancies between the expected Behavior and the actual output or errors indicated by Uh current incorrect results then generate a correct version of the code um that accurately implements the expected Behavior Uh while addressing the identified issues so this one looks pretty good I pretty sure that the um generate code instruction here um ended up overfitting to um the one specific example um where it's talking about like uh how to go about basically one problem um which we also have seen in other tasks sometimes this actually ends up being the most performant prompt which is interesting it's almost like some form of um like f shot example in itself um but in this case it didn't really do anything to either impact their performance positively or negatively so I would say for like I guess in terms of my recommendations we have seven minutes left so maybe it's a good time to move into that unless you all have any questions or if any have come up on the chat we have only one question on the chat about style basically if you can use it for yeah having Nuance cases like writing on specific style um oh interesting um sorry I I'm not sure if I understand what's meant by style in this case uh they're talking about like writing style so if you want to like have something right like I suppose like check Speare or oh sure yeah definitely I I would say it's just like any other like task if you want your output to be Shakespeare um having some like you know instructions around uh or like examples of what Shakespeare looks like um you could also have in this case uh since evaluating whether or not something that's outputed is Shakespeare or not might be a little bit uh more difficult to do you could have another like language model program as a judge um to sort of like actually evaluate the output and say whether or not it's Shakespearean um so that could be used to uh sort of train or optimize your program um I think I think people also interested in like I want to I want to kind of chat or AI That's capable of writing as me so we could read my emails maybe and like reply myself I see capable of writing emails you said maybe I don't know like I'm thinking more of like a a real use case than writing check steer I see I see uh yeah I would say in that case it really comes down to you could certainly use gspy to like create this Pipeline and then it would probably come down to what is your metric um and so are you able to evaluate like what is a good email effectively or not um so there might be a good way to do this again using another LM or some sort of training set of good examples that you have uh but I think that would maybe be like the the bigger question there but you could certainly use DSP for this cool we don't have much time but I want to if we can dig a little bit in this optimization like when we do optimization in the sense of yeah calculus we have the function that we can compute derivatives and have a direction to move like this is more of an holistic can you right make the parallel on both yeah definitely so here we have to do this optimization in a non differentiable way because we have non-differentiable components of these like different maybe API only models that are sort of interacting with each other um via strings uh so right now we're looking at it through this sort of like broader optimization lens where we have to leverage techniques like Bean learning or other things that don't require gradients um so I yeah I think that's maybe just a a constraint that we impose because we wanted um for these types of techniques to work in this particular setting but you can also Imagine uh you know if we did have access to log probabilities or model weights uh potentially other techniques could be used and certainly other works have have looked at um doing this especially for a single model um uh leveraging those sorts of things uh like prefix tuning and whatnot um but in our setting we wanted to keep it to sort of working for these non-differentiable modules if that sort of answers your question but I agree the term optimization is is quite broad yeah thank you very much no this is this is super interesting and I I want to give it a try awesome well yeah definitely check out the um the tutorial here and thank you again Thomas for uh all the code that you wrote um that um helped get this uh started pretty easily I appreciate it cool thank you um I will let to wrap it up and see you soon cool all right well then thanks uh thanks Krista first of all for joining and giving a such a great presentation uh thanks everyone who listened in and thanks thas and yeah um we have a Discord uh for the hacker AI um conversation so if you have any further questions I want to discuss it afterwards um we're going to drop a link um in the YouTube um chat so you can join in there and yeah good luck in the competition and see you in the final session tomorrow uh tomorrow we'll be presenting rag thanks everyone everyone and thanks for having me