Promptim: an experimental prompt optimization library

Key Takeaways

today I'm excited to announce promp an experimental library for prompt optimization at its core prompton Builds on top of evaluation driven development evaluation driven development is a process we've been enabling for a while with lsmith with Langs Smith you can build data sets Define evaluation metrics and then track those metrics over time as you change parts of your prompts or part of your system this makes sure that when you make changes you're actually improving on metrics that you car about and you're not introducing any regressions we've heard people asking for prompt optimization for a little while now there's a few reasons for this first it can save time prompt engineering can take a lot of time and the idea of automatic prompt optimization offers to save some of that time second it can bring more rigor rather than blindly making changes you're letting some process do that and you're just measuring the outcome you go from working on prompts to defining data sets and building evaluation metric third it can facilitate swapping between models rather than spending time on prompt engineering which is often model specific you spend your time on defining data sets and evaluation metrics which are not model specific the core idea of promp is pretty simple you first specify a lsmith data set a lsmith prompt to optimize and then you define evaluators locally we then run the prompt over the data set or some Dev split of the data set and we measure initial metrics we then Loop over all the examples in the data set or the examples in the training part of the data set score them and then pass all of those to a meta prompt which suggests changes to the original prompt we then use this new prompt that the metap prompt produces to score the data set again or the parts in the dev split We compare the metrics of this new prompt to the metrics of the old prompt and if it's better then we keep it we repeat the cycle end times we've also optionally added in a step where you can leave human feedback this uses Lang Smith's human annotation cues to do that this is useful when you don't have good automated metrics or you want to provide more color and optimize other things than what your metrics are measuring let's take a look at it in action first I need to make sure that I have the proper setup in L Smith so here I have my application my email assistant I can see that I have a prompt defined this is the prompt that I use to triage my emails if I click into it I can see that it's kind of simple we'll work on improving this I then need to make sure that I have a data set defined this is the data set that I'll be optimizing it has about 20 different examples each example is an incoming email and then a treiz result this is a multiclass classification task let's now get started optimizing first thing I'm going to do is PIP install prompton I'm then going to make sure that I have the proper environment variables exposed I'm going to expose my Lang Smith API key so that I can connect to that an anthropic API key that's what the meta prompt will be using and then an open AI API key that's what my prompt the one I'm trying to optimize will be using I'm going to pause the video and do that so you don't see my secrets I'm now going to create the task a task is just a representation of something to optimize so I'm going to put it in a directory called email opt it's going to prompt me for a few things it's going to prompt me for a name I'm going to call that email opt as well it's going to prompt me for the identifier of the initial prompt this is just email triage it's now going to prompt me for the name of an existing data set or URL of the public data set this is just Harrison email assistant fot a description for a task classifying Harrison's emails and then it prints out a bunch of stuff notice that it reminds me to Implement my custom evaluators in email opt task. let's go into email opt and see what's there I can see that there's two things if we look at the config.js we can see there's a few things there's the name of the task that I'm optimizing there's the data set that I'm going to use there's the description that I typed in there's a path to the evaluators I'll need to change this there's some evaluator descriptions I need to change this then there's a config for the optimizer so I'm going to be using CLA Sonet then there's this initial prompt email triage and there's a model config this will use this model if it's not provided as part of Lang Smith's prompt tub and then that's basically it let's now Define some evaluators in task. piy I can see here a nicely populated template for me to Define some evaluators I've got this example function and then even some comments that contain the schema of the data that I'm working with so I want to Define a simple accuracy evaluator so I'm going to change this to accuracy and then I'm going to change this to accuracy down here this is a list of all the metrics that I want to calculate in this case there's just one I can now look inside and what I really want to do is I want to calculate the score as just accuracy so the predicted thing will actually just be outputs reference that's what I call it in my output schema and then I want to compare that to the reference outputs which contain this triage key so let's do int predicted equals equals reference outputs triage I'm going to update my key to be accuracy this is the name stage score I'm going to get rid of this comment this comment can be useful to pass in things to help the meta prompt but I'm just going to remove it for now and I'm going to save it this is my evaluator that I've defined I'm now going to go in and modify the config.js that I have so this is what I covered earlier I've now opened it up in a text editor I can see here that I have evaluator descriptions these are descriptions of the evaluators that I'm working with the reason that this is important is these will be passed to The Meta prompt so that the prompt knows exactly how to optimize the scores that it's getting otherwise if it sees some evaluator metrics it may not know exactly what those mean or how to improve them so I'm going to change this to be accuracy and this is the name of the metric that I just defined and then I'm going to change the description of what it is as well great this is my new config now it's time to actually optimize the prompt let's now kick off the optimization Loop we can run the promp him train command and pass the config.js we can see that it prints out the original prompt and then it calculates a baseline score it gets a baseline score of about 42% it now runs the first iteration it passes back this new prompt with green and red highlighting where it changed the prompt it then prints out the new score we can see that it improved to 57% it's now going in to the second iteration it has a new prompt here but this is actually worse it decreased down to 28 so the final prompt that it uses is the that scored 57% and we can see what it looks like here we can also see what it looks like if we go to the link Smith UI if we go into our prompt now we can see that it's been updated to this new prompt that the meta prompt suggested that's it for the demo I didn't cover how you can annotate results using an annotation cue but that's a key part as well as for future things we want to push on a few different angles one we want to integrate this into the lsmith UI so you can do this all from lsmith you don't actually have to use a command line tool so we're actively working on that we're also actively working more on Dynamic usot prompting so right now we rewrite the prompt but there's other ways to optimize prompts one of the main ways is by including examples and so we really want to push on that prompting right now uses a pretty simple optimization step and so we want to work on more advanced methods and then finally we want to work on optimizing Ling graph graphs as a whole not just individual prompts thanks for watching and if you have any feedback would' love to hear it