Advancing Prompt Engineering Techniques in LLMs: Chapter 10

Key Takeaways

[Music] in this video we will explore different prompt engineering techniques and experiment with them in code and in order to do this we need a motivating use case remember what we're trying to build in this course is an LL owered application and we specifically work towards some an application like weight and bies onebot the application that allows um weights and biases users to ask different questions and get responses on how to get stuff done with weights and biases and in order to to build this application um one thing that we will need in the future is a set of questions and answers that's something that might be helpful for example when we are evaluating uh different prompt engineering techniques uh of um and and different design choices in our application so in this video we're going to generate a synthetic data set of weights and biases user questions let's dive in again we will use a Jupiter notebook because we want to interactively experiment with the apis uh we need uh to install the libraries if we haven't done this before import them and have our open AI API key configured uh we'll also keep logging uh the results of our experimentation into weights and biases with the outo log function one new thing that we will introduce in this notebook is completion with back off as you've seen uh open AI uh API is very popular uh we can sometimes get this rate limiting error uh which uh we' like to handle more gracefully and for that reason we will add this retri decorator and whenever we uh we hit one of this uh limits the uh function will wait and then we'll try to uh make that uh request Quest again so that should make uh the experience of working U with this notebook a bit easier so let's um run the cell Let's uh start with GPT 3.5 turbo model here but we'll switch ultimately into GPT 4 um and because we're using a chat model we need our prompt to be to be done in two parts the first part is the system prompt and this is the part that should determine the behavior the qualities uh of the LLS uh it it it becomes the Persona that it tries to follow as it tries to answer our question and then the user prompt actually like defines like what we ask of the language model to do in this case we'll keep the system prompt very uh limited and will um will generate a very like we put a very simple prompt which is to generate a support question from a weight Andes user so if you go back to prompt engineering techniques this is a zero shot prompting we're not giving the model any examples we're not giving it any context we're just asking it to do some work generate a support question and we'll fit that uh into the the chat API uh with back off uh because we want to be prepared for the uh the rate limiting errors and uh will generate a bunch of responses we can actually ask the model to generate several responses and doing this through the API is um better than simply repeating this call several times because it reduces the cost of generating this responses and then we'll display them and print them this time using markdown so it's a bit easier to read in a notebook so let's run this and see uh what comes out okay we can see add this one pretty quickly uh we get a bunch of questions how do I track the performance of different models over time on weights and biases how do I connect and monitor my machine learning experiments in weight and bies how do I save my project to a specific directory in weights and biases and so on and we can see um these are pretty good but these are also very generic questions and I'm not sure if they are actually similar to what real users are asking that's something that we can evaluate later on um but uh The Next Step that we can do from uh Beyond Z shot is to give the model several examples and this is the few shot example in this case so how do we do it first uh we collected several uh user questions from our Discord server and we put this into examples uh text file which is uh to The Limited so we can now read these examples and we can display a sample question so this is a real uh question one of two like a little bit above 200 queries what are the risk um of supplying proprietary data into a third party cloud provider such as weight and biases uh interesting question not something that I would expect but it reflects the the distribution uh of of user questions in the real world so I think maybe this will help uh the model to to generate like to create more diverse examples so for the F future prompt uh we will uh add a random choice of real questions into the prompt and we will share that as examples of real user queries and again we'll ask the model to use this and to generate um several examples this is again the printed version of The Prompt that we're fitting into the model now we're giving it real examples from our list and let's run the model and see what uh it comes up with okay we can now see uh the questions that the model generates uh the first one is quite similar to the previous ones uh then it asks how do I visualize the distribution of a specific column in my weights and bues table using histograms and density plots I think this one is a bit more specific might have been inspired by one of the fure examples and there's this artifact here which I actually would like to avoid uh the model responds with sure here's a support question based on these examples that's something I would should probably put into the prompt that I'd like to avoid just give me the question don't give me extra text uh and again I think this these questions are a bit more diverse I think this few short um examples are helping but we can we can go even further and um to go further and to increase the range of questions that we can ask the model uh we may uh want to give it some more context and um the type of questions that we actually want to be like should be able to respond are the type of questions where we also have some documentation available as long as that is the documentation for a specific user question we should be able to answer that question so in order to to evaluate our model we especially want to make sure that whenever documentation is available the answer is correct for that question so why don't we use documentation to also generate synthetic questions and to do this uh we provided in our um in our course repo we provided just several examples of weight and by docs we don't um we don't um provide the entire docs they are available at docs. 1db doai uh however um going through all of these documents would uh take a lot of time and a lot of uh API calls so we just I will just use a small sample of that um what you need to uh understand as you using just a small sample of of docs then this data set of questions um will also be limited to what is available in this docs so first let's process our directory and uh collect all of the markdown markdown files and we can see there are 11 documents uh in this in this folder uh then um we want to fit this documents into the prompt into the context window of our model uh so let's check how long these documents are and we're interested in the number of tokens we will use the tick token library to calculate the number of tokens per document and we can see that some of them are actually quite long we don't need that much text in our prompt and and for those uh documents we will um just extract a random chunk of a document to um uh to to to inspire the model to create some questions so let's also Define this uh random junking function and now uh we will generate a context prompt where um we ask the model to generate a support question from a weight invis user and we give a a context of fragment of weight inisis documentation and we tell the model that this question should be answerable by the provided fragment of weight inisis documentation so let's um do this let's see this uh prompt so in our case um the model picked randomly um let's see I think a fragment of code from one of our Integrations let's see if we can see which integration this is no I think it might be just a simple standard example um of um weights and biases and then it talks also about defining metric it's probably from the intro uh collab okay let's see uh let's see what uh what this uh prompt results uh in and what type of generations it can it can trigger let's give the model a couple of seconds and then see the outputs okay so uh the model asks the following questions what is the purpose of the defined metric function in weights Andes how can it be used to control the display of summary metrix um what is the purpose of get bread loss accuracy function in this weight and bis documentation fragment that's not something we're looking for this uh definitely looks like a synthetic question where um let's call it the the model or the user has seen documentation and then ask questions about it we are actually interested in natural questions in questions that a user might have without seeing the documentation however we do want to make sure that documentation can answer it so probably we have not done a good job with a prompt so how do we how do we do a better job on the prompt side