Prompt Engineering for Code Generation

Key Takeaways

in this video we're going to be looking at a code generation use case uh for prompt engineering now the problem we're going to be trying to T tackle is the problem of creating a simple python function that calculates uh uh some operation between two numbers all right and the reason why we Define a very simple task is because we're not really concerned about the code that's being generated but the process to use large language models to generate that code and how that can come about leveraging those steps that we've discussed previously so task definition to generate python code that can you know produce a function that calculates that takes in two integers and and a string that represents an operation and then Returns the result of that operation okay so for our evaluation metric uh we're going to be using test cases so essentially here there are many ways that we could do this and there are many different ways that we could evaluate the uh the quality let's say of the code generated however for to simplify things what I want is just to generate code to test it against some uh pretty welldefined set of test cases where if that function or that piece of code passes those test cases that means that we can trust that you know that function will be reliable this is not absolutely perfect obviously and there are many there are many ways that we could expand on top of this but but what I want to introduce is this concept of developing code by having test cases that have you know example inputs and outputs and then testing the code generated by some large language model by running the function generated by that model against the test cases predefined all right so in this case uh I'm going to set a set of test cases where the functions called calculate so it takes in uh a number like five and three and then uh the add symbol and the output the expected output is eight and you the same thing for a few examples and here we could have many many many examples and many different conditions where we want to see the function uh work well like for example uh handling other types of potential errors and exceptions Etc however to simplify things we're just going to have the test cases to be some simple use some simple examples all right and now we generate the prompts right and we're going to have two prompts one for the system message of the model and one for the prompt that will generate the code so for the system message of the model we're going to want a python code generation engine and we're going to say you'll be fed prompts with code descriptions or half finish code and generate the appropriate python code for the problem task described all right and the prompt to generate the code will be generate this entire python function and then I start the function and I continue the completion of the function by having these uh you know HTML style btics which uh is an approach that's known as structure prompting we've looked at this a little bit when we've discussed um output indicators so this is just a very fancy type of output indicator where I exemplify the structure in which I want the response to to be and this is actually uh inspired by a very famous tweet by Riley Goodside who is known um as the first ever prompt engineer and I really like some of his tweets I like how he thinks about prompt engineering you should check him out at atg Goodside if I'm not mistaken and this was inspired by his initial tweet uh for structured generation of python code using large language models so here I'm saying one line doc string for a python function to perform my arithmetic operations then a whitespace then some code and then the return statement for that function so the only python code inside of this uh prompt is this line and we want to generate some code out of that now this is an approach that I like because uh when we know what we want we can start the process for the model to simplify the completion remember when we talked about task specification that we want the prompt to constrain the model towards fulfilling the task right so this is what I'm trying to do here so this is going to be the prompt that we have have and then for the experimentation process it's going to be very simple so I have here the setup for the uh for calling the CHP API so I I input the system message for the model I input The Prompt and I get the answer and here I'll be using GPT 4 now notice here that we're going to be doing a simplified version and and we're not going to go through the process of setting parameters at the beginning and having an entire prompt engineering experiment Pipeline with tables and so on because we just want to see how the code generation would differ in terms of how we would inspect that the output is correct and you know potentially how an iteration would look like so this is going to be a very simplified example so I generate the python code right and if I print it with markdown this is what I get here's the complete python function as requested and I have here the function that uh was returned by the model all right so it's pretty good I'd say it's it looks pretty nice however ever you notice that this is marked down right which means that we need to extract just the python code from here there are many ways to do this uh we saw in the video about structured outputs that we could use pantic in link chain they have output parcel specialized for extracting python code from um uh from the outputs of a model like GPT chpt however to simplify things we're just going to do a very simple reject extraction of that python code and when we do that this is what it looks like if so I let's run this again so I'm running this not live because this is recorded but I'm running this right now and all right perfect so we get that function and now I'm going to generate the python code I'm going to get the python code and I'm going to just print it here so that you take a look so now we got just the python code perfect and now I'm going to execute the python code so so I'm going to be using Python's EAC method to execute the code that was generated by the model now remember this is for demonstration purposes and executing untrusted code is something that you have to be very careful about because that can lead to issues because the model might generate some code that you might not want to run on your machine so usually we would do this in an isolated environment or in a sandbox environment and there are many different approaches on how to set that up however just to simplify things this is just a simple calculation I'm going to be running this code right now and when I run the code it means that now I will have access in the environment of my uh jup notebook access to this function called calculate so when I run this I can now without having specified in a traditional jupter notebook cell I can run this code and test it to see if it's working so as you can see it's working great and it's making the calculation that I wanted so I don't need this one anymore and now that I have that function executed and implemented my environment I can actually um uh do a unit test to check that the um the function passes the test cases that I defined remember this is the evaluation metric this is what I use to define whether or not this is acceptable for use or not so when I run this I get a mistake because I didn't import unit test so let's import unit test perfect so uh when I run these test when I run this test I get that it works great and it passes my tests so this would be the criteria for me to say okay this is perfect this is actually uh running the test correctly now the way to iterate and evolve on this would be to if I want to add functionality I would add tests and then prompt the model to generate that function and then test it against the test that I prepared as the evaluation metric for that functionality and then incorporate that so have a model that can you know refactor the code and add it and add more stuff and then run a set of tests make sure that they pass the tests and so on and so forth until a point where I can you know make a pi push Etc uh so obviously um there's a lot of more complexity that we can add to that and there are a couple of papers that are really interesting one is called uh Aid driven development that came out earlier this year and another one that's being uh very popular recently is Alpha codium that essentially try to look at what an AI driven development pipeline would look like and it's extremely interesting because it builds on top of these simple ideas of having models to generate code and then having this code run against tests and then having a pipeline for checking that the the the code is appropriate and so on and so forth so I definitely recommend you check those papers out and that's it for this video uh on the next video we're going to be looking at a fun demo on how to understand research paper using uh prompt engineering so see you there cheers