Metaprompting with GPT-3 to dynamically generate arguments

Key Takeaways

happy monday morning everybody um david shapiro here with another video uh let me go ahead and introduce today's topic um so i posted on the open ai forum um just say hey bring me your research and business problems and i'll see what i can do and so anna r mills is an expert in constructing arguments and asked me to take a look at doing this with gpt3 so i'll show you the data in a second but the most important thing is the instructions that she gives her i guess students um and it's it's basically teaching them how to how to construct um critical arguments against something so write a thorough summary and critical assessment of the argument the summary should describe the key ideas of the argument including the main claim uh key reasons counter arguments rebuttals and limits the assessment should discuss the strengths and weaknesses of the argument what was compelling persuasive troubling unclear or problematic choose phrases like elegant argues throughout to show that the writer's purpose to show the writer's purpose at each point write most of the essay in your own words but consider using the occasional direct quote where original word choice is critical the introductory paragraph should include the title of the argument the author's full name the argument's main claim and your overall assessment of the argument's validity okay so um this might sound like one task but this is actually many many many tasks and so let me show you some of the data that um she sent over to show you what i mean and so this is this is one reason that a lot of people think that like oh gpt 3 is really stupid because you ask it to do like all these things at once and it only does one of the things and that is because what we need to do is we need to break these things down into separate cognitive tasks so here's the instructions that she gives um and really this is a procedure right it says do this this this this is how it should look um gpt3 will be able to get some of this but let me show you an example of one of the inputs so here's one of the here's one of the essays um it is 1500 1600 characters long and then we go get the the critique look how long the critique is the critique is way longer than the um then the then the uh actual essay it's 5800 characters long okay so that's that's a lot of critique um unfortunately this is probably a little bit too long to fit in in a single prompt which is another reason um so let's see how many tokens this is oh only 1500 tokens okay never mind um so this could actually fit in original davinci however what we're doing here is is basically i'm going to break it down into prompt chaining so with all that said we are going to i've already got the repo created so let's go ahead and clone this down um and let's get clone yes i fat fingered that there we go okay so then we've got our brand new repo here critical argument yep that looks good and then we'll open another one um increasingly verbose bot copy my git ignore and open ai key okay and then increasingly verbose just go ahead and copy a script over i'll rename it paste okay so this will be um critical argument um i'll just call it critical argument so what we're going to do within this though is we're going to set up prompt chaining so the first thing we need to do is let's look at the instructions write a thorough summary and critical assessment of the argument that is the highest order instruction and then this describes the summary the summary should describe the key ideas including the main main claim blah blah okay so we'll say this is this is subordinate to um the first instruction right let's see the assessment should discuss the strengths and weaknesses of the argument [Music] what was compelling persuasive troubling clear and problematic throughout okay so the throughout is global instructions write most of the essay in your own words also global instructions let's see the introductory paragraph should include the title of the argument main claim in your overall okay okay so basically what i'm doing here is i kind of chopped up the instructions to show that it's like uh let's see that it like it's it's separate tasks um i'm i don't i don't think i'm wording this correctly but i hope that by showing you um so like there's an order of operations basically the introductory paragraph right should include this choose phrases like elegant argues throughout to show the writer's purpose actually i wonder if this should be subordinate because this first instruction is kind of the global purpose right that is what this paper is um rightmost the essay in your own words okay yeah i think that's good um and then yeah okay so the reason i'm chopping it up this way is just to show how the mind how your mind might organize this because the instructions that she gives kind of talk about this part and this part in this part but what what happens in your mind as you build it up is you're kind of constructing an image um so our brains are kind of like um object-oriented coding where um you know it's like okay so you start with this primary idea and then you attach a new feature to it or a new um or a new uh characteristic and then you modify that characteristic and then you add a new you know value or whatever or a new function to it and so you keep operating on this object in your mind which is constructing this argument now before we do that let's just do an experiment because right now i'm operating off of intuition so it's like what if i'm wrong right what if what if gpt3 can handle can handle this just sight unseen so let's go and test that real quick so write a thorough summary and critical assessment um and then argument actually here no we don't even need that just go grab it essay to critique add some more new lines okay credit uh let's see we'll say summary and critical assessment now this is da vincio 2 so um okay not bad may contain sensitive content all right so in wouldn't we all cross the border yep um anna mills argues that illegal immigration is not inherently wrong or unethical mills points out that if most people were in a situation where their children were starving and they had a chance to get to the us they would take it regardless of the legal consequences therefore mills concludes that we must find a border policy that treats migrants with empathy and respect this is a compelling argument that highlights the human side of illegal immigration now that's still really hot mills effectively uses hypothetical situations to illustrate her point that most people would do whatever it takes to get their families to safety however the argument does not address some of the more practical concerns around illegal immigration such as national security overall this is a thought-provoking piece that challenges readers to consider the issue from a different perspective i would say that is a phenomenal opening paragraph however let me show you how long this is in comparison and part of the reason that it's shorter is because we're using text davinci o2 so text davinci o2 is fine-tuned on on on the instruct series data so it's basically saying okay i'm going to put out you know one sentence a couple paragraphs sometimes you get run on stuff but with davinci o2 its output is generally very short especially compared to some other ones so this is this is what gpt3 put out and then this was what a professional wrote i don't know if anna wrote it or if um if a student wrote it but you see like okay so we've got um let's see 906 character response versus 5800 character response so we're not quite there but the quality of this response phenomenal right this this i would say like um the construction of this it is very obvious that someone is responding that this author is responding directly to this essay whereas you say like you know the the response title was contested territory okay that's not even the same title right in recent years like someone is just kind of summarizing it um but honestly uh this i don't i don't think that this is as good of a of an opening because someone is like whoever wrote this they're not actually saying that this is what anna mills claims there's just stating this is a fact and so it's not clear that this is actually a critical response just from the get-go um so that's my personal opinion i'm not an expert in this this is just i'm reading this and if this was an art like if i saw this as like a blog and someone was like you know critiquing someone else's blog i'd be like what are you who are you like contested territory okay i would search for contested territory like is that what you're responding to um the second paragraph seems seems spot-on uh mills questions the ethics of enforcing immigration laws that was you know gpt-3 picked up on that as well um while the essay argues for empathy it limits the scope of its argument to from offering comprehensive vision okay so we've got all the different components um so for the opening paragraph it seems like if we just use this whole thing that's fine but what i'm wondering is if we can break it up into sections so first let's just copy this prompt down because it worked so we'll do essay summary and critical assessment um okay so we'll save this as prompt01 so we'll use this to generate the opening paragraph let's run it again just to see if it produces something similar so i'm guilty of this sometimes like i see one good output i'm like oh it's good um this it's almost identical okay yeah so the output is darn near identical let's turn up the temperature just to see if it gets really weird um so one thing is that um the quality of the instructions you know you see like there's a whole paragraph of instructions um whoops that actually can sometimes help sometimes if the instructions are not clear or if you're trying to do too many things it'll get confused okay so let's see our anna mills um challenges the morality yeah um she appeals to the reader's empathy asking to them to imagine strengths is delight in its emotional appeal and examine examination moral implications yeah okay so we've got that um but because we so there's a few there's a few like universal rules right um so what we want to do is break it down into smaller parts because um you know i think we do have weakness oh nope it didn't address the weaknesses so here's what i typically do when i want to deconstruct something um is i'll break it down into smaller and smaller parts um and so what i'll do is let's um well here let me make sure i've got a copy of the instructions nah no okay we got the instructions here and let's close that we don't need that um here's where i deconstructed it i actually don't like that let me close that i made a mess of it don't need that okay we'll get back to the script once we do some prompt engineering um yes so uh the key ideas including the main claim key reasons counter arguments rebuttals and limits all right let me now copy all this into a new prompt and all we're gonna do is we're gonna we're gonna pare this down so we want to summarize well here i'll borrow the same language that annie is so um we'll say write write a summary of the following essay um uh while focusing on focusing on the counter arguments rebuttals and limits assess the strengths and weaknesses of the argument okay so now let's take this and we're going to um assessment of strengths and weaknesses and if it's not apparent yet what i'm doing oh here let's just grab this hey yeah we can grab the whole thing and then we'll go grab the oops i closed it didn't i go back here so by breaking it down into smaller parts i'm basically going to say okay write each section separately so let's see what this does let's see if this works it might not let's turn this back down to 0.7 and here we go so this is this is a little bit better um the essays weakness is that it does not offer any specific solutions to the problems the author admits that open borders would be dangerous but does not say how um we could regulate the border without criminalizing immigrants this lack of specificity leaves the reader wondering what exactly the author is advocating for not bad so i pointed out on the uh on the forum that gpt3 was trained a lot of reddit data so it's really good at constructing arguments it's frustratingly good let me run this again just to make sure that it's consistent in its performance let's see it also does not consider are not fleeing difficult circumstances but are instead seeking economic opportunities not bad okay um let's do a thorough assessment let's add thorough back in because i'm wondering if we can get a little bit more we can squeeze a little bit more juice out of this because if it writes one paragraph oh here we go oh much better okay yeah yeah okay all right so this will this will looks like it'll reliably generate two to three paragraphs sometimes one i really like it though so we'll update our prompt whoops where is the prompt go ahead and save this as prompt o2 because this is like kind of the second part of the paragraph oh all right hang on my dog's outside gotta let him in okay i'm back um yes so write a summary write a thorough summary so it all comes down to um right word choice to get the uh to get the desired effect um so the prompto one this will generate our introductory paragraph we remove that verbage from this one where we say what was compelling persuasive troubling and actually so it did a really good job of doing strengths and weaknesses um let's actually separate prompt o2 out into a third prompt and we'll we'll do um so like this this prompto2 is going to be strengths and weaknesses um so let's focus on the compelling persuasive troubling unclear or problematic parts of this one so if we do um thorough assessment of let's see what was compelling persuasive troubling unclear oh whoops crap crap no i'm modifying the wrong one copy that go here whoops okay sorry my brains are a little bit like scrambled eggs this morning apparently prompto3 okay so let's leave let's leave the original prompt as it is but then um here we'll just we'll just copy this and do it all caps so someone asked why do i do caps um and that is because i think i've explained it before but just in case you haven't heard um a caps lock version of a word looks different to gpt3 it uses a different set of tokens um and so gpt3 will see like okay this is like and it looks like a block of instructions or something um okay so let's just change this and let's see if it let's see how much just changing the final kind of thing you know final instruction changes the nature of the output um well that was short maybe maybe maybe my intuition here was wrong okay this is pretty similar all right i think i think my intuition was wrong i don't think pro i think prompt 2 is fine as it is it looks like it got most of it in there um so let's go back to the original um problematic choose phrases like okay word choice is critical introductory okay well i mean let's just let's just start here with prompt one and two and just kind of run it a couple times and see what's missing because my brain is not giving me clear like signals as to like okay what is the next cognitive task that we're missing here okay so i've got this um let's see i don't think we need to do chunks in this one we'll just say text equals open file input.txt so that means we need to have an actual input so we'll do essay to critique and we'll save this as input.txt all right and then result equals list um so what i what i often do when i have like um when i have prompt chaining is i'll do um i'll do a variable called like prompt files equals and then i'll do um you know prompto1.txt and then prompt 02.txt and actually there's an even easier way since i'm following the same format make sure i import os so i'll do um prompt files equals os dot lister do the current directory and then actually put this in a list comprehension so this is a fun thing i 4 i n os lister where no if if prompt in i so what this does so python has this fun thing called list comprehension where you can actually declare and and populate a list in one go and so i for i you would see you would recognize this as instantiating a for loop so it's it's um so the variable name is going to be i and the for loop is for i n os lister so it's going to list all the files in this directory and then it will include it it'll include i so it'll include the file name if prompt is in the file name so this is a fun little one liner that you can do with python so then we'll do for file in prompt files so we'll just iterate through them um and then we'll do prompt equals open file um so we'll actually just replace this with i because i is now the file name and do sa and then we'll replace because we don't have any chunks anymore so then we'll say argument equals gpth3 completion from the prompt so we've populated the prompt and then the result is we're just going to accumulate it all together right and so we'll replace this with argument argument and then once it's done we can get rid of all this we're just going to save the file and we're going to have a double new line between each one and we're going to save it as output.txt so this is pretty compact and so again what this is called is called prompt chaining where it's like sometimes you take the output of a previous one i did this very extensively with like the summarizing and novel writing one but when you're constructing um like especially non-fiction where it's just kind of like one idea after another or one kind of dissection after another it's much much more procedural than especially than fiction um because each passage or paragraph in a in a non-fiction essay particularly if it's an argument where you're skewering one essay each each paragraph can kind of stand on its own okay so let's just run this real quick what is this okay i don't need that um yeah and we'll actually i will leave that in just so that you can see that it'll it'll it'll stick three arguments together um okay so cd gpt3 critical critical argument python critical argument dot pi well that was fast something is amiss did it generate an output no hold on hold the phone i bet i did something wrong i wonder i wonder if my um i wonder if that was it um print prompt files am i not getting anything hmm i have done something wrong duplicate i know i've fat fingered the syntax somewhere so let's just say os dot lister and then we'll add an exit right after that just okay yep okay so lister works oh wait hey there we go okay just lister with nothingness with no arguments that's how you uh that's what i did i is not defined what do you mean i is not defined oh derp i instantiated it here this is i so typically here's here's what i did wrong typically um for shorthand is you'll say you'll use i in a in an iterative iterable um variable in python so it's for i an iterable thing um and then i copy that instead of switching it to file um sometimes you sometimes you'd be like for i and prompt files um that's bad practice though because if you reuse the same variable name over and over again i've run i've run afoul of that before okay figured it out that's what i did wrong so let's see what it generates no such file or directory ah darn it i forgot to create my gbt3 logs always something gpt-3 logs third or fourth time's a charm okay and then what we'll do is i'll post this back on the forum and see what she says okay so now we've got the um now we've got the output populated reload it yes okay so it's only 2 000 characters long but that's not so bad um anna mills argues that illegal immigration is not unethical um migrants with edmonton respect okay didn't use the the title in this one though um all right so i some some stuff is starting to crystallize in my head um let me make sure i get a copy of the original instructions and ideally what we would do is actually have gpt3 generate the prompts that it needs so that is going to be a big thing in the future with artificial cognition is what's called cognitive control but we'll get into that at another time um okay so let's change this we're going to write the opening okay right the um let's see ah no oh this is a tough one oh oh oh okay i know what i should do i'm gonna pause it for a second because i need to do a little bit of tinkering okay we're back um this video i realized probably seems like it's a little bit rambling and unfocused and i'm taking a risk here because this is not a problem that i've thought about a lot and i just went in cold so i'm showing you my whole process of like okay let me take kind of initial stab at it and see what happens and then we kind of pull it apart and then go at it again so what i've done is rather than me design the argument i'm asking gpt3 now to design the argument so what i've done is i've written a prompt that says read the following argument and brainstorm the outline of a critical assessment write a thorough list of the key points in the essay with some critical evaluation of each focus on constructing a thorough evaluation and deconstruction of the argument presented below um so then i said thorough brainstorm outline points um yeah so that's that you know just let's see let's take thorough outline of points um and i've ran i've run this a couple times and it's pretty consistent um there we go oh much better okay so then what we do so this is this is cognitive control so what we're doing is we're asking first for gpthree to plan its argument um let's run this one more time just to be sure um this is writing first person okay i don't like how it did that we can also increase the maximum length um let's see let's say um describe the points in your own words um yeah there we go the author argues okay all right cool so let's copy this this will be our new first prompt let's just nuke this one and i know some of you will be like ah no save the thing essay thorough outline of points okay so this will be prompt01 and then prompto2 we will keep we will keep some of the original instructions um and so then we will but but but what we will do is we will ask to focus on a specific point okay write a thorough summary and critical assessment of the argument the summary should describe the key ideas we've already got that the summary should focus on the following let's see point and so then we'll do point and so what we'll do is we'll we'll copy paste the output one of these items so she poses the question of whether or not illegal immigration is actually wrong or unethical um and so then we'll say uh let's see we'll put this in quotes that should be fine let's see your assessment of the argument um your assessment of let's say of this point of this point should and we'll go down to here um discuss the strengths and weaknesses of the argument what was compelling persuasive troubling unclear problematic choose phrases like throughout to show the writer's purpose write mostly essay in your own words but consider using the occasional quote there we go and then we can remove the introductory paragraph um actually i think we need to bring that back so this will actually be like yes okay i know i'm jumping all over the place and i promise there is method to my madness um and i'm gonna give these better better names um okay so we'll start this with essay and then we'll do we'll go back to kind of what i originally had which was um i think it was something like critical assessment and evaluation thorough okay that's probably good enough okay so this this will be prompt um intro and this one we're gonna rename to prompt brainstorm so that way i can tell what the heck it's doing um prompt brainstorm and then no don't keep it in the editor this one is going to be um each point so prompt point whoops i've already got it open that's fine close all those prompt point and then we probably need a conclusion let's let's not get ahead of ourselves because we're already breaking this down into a meta prompt um which is the cool thing okay um so let's go back into here so now we can't just enumerate all the prompts and just run them in order because that won't make any sense so we'll do prompt files equals actually this is going to be super procedural so we have to just do this one at a time all right so right intro equals prompt so what did i name this it's prompt intro.txt and then we will do argument equals gpth3 completion prompt okay and then we will do um actually here we can just shorten this do result dot append okay and then let's do the brainstorm brainstorm the rest of my essay actually we'll say critique to use the right terminology okay and so then we do we do this and we do brainstorm and so in this case we'll do um key points equals dot split lines okay so what we're doing here is when gpt three re sends this back we're going to split it into lines um so that way then we can enumerate for each line right for each point let's see and i think we need to make them stand alone um that's not right okay this is better yeah so this is okay when it says the author believes so what i'll do is i'll add the author to the beginning of the prompt so what especially this works especially well with um davinci instruct if you want a list of things and you want it to follow a particular format you just kind of start it and it's like okay the author does what and then it'll follow that pattern so then what we do is we just add this back the author said something to the beginning of um of the first one yep that should be fine key points um so then we'll say key points 0 equals key point actually we'll say the author plus key points 0. and the reason that i'm doing this is because this is going to get cut off it's not going to be part of the response so we want each each bullet point to be the same okay for point in key points now we get back into kind of this for loop um prompt equals open file and we're going to use the um prompt what did i call it the prompt point i come up with such original names prompt point dot text replace essay and then the argument is going to be gpt 3 completion we'll do that um yeah okay let's run this i think i think you guys will be surprised by this so basically what i've done is rather than me construct the argument i'm asking gpt3 to construct the entire argument because all i did was i said brainstorm i said brainstorm you know what the key points are and then in the first one um uh the intro i just said you know write the intro and especially so the introductory paragraph so that's what it's writing here okay um yeah so we've broken this down into three cognitive tasks which is write the intro brainstorm your points and then make an argument for each point let's see what happens cls to clear screen and then it blows up always does um what happened did i forget to do something um see 625 so did thorough assessment and evaluation yeah okay that looks right thorough outline of points the author oh that's what i did wrong i i yeah okay oops i used this one got to get rid of that we want that we want to preserve the new lines okay let's try that again so what happened was i went back through my logs this is why i keep logs and you see the response i said the author and then it begins by discussing but it's all on one line it's because sometimes what i like to do is compact the responses but in this case i don't want to compact the responses okay so let's try that again all right that's pretty typical for what we saw before yep it looks kind of repetitive so let's see what the final output looks like you see there's actually quite a few more steps so at 627 it ran it you know there was the introduction the brainstorm and then there was looks like four points that it made so let's go back out here and look at the output let's see this is a persuasive argument based on empathy and personal experience mills makes a compelling argument that illegal immigration is not necessarily wrong one weakness is that it does not offer a specific solution it looks like it's actually repeating itself almost verbatim let's take a look at some of those um some of these logs especially these last four because these are going to be where each point was oh that's what i did oops oh no wonder they look identical i'm not populating it um we're not using that we gotta we also got to replace um replace point good grief with the point because let's go to the gpt3 log and you see here where it's like the summary should focus on the following point point [Laughter] okay good grief i'm dumb you can tell i haven't had enough caffeine today yet it's also still 6 30 in the morning how long is this video while that's running oh this is 45 minutes yeah we're we're calling it here uh regardless but i think you guys get the point there we go i think this is probably good okay we're generating some output there we go oh wow it's kind of going back and forth but that's fine it is a little repetitive but this is certainly better okay so now let's see what the final output looks like reload it oh look at that this is long this is a 6000 character response i'm not going to read the whole thing but i'll save it in the output so you can see kind of what we've done and now you've seen meta prompting or prompt chaining in order to construct a very thorough argument i imagine with the right fine-tuning and refinement you could do the same thing for any kind of argument a legal argument or otherwise okay so i will go ahead and get this set up and um uploaded and uh you can check it out on github thanks for watching