How to create synthetic datasets with GPT-3

Key Takeaways

hey everybody david shapiro here with a video about gpt3 fine tuning and generating synthetic data sets you can find me on github and it's literally just dave shap that's me you can find my picture so gpt3 and fine tuning are super popular right now and i have gotten really good at creating synthetic data sets what do i mean by a synthetic data set so in data science or machine learning a synthetic excuse me a synthetic data set is data that is artificially created it's data that you create using other machines or programming and uh so the ultimate result let me just kind of show you what i've recently created so two weeks ago i put all of my fine-tuned data together in one repository i've got a compassionate chat bot here's the data right here it's ready to go you can put it you can grab it and fine tune your own chat bot i've got some core objective functions which are really big but it's basically my core objective functions are how to create a benevolent agi and then lastly a question generator so question generator is you give it a topic and it asks relevant questions about it so i've got these three very different cases about synthetic data set and fine-tuning gpt-3 i'm not going to spend any time kind of showing you what those actually no i take that back playground so let me show you how these actually behave so scroll down here and i will grab um fine tune that's curie that's okay um user so let's say hey eve i'm sad and then um it'll go from there uh it'll probably fail but i'll keep talking while it's loading so anyways um this this uh fine-tuned model took only 200 samples and i've got a chat bot that is uh pretty powerful um [Music] i created this idea there you go um so it's just creating this whole conversation i forgot to add the stop sequence so that it can so it will not keep talking okay so the stop sequence will be user and eve so that means it'll stop so let's see what eve says user i'm sad um i'm sorry to hear that what's wrong user um my cat died eve how did she die okay so you see where this um this chat bot is it's uses a technique called compassionate listening okay great so how did i how did i come up with this well first we need to go we need to rewind and i need to tell you about synthetic data sets all right so synthetic creating synthetic data sets is difficult but it might be easier than actually going and getting real data from the real world or curating a data set by hand so here is an example of where i started this was the first synthetic data set that i made you can go explore all of this it's um the nauka question generator so i started with a whole bunch of stuff that i pulled from all over the internet these are all publicly available data sets i found most of them either on kaggle or google data sets just search for nlp data sets and you'll get stuff like this where it's like here's a news article that's great here's an entertainment article again great so this is all bbc news this is all stuff that's pulled from reddit it's way too big there's also plenty of scripts to um to pull data from reddit there's medical stuff movie lines this is the cornell movie database um stack exchange um so basically you start with some raw data whether that is whether that comes from stuff that you scrape from the internet yourself or gutenberg gutenberg is another great place to get um text data so then what i did was i wrote a few scripts to kind of pull that and massage it into what i call contexts so context is like some fuel it's some food for thought it's just a basic text file scroll all the way down um it goes way down okay well it's not going to show me anything other than a dialogue that's fine so the point being here is that you start with some raw material and you can there i will show you how to generate raw material in the future later in the video but this is this is where i started so i'm just kind of walking you through the story so what i did then was once you get some context um this folder has like 50 000 contexts in it to choose from again feel free to use it yourself then you write some gpt-3 prompts so you say uh let's see write a list of the most important and salient questions an observer would ask about the following passage it took a lot of tweaking to get that to get this prompt right so let me grab this prompt we'll go back over to the gpt3 playground and we'll switch back over to text davinci important observer questions and so let's then also go back to grab one of these contexts so we'll just grab a random dialogue uh that's not quite long enough usually does better if there's a little bit more information so just click around until i find a longer there we go perfect okay so we'll grab this this is a context and because i used medical text reddit posts all kinds of stuff in creating this synthetic data you can see this is what you're seeing right here is i you just automate this you put in a whole bunch of stuff you ask gpt3 to um [Music] uh to generate a list of questions and then you record that output so let's see what it says and these are all default settings too um so it'll just say okay what is the dynamic between the characters what is the settings of the passage what is the tone of the passage great so it has intuited that that what we're seeing is um is like dialogue happening between individuals um if we run this again with a higher temperature it might ask questions about the characters so let's turn the temperature up there you go so you see it has asked like what is their relationship um and then so if you if you do this again and let's just be like well no i'm not going to do that you can you can play with it yourself these these prompts are available online for free so you run a few hundred or a few thousand different contexts and you generate the questions and you store those questions um so we're already most of the way done with this first data set or this first um method sorry let me mute my phone i did not realize it was not muted okay no more distractions okay so once you get a whole bunch of questions and a whole bunch of contexts so here's the folder i accumulated all the context and then i accumulated a bunch of questions so you can see here like it generated questions i clean them up and so what you end up with is a pair of contacts which is a context is just a situation can be a reddit post oh here we go medical questions what is the patient's diagnosis what is the anesthesia used for so this basically the whole point of this was i wanted to create um a a curious model so a curious model increase understanding that is um core objective function number three curiosity is going to be an imperative uh quality so that we can have benevolent um artificial general intelligence and so the ability to have a model that will just automatically ask questions that was the purpose of this so we've got context so scraping together context that's the input that was a big big chunk so we got context and questions right and then i started working on summaries i didn't really do that that's fine um ultimately though this was the result and i know it's huge and super confusing so we will just say like okay so here's a prompt which was dialog and then i end it with questions so you need a dmarc and then i just copied those questions so you end up with a pair and let me actually copy this so i can show you a little bit better what it would look like to a person okay so let's clean this up just a little bit and i will zoom in bear with me there we go okay so let's remove let's find that find what backslash backslash replace with a regular backs a new line there we go okay so then we say this is what the input is going to look like and then so i'm basically i'm just reverse engineering what all my dataprep scripts did because when you fine tune it needs to be in a particular um it needs to be in a particular format and so the input is going to be this bit here and you and i what i what i always recommend doing is you you append at the very end kind of the the instructions and this will this will train the model that whatever came before it whether it's a medical text or a legal text or a reddit post this says okay time to ask questions and so then this is the output this is what i trained it to output and i because i was using a more primitive gpt3 model when i did this originally i had another cleanup script that removed anything that didn't have a question mark at the end of it and so the ultimate result is now i've got a fine-tuned model that is an expert at asking questions for all kinds of things so there you have it there's step one or experiment one all the data prep scripts are in here you can take a look at those there's also a few scripts out here like format the training questions generate the questions you can look at the prompts that i wrote for asking questions like predicting the future and whatever but really this one is what are the most important and salient questions that this wording turned out to be the best because it allowed gpt3 to be flexible it allowed it told gbt3 okay be a smart person and ask probing questions about this um oh that that term probing questions that can also be a really good prompt so yeah okay so that was that was synthetic data one come check out this repo if you want to the second one is going to be my core objective functions so i'll show you let's see to do i started the same way with actually no let me go to experiment one so i copy-pasted those contexts and i created a series of prompts and you can see this repo is a little bit simpler again this is publicly available you can come take a look at it yourself but i had prompts that were a little bit different so these are few shot prompts and what they did was discuss suffering in the following contexts so i gave it a context and then suffering context suffering context suffering and so then i would just replace the context with one from the internet from a data set and then i would ask gpt3 to produce some dialogue about the suffering and then the ultimate result was something like this where it says this passage indicates that the person has suffered from isolation and feeling trapped so again the goal here was to give agi the ability to understand human suffering because if your agi if your machine learning model has an intrinsic understanding of suffering well it can avoid suffering so that was the point of that experiment but again it followed the same process where you start with some raw data so step one get some raw data step two get a prompt and step three is record all of the output so then i did the same for these core objective function two is about understanding so increase understanding means that it wants to teach people it wants to ask questions that sort of thing and all those prompts are going to be um so here's uh here's some of the prompts um here's some other prompts and so there uh so on and so forth now i started working um i started working on downloading data directly from reddit so that that way i could just say okay let me go to a reddit download some some contexts from there um that i developed in future experiments so let's see i've got even more here download reddit posts yeah so here you go this is reddit is a great source some people are concerned about privacy and the legality of downloading posts the us supreme court and superior court have both recently reaffirmed that any any data that is publicly available on the internet is legal to use for pretty much any purpose and this is certainly not a malicious purpose so as far as i can tell the law of the land is settled you're allowed to use stuff that's publicly available on reddit um and that's as far as i can tell that's the end of the end of the conversation oh sorry one thing that i did forget to mention is the end result is always a json l file so in this case here you see the same same format there's a prompt and then there is a completion now in this case the um the demarcator is reduced suffering let me zoom back in let's see reduce suffering where did it go um okay reduce suffering so this tells it what the instruction is this tells it which core objective function and then this passage indicates the person has suffered from isolation and feeling trapped and so this is the training data that you give you give it and then you end up with a model that in this case abides by my core objective functions so this first model in natural language natural language cognitive architecture the question generator that's for generating questions this one the core objective functions is for reducing suffering increasing prosperity and increasing understanding so that's a very different purpose you can you can browse through the data it's all in here under um as a json l file it actually looks like i didn't finish this experiment because i kind of realized that i needed to go a different direction but still generating synthetic data you can look at each of those steps in here and how i got it all this is going to be the most comprehensive one is the core objective functions and so lastly i created this um this chat bot and this one was fully synthetic i had no raw material so let me show you how i did this so there's a prompt let me show you this prompt and what it does when you come up into gpthree so i say eve is a digital life coach that uses compassionate listening and is inquisitive about the end user eve's goal is to understand the end user and help them untangle life problems every response should either ask a probing question so there's that term probing question provided definitive information or make a concrete suggestion so convo so this was one i actually had there was another prompt what was it synthetic prompt that's what it was so this is so there's um there's several several prompts um this one i actually realized was better so let me actually show you this one because this allows it to synthesize all i did was come up with a list of topics so let me show you that list of topics so what i did was and all the scripts are here actually this this repo is not public so you can't see that see this one um but yeah so i came up with a list of topics and then i fed it into this prompt and let me show you how effective this is um so navigating office politics so navigating office politics so all you do is just start here and the conversation starts with someone saying hey eve let me turn the temperature back down to 0.7 and watch what this does oh whoops i need to remove those we don't want it to stop here of course what's going on user well there's blah blah blah and so you see it's imagining an entire conversation entirely on its own all i did was give it say this is this is what the prompt is imagine a long text message chat between a life coach and a user the life coaches eve who uses compassionate listening to help her clients succeed in life eve frequently offers ideas and suggestions or teaches her client helpful facts eve maintains a professional tone the topic of the following conversation is navigating office politics okay so you see there's one synthetic conversation right there all i did was tell it what to do and it generated the entire thing the topic of this next one let's say um quantum physics just to show that this prompt is infinitely flexible do you know what do you know about quantum physics and then of course it gave up user let's say absolutely nothing i'm an idiot don't say that about yourself you're not an idiot you're just new to the topic and so look she says let's start with the basics quantum physics is the study of behavior and matergy at the subatomic levels um okay that makes sense how can quantum physics help me in everyday life um yeah so there you there you have it like any you just swap this out with any topic let's say um i'm not going to choose something controversial um let's see um oh here um doing taxes this is something that we all just recently had to do can be really tough to get started on taxes and then she makes a list etc etc there you have it yeah so this is these are some of my tips for making synthetic data um this repo was never meant to be public but the the fine tuning set that resulted from it is publicly available it's under here my gpt3 fine tunes under compassionate chatbot here's the data right here i do also have the synthetic prompt and the topics so what all that you need here is available under compassionate chatbot and i've even got the script to synthesize the conversations so that you just look at the topics and it'll synthesize all kinds of conversations for you so there you have it there is my tutorial on generating synthetic training data with gpt3 and for fine tuning have a good one thanks for watching remember to like and subscribe if you found this helpful