Finetune GPT-3 to write an entire coherent novel (part 1)

Key Takeaways

hey everybody david shapiro here um so there was a post on the open ai forum that was asking about well here let me just show you um what was it the guy was asking about where was it um he was asking how to fine tune from scratch uh oh it was this one okay okay so let me show you guys this so this guy asked i'm using short stories that i wrote to bias the voice of of what gpt3 generates i've tried to format data in multiple ways but maybe before i spend another chunk of money someone already went through this and has some tips to share so he tried a few strategies some someone tagged me thank you for that pulling my attention to to this post i had done a similar experiment to basically using um fine tuning or gpt3 to do fiction so that was my auto muse project so the code is here it doesn't really work i abandoned it six months ago i just got a couple things working i didn't know as much as i know then i've been experimenting with it ever since so that's the background so here's my response i said achieving coherent fiction as as we imagine it is perhaps one of the most elusive tasks yet i would recommend following the document example of fine tuning where you leave the prompt empty and put the whole story or a big chunk of it as the completion this requires more samples but it can fine tune the model on your style and tone alternatively you might try having one paragraph of the prompt and the subsequent paragraph as the completion so like paragraph a is the input paragraph b is the completion so it knows you know follow up this paragraph with the next one and i said apparently ai dungeon uses a lore book mechanism so that the major details of the story can be referenced at all times for each completion with this you might be able to have a lore book section of the input prompt as well as the previous paragraphs and then you can have the next paragraphs as the completion if that doesn't make sense don't worry i'll show you what i mean because i'm just talking it talking through it so i said this gives me some ideas let's go it so that's that's what we're doing today i'm we're experimenting you're just writing shotgun watching the whole process i don't even know what i'm doing this is this is rapid prototyping okay so we're going to call this auto muse 2 experiment to generate fiction um let's see novel length fiction from a single story premise and i've already got a premise for this uh add a readme i'll be recycling some code okay so we got this clone it down get clone auto muse too okay so i've got a few things that i'm going to borrow there's the where is it um what was it the movie script generator hold on what did i call this oh that's right i was named incorrectly um okay so i already generated a whole bunch of movie premises so if you watch a previous video i go over about like just generating this but basically this is like a whole premise of a story um okay so let's copy the premises into our new um into our new uh the repo that's the word i'm looking for the new repo will be brain what are you doing what did i just name this thing auto muse too there we go okay so we're starting with some data that's what this is so i've got a folder that just contains uh it's 200 story premises and yeah they must now find a way to warn other humans of the impending danger um yeah so all very dramatic uh so here's my idea so my idea intuitively is we'll put in the premise playground so let's just this is usually where i start is like again you're riding shotgun this is this is a cold open um so we'll say like premise um let's see write a novel based on the following premise um and uh and yeah so we've got the premise here and then after that we would have like uh the story so far and probably do this and then i would do like um [Music] uh opening so like nothing has happened right the beginning of the story um let's see the human race has reached a point and um someone i think was on youtube someone left a comment like oh you should you should do a test where you um you say like write in the style of so like in my first book natural language cognitive architecture i showed that gpt3 like if you say like write as if you're a victorian gentleman it'll write like that if you say write like shakespeare it'll write like that if you say write like a chav which is like um uh what is that that's like a a ruffian from england um it'll write like that um so let's see write a story based on the following premise um write like frank herbert so we'll say like write like the dude who wrote dune um copying his style there are ethical implications with that um we'll see how it does so um novel uh let's see chapter one let's just see what it does uh it looks like it's probably just gonna copy yeah i think it's just copying what it had already the human races where i had the human race has reached a point where it exhausted all of earth's resources in a last-ditch effort to save our species we sent a colonization so yeah they were soon by met by a hostile alien race let me copy this into um notepad plus plus this might not work at all yeah it looks like it's just gonna copy copy it verbatim so this is this is a problem um with uh with with gpt3 especially davinci 2. um let's turn up the temperature crank it up to 11. um so with with fiction you usually want to have um the the temperature higher because that makes it more creative and less deterministic um but also i found that turning the frequency penalty up uh a little bit not not a whole lot but just turning up the frequency penalty a little bit tends to make it be a little bit more creative i figured that out when i was making the um the the movie script generator was because it kept generating the same like story patterns over and over again but if when i turn the frequency penalty up and the temperature up it got a little bit better so let's see what happens now yeah it's just copying itself um write a novel let's see if i can conjol this into add a presence penalty as well this like i'm saying this might not work at all um yeah okay this isn't gonna work that's fine i wasn't expecting it to i just wanted to do a test um the reason that i'm doing this is because a lot of you have left comments both on the forum and on youtube saying that you like seeing the whole process including what doesn't work so leaning into that see if you guys like it here's what i was planning on doing originally so big big step back how do you know how to write a story um practice uh no that's that's not just it you so there's there's several schools of thought when it comes to writing fiction there's there's plotters and there's panzers so like plotters are the ones that like i write an outline and then i plan out each scene and then i do this in planning planning planning and then finally you write the actual pros and then pantsers are the ones who just sit down at the keyboard and start typing most people are somewhere in between what i do for my fiction is i i start with an outline that is about a page long and then i just start writing the outline just says chapter one is this is what's going to happen chapter two this is what's going to happen so i just kind of have a vague idea and then my brain expands on that now can we teach gpth3 to do that so far it seems like it really sucks at this but sherlock holmes actually here we'll just do browsing options where is the top books popular there we go our most popular books sort order and then so what i'm doing is i've got this idea where if we just grab the text versions of playing utf-8 brilliant um is this the whole thing so basically my thought is let's grab the text versions of of of stories from gutenberg it's in the public domain so it's perfectly free there's nothing dubious about this this is all completely free data let's see which one is this this is frankenstein so we'll go back to the auto muse make a new folder called books go in here and we'll call this frankenstein which frankenstein is actually the scientist not the monster um you can tell i date a librarian um okay and then pride and prejudice so we'll save this pride and prejudice and this honestly like probably just one of these books will be enough but but my concern is if we if we generate fine-tuning data from just one book we're basically just going to be making a um a fanfic generator which nothing wrong with that there is a huge hunger for fan fiction out there um if this works i will try and make more harry potter fan fiction than you could ever read um that's not my shtick but i know plenty of people who have it one of my good friends actually one of my writer friends um she wrote a mass effect fanfic and uh this was like years ago and apparently she still gets emailed for it people wanting updates and she's like i've abandoned this years ago let's see alice in wonderland but if i can make a machine generate an unlimited amount of fan fiction then then we're really in business okay so save link as great gatsby and um the way that i'm planning on going about this is ultra janky um you're probably gonna laugh um but there's there's a reason that i'm choosing to go about it this way because i also want to show you that you can be super squishy with this gbt3 is surprisingly forgiving because remember all it does is predict the next character so if you just start in the middle of a sentence and end in the middle of a sentence it doesn't really care because it just says okay wherever we were let's pick up from there and keep going um so you don't need like clean divisions and stuff i'll show you what i mean in just a minute um sherlock okay so we've got five stories we've got um we've got a famous thriller we've got great gatsby which is um uh uh literary fiction uh alice in wonderland which is fantasy pride and prejudice which is historical and frankenstein which is um old sci-fi so we've got a we've got a decent decent enough cross section of um of of types of types of fiction okay so what am i going to do with this let me show you in principle what i'm going to do all right so gpt3 you can only grab like a certain number of um of characters or tokens at it at a time so what i'm going to do is i'm going to split all these up into chunks so this selection is 1000 characters basically what i'm going to do is i'll probably look for the double new lines maybe not i'll probably just grab like maybe 2 000 characters at a time and just split up the whole story into chunks sequential chunks and from there i will um i'll summarize the story as it goes so it'll be sequential chunks i'll summarize it as it goes and basically the fine-tuning data will be the story so far or first the premise so it knows where the story is going um so it'll be the premise like the outline okay outline's probably better so top of the input will be outline then it will be the story so far so like a summary of what's happened and then the paragraph you know the last paragraph and then the output will be the next paragraph so what that'll look like is so this is this is the fine tuning data so prompt um actually here i'll do input input prompt so this is this is what it'll look like so it'll be um um story outline and it'll be outline so this will probably be like a numbered numbered list uh story outline and then um story so far actually summary so far like where in the story we are uh maybe like where are we in the story and that'll be um summary of where we are and then um last chunk so this will be like last paragraphs and then next chunk and then this will be the end and then so this will be i'll just do end and so that's this this chunk here i was going to point out it on the screen you can't see my hand so i highlight it this chunk here will be the input and then the output so output slash completion will be um just you know next paragraph paragraph or two of the story so basically let's see if we can train it to spit out one section of a story at a time i doubt it'll work but i think it'll be a cool experiment all right i'm gonna pause the video because this is a lot of tweaking and stuff and also i'm recording at 60fps which means it will um it'll blow up the file size and i don't care i don't care to do that so i'm going to pause the recording and come back when i've got this split up i'm just going to write a couple scripts that'll basically do this data prep and then i'll be right back okay we're back prematurely because i realize there's something else that i'm doing that you'd probably benefit from seeing um i use gpth3 to give me a summary or the the outline um because i don't want to write an outline for these and also i haven't read them all and just so shoot shoot me sue me um okay so here's the file of frankenstein the story and then here's the outline so then this is all going to be broken down and summarized so let me just show you the prompt right the full detailed outline of the book alice in wonderland i put the um let's put this up to a thousand i know 500 tokens if it's longer than that it might not fit in the fine tuning data okay so then introduction that doesn't help detailed outline buddy detailed there we go hmm i don't like this we want you to give me a long write the long full detailed outline of the book alice in wonderland it seems like it is trying to shortchange me unacceptable machine unacceptable realizes it was all just a dream um this worked for frankenstein it's not working for this one so that means we have to do some prompt engineering uh let's see come on brain we want not just an outlined um let's try synopsis synopsis is a good word there we go um let's see uh let's see list every scene ah so this is like gpt3 has the information i'm just trying to like get it out of it that's one of the most like frustrating things is the information is in there this looks a little bit better okay cool or she wakes her up from her dream okay i don't think this is sufficient but you know blah blah it tells us most of the story okay so go save this and this will be outlines alice in wonderland okay you've seen the process i'm going to do this for all five books just that way we have a a summary or outline or synopsis to feed in so that that way the the auto muse generator knows where the story is going all right i'm gonna pause the video again and we'll come back once i've done some scripting okay i've got the first script uh well started i just want to show you what it does um so that you understand um as we go uh this is a really short thing i had to find this module but it's just called text wrap and what it does is it splits splits strings and python into whatever size chunk you want so what i do is i open open the book just as a text as a string and then i break it into 1500 character chunks and the output so far looks like this so i just run book to chunks and it says alice in wonderland.text 110 chunks frankenstein 293 great gatsby 194 pride and prejudice 518 is a long story sherlock 389 okay so we'll have a bunch of sequential chunks in order to play with this excuse me um yeah so we're off to a good start these chunks for reference they're gonna go here so basically the um the the outline which was already generated let me show you the outlines um so the outlines are here so we've got an outline for each of these that'll go up at the top so the story outlines so every every time you input this um gpt3 because here's the thing gpt3 has no long-term memory everything that it needs to know has to be in every single input this is where human brains are far superior to gpt3 so neuroscience lesson time if you look at like you know if you ever google like how much working memory does the human brain have it'll you'll get a some idiotic answer like oh the human brain can only hold seven things in it you know in short-term working memory that is patently absurd um any writer will laugh at that idea because when you're writing you have to remember all the lore from the story you're writing if it's a sequel you have to remember all the lore from the previous one um all the characters the character voices language stuff now is that something that you're holding in your in your memory and you're manipulating no so technically i guess you could say it's not in your working memory but it is in your recall so there's a difference between working memory and recall so gpt3 which can just recall stuff off the bat right off the cuff so if you go here um close some of those um this is what you might call recall gpt3 has been trained to just instantly recall a whole bunch of facts now if you're writing a brand new story it doesn't have that right it has no recall because you can just ask like you know you can ask it about the adventures of sherlock holmes and gpt3 can spit out usually a good answer the reason is because it's got recall however it does not it does not have the ability to recall something that it was never trained on so you have to handle the recall manually this is where all my work on on artificial cognition comes in because everything that you need it to know everything for any given situation has to be put into this corpus this input so these um these summaries these outlines that's got to go in because this because gpt3 needs to know where the story's going um let's see uh where it was there we go um and so then you need a say like okay where are we in the story um because if you just give it an outline and a paragraph like you if you are familiar with a story then you can read a paragraph and kind of figure out where you are if however you are not familiar with a story uh or it's a news story and you just grab a random paragraph you're not going to know where you are and gpt 3 is not magic right it's just it's modeling human writing that's all it's doing so if you just give it a random paragraph it's going to be lost it's just going to guess where it is in the story so that means we also need this in in what's called the working set so this is another part of neuroscience when you are working on a task um it's so it's a task set really it's what it's called so if you've ever heard about the term context switching if you switch from one task to another the task set which is all the memories and facts and knowledge that you've accumulated in your brain has to get shuffled off um think of it like like an old-fashioned office where everything is on paper so like let's say you know you're the the the imaginary people in your head they go fetch all the project documents for project a and then it's like the boss comes in and says wait we gotta work on project b so they gotta they gotta take all that all that task set all those papers file them away and then go get project b so what we're doing here is we're basically recreating the equivalent of human memory task set what's called a task set so that's all relevant information knowledge etc has to be in every prompt for gpt 3. in order to have in order to handle larger tasks now i didn't invent this this is how the human brain works i'm just approximating human neuroscience in gpt3 okay so that's why i'm doing this i wanted to sprinkle that in as i go let's see what do i have to do next i got to finish this that's fine i just save that out to file and then we've got to do the summary so the hardest part of this of this data because like you know i just downloaded the books and that's going to be the huge chunk of our data we've got the outlines again that's a huge chunk of the data the hardest part is this because i'm going to have to use gpthree to summarize where we are every step of the way i'm not going to do the whole books i'm only going to do like the first you know few chapters um because one i'm not going to spend like a thousand dollars fine tuning this for five full books um it probably wouldn't cost that much um this should just be a good proof of concept um but yeah so and basically just creating a live a running summary of the story that's gonna be hard it's gonna be expensive okay pause the video again we'll come back when i'm ready to show you more standby all right quick check in as promised um i finished the script there's got to be a better way to do this i don't know how to pad zeros but basically the purpose of this um little bit is to generate um sequential files so you see each of these stories is broken down into sequential files uh so they're just straight up numbered it's in the chunks folder so now each chunk is just a bit of the story also it did this weird thing where there's no vertical white space i think that's okay i'm not going to worry about it this is not going into production i'm just seeing if it even remotely works um yeah uh oh sorry my phone's going off anyways i will uh be back shortly all right i was about to give up on this and take a break but i figured it out well i don't know if i figured out the root cause but i was getting this error let me show you where i kept complaining about um like could not encode where is it i'm just probably go all the way up yeah so you see here error and communicating with open ai charmap character kodak can't encode character blah blah blah all right so there's something funky with these files from gutenberg so the first thing i tried um was to go back whoops we can close this one um sorry this might make a little bit of noise okay so my first right the first thing that i did was i tried to go and change the encoding of the books so in coding it was utf-8 bomb so i just converted it to utf-8 that didn't fix it um so i was like okay um gpt3 should be able to accept all utf-8 but there's some artifact in this that it didn't like so here's what i did was i said okay let's open the prompt and we'll get we'll encode it to ascii which is much simpler than utf-8 and then we'll decode it back to a normal string so it's basically saying okay whatever whatever this is ignore the errors simplify this codec make it ascii standard and then decode it and then gpt3 likes it now so it's uh recording this basically what i'm doing um while this is running so we've got all the chunks uh here and then i've got the summaries so i'm making a summary of the first um you know few uh few pair uh passages of each story um so alice gets bored sitting by her by her sister and sees a white rabbit with a pocket watch she chases it and falls down a rabbit hole so basically these summaries can be used to stack up and fill in this part here let me just save this file we'll save this in the um here actually i'll put this in to the readme do the readme all right general idea and we'll save that um so this is what it just so that it's there it's saved and then i can close this okay so the outlines we've got the outlines i will show you that again where are we in the story that's what their summaries are going to be used for excuse me so each chunk is going to be summarized and this one you know it's down to two sentences so alice in wonderland o2 if we open the summary we can um compare oh that's that is the summary sorry i need the chunk so chunks um alice in wonderland o2 so you see it goes from 1500 characters down here down to 127. so by a fact it's compressed by a factor of 10 and then you could also probably like recursively summarize once it gets too long i'm not going to worry about that this time because this is just a huge experiment in generating something as far as i know no one has ever succeeded in generating a whole actual novel with this kind of structure so this should be done by now excellent all right so if we go into the summaries we've got 11 chunks for each so that's 55 i'll probably need more let's see hmm let me think about this because this is 55 55 summaries which will then be used because if i have five stories i would need um why can't i do math 5 times 40 i would need 40 for each one in order to get to 200 samples okay so i guess i will need to do a little bit more darn okay i will let this run i'll figure out how to uh pick up where it left off and um we'll go from there be right back all right um this is getting rough so i'm gonna have to split this up into two parts but let me show you what i've got so far so first let's just step through the data because that i think will help i've only got a few scripts so i've got the summarize chunks prepare json l and the book to chunks so the books start here this is like the super raw data it's just a gutenberg book it's plain text there you go nothing special there so let's close all these to clean it up all right so we start with books right then we break that into chunks and each chunk is 1500 characters long um 1522 why is it 1522 whatever it was supposed to be 1500 characters long close enough they're all the same length more or less i'm not sure exactly how that um text wrap does does its thing anyways more or less the same length broken up into chunks so because basically think of it like an inchworm the idea is you write one paragraph at a time or so um so that's the idea all right so we take the books we break it into chunks then we summarize each chunk and so each of these summaries is nice and concise very short um oops don't need to go there you don't need to see all the research i was doing to try and figure some of this out okay so then we've got the summaries finally so we got oh we also have the outlines sorry um the outlines which kind of outlines the whole story just so that that way it knows what we're trying to achieve okay so finally we put all that together in a prompt um so here's the the full prompt so basically we say okay here's the outline here's the story so far and here's the last chunk and we're asking it to produce the next chunk so what i did was i saved all those out to the same file names this this just makes it really easy but one thing you'll notice is that these files get progressively larger um so whoops didn't mean to close that so let's see which one is the shortest one i think alice in wonderland is the shortest so the very first prompt outline alice is sitting in in the river or sitting with her sister by the river so cool that's fine um story so far alice adventures in wonderland is a novel by lewis carroll that's fine last chunk so last chunk is like what what it's so the the last chunk is that is the full prose and the story so far um this one because it's the very first one um just has the uh has the summary of this um yeah and so then the next chunk that we're going to ask it to produce is going to be the same as so notice that uh this prompt or this this one is 0 0 0 1 so the next chunk that we're going to match it to is so we've got another folder called completions so it's going to be here so alice was beginning to get very tired and so what we're asking it to do is um very tired uh beginning to get very tired of sitting by her sister on the bank so you see it it just continues right along and so by having the pairs of prompts and completion so you see there's the prompt and that's what the prompts look like and then the completion that we want it to achieve same exact file name so it's easy to pair them but so this is the first one but let me show you how big the last one gets because it's accumulating and this is um this is this is the biggest problem um let's see so the completions are the same time same size each but the prompts so the first prompt is 4 kilobytes the last prompt is 12. so the outline stays the same length that's fine story so far you see it's getting longer and longer oh but wait there's actually new lines so finally you know the the summary gets super super long so instead of just being this first one because you remember you know the queen's croquet game this is where the first one ended so what i'm doing is i'm accumulating them all right here so it's a summary of the story and so instead the summary is now 11 000 characters long typically for the original da vinci it would average about 6 000 characters with is 2 000 tokens um so we're way way over that we're about twice what could fit um and and and we still need to get the rest in um the the the last chunk so that was just the length of the summary um all told oh and also not just the length of the summary and the the last chunk but the next chunk has to fit into that completion as well so these are just way too long but i've got the data so i've got the um oh that's not the right file delete that one it's novel.json l so this is a two megabyte file and this is what it looks like so it's just this huge mass of json-l okay so that's about as far as i'm going to get today this code will be up on github i'll come back to this soon i'm exhausted okay so basically what i need to do is let me show you what the script does that accumulates this so the script that composes the json l is right here it's called prepare json l it's really simple so basically what i do is i grab all the books in the directory called books ta-da grab that there instantiate a new list and this is what i'm going to accumulate all the samples in so four book in books so we iterate through each book iterate through each one of these we grab the name of the book and then from with the name of the book we grab all the summaries that are in that book so then we go to the summaries tab or a summaries folder so it's like okay so if we're starting with alice in wonderland we just want the summaries that have alice in wonderland in the name so that'll give us these 41 summaries oh and also they get really big and this isn't even done this is just the first 40 41 sections pride and prejudice has over 500 sections so how the heck are we going to summarize that that's going to be difficult because you're basically going to have to like super super ultra paraphrase it [Music] i don't know if it's going to work but because we're working with a very narrow window and so this is why i laughed at the beginning of the video where i said you know anyone who says like the human brain only can only hold like seven things in its working memory no you you you try and actually summarize and figure out how much working memory you need to write a novel um that or us novelists have a working memory that is several orders of magnitude greater than the typical person because you need to keep so much in your mind when you're writing a story unless there's something i'm missing who knows um anyways so back to the script um all right so we get so we get all the summaries for this particular book we get the outline for this particular book then we instantiate a new string called summary chunk so this is what this is what accumulates um those uh those summaries in the uh prompts right here so you know we go down to this one so the the story so far this is the summary and so this is why it keeps getting bigger is just because i just add to the next um each time so it gets just longer and longer and longer every every iteration so last chunk is open file that has the same name as the current summary because remember the um the summaries and the chunks have the same name because the summary is just um you know it's a summary of that chunk so alice in wonderland summary o3 is just a summary of this chunk of o3 so suddenly alice had the moment to stop the thing uh before falling and so then you look at summary alice falls down a very deep well and has plenty of time to look around as she falls so you see like okay cool um so we've we've reduced the length but that's still not enough because it accumulates here okay so we we we incrementally build up the summary next chunk prompt so then we compose the prompt which is basically just load this and populate it so you know outline summary and chunk the outline is is populated with this variable you can see that here so replace outline with the outline replace the summary with the summary chunk which is the uh the string that keeps getting longer and then we um finally we replace uh next chunk or sorry last chunk um with last chunk is is basically the current chunk um and the next chunk is i wrote a quick little um whoops did not mean to close that uh prepare json l edit um so i wrote this quick little script that you just pass it a file name or file path of a given summary or chunk and the name of the uh of the book and it will it will just increment and figure out what the next um next chunk is for the completion and then it'll just pass that data back so this is what it looks like i had it output the length of each of each one as it goes let me let me add a little bit more so you can see let's see so we'll have name so that's the name of the book and then we'll have summary is going to be the name of the file so you'll be able to see like which well i guess the summary is going to have the file name in it so we'll just do summary and then length of the prompt plus next chunk so you can see how big that sample is okay so for sherlock 01 the length is 5500 characters that's about at the limit already and then by the end for sherlock 041 it's 14 000 so that's more than twice what we could um expect to fit and that's also not even like not even the full length of the of the story so i'm gonna have to pause here and go back to the drawing board and do some do some thinking some noodling on this um so stick around for part two it should come out within the next week or so thanks for watching