Finetune GPT-3 to write an entire coherent novel (part 2)

Key Takeaways

hey everybody good morning david shapiro here for another video um we are going to pick up on auto muse so last time quick review this is the second video last time we left off with we kind of ran into a problem because because each each of the chunks got successively larger and so what i mean by that is if we look at the data so what what we're trying to do here is we're trying to like build up a book through successive chunks and so what we do is we include the outline and then the most recent um well here let me copy uh this will be easier if i just show you kind of like the whole thing um okay here we go that was a summary actually i apologize um sorry we're all getting recalibrated here um oh well i can just show you the prompt okay so here's the prompt that we're filling out this will make the most sense oh let me zoom in so you can see it hold on hold on there we go okay so this is the prompt that we're filling out because and this is this is the prompt that we're going to be fine-tuning also let me make sure that you can hear me hello okay cool um i replaced the the cable on my microphone to me when i record it still sounds quiet but y'all will let me know if it's if it's too loud too quiet the gain is turned way up i think this microphone is just dying anyways sorry for the aside um okay down to business the outline so in order to write a story we're having to imagine like okay it's got the plan um because if you just like give it you know give it the reins it'll kind of just wander wherever but uh in a past experiment um i generated a whole bunch of plot premises so let me show you those so here's an example in a high fantasy world there's a war brewing between the humans and the elves right okay so here's a whole plot premise the humans are forced to flee back to their world where they must start rebuilding their kingdom right so this is this is a story premise um and you take that so you put that here and then um the story so far so this is like just a natural language summary of what has happened in the story and then here's a chunk of prose like so that what's actually been written and then what we're going to ask it to do is write the next chunk and so what i've been doing is i've been getting all of this scraped together so we've got outlines we've got story so far last chunks we've got all that accumulated here so we've got chunks so i've got alice excuse me alice in wonderland frankenstein and all these different stories split into 1500 character chunks right so we give it one chunk after another and so that way it can kind of keep writing right it's like an inchworm inching along um but the idea is we want to train it with existing novels so that that way it can learn to write entire novels now the problem was as we get further into the story the summary of what's happened so far gets longer and longer and longer so let me show you the script that i used to do this um i think it was this one yeah so did i do 1500 i did not okay so this is um there's a well let's let's go through all the all the all the um things so far okay see this is the problem of coming back to code is like where was i where was i what was i doing okay um book to chunk so this was the first script and basically all it did was um break each book so we started here we took these are the whole books from downloaded from gutenberg so i took the books and broke them down into chunks so that's that's the first step that's all that this first script does is it generates the chunks and they're in they're in sequential order um so you know it's just manageable manageable chunks of pros um and then summarize chunks so for this what i did was i just took the first like 11 chunks um or sorry 40 chunks um and and summarize them so the summaries are here and so it's uh it's summarizing each chunk and it's it's a one to one so if we go to alice in wonderland 15 um in the chunks the summary is going to be a summary of that chunk so was it 15 so thought alice i dare say it's a french mouse come with william the conqueror blah blah blah so then you read this alice encounters a mouse who's terrified of cats the mouse explains that his family has always hated cats and he doesn't want to hear their name again um so yeah you can see it took what 1500 almost 1600 characters and summarized it down to 154 so that's a compression factor of 15 to 1. so the idea was we would take these summaries and we would so you see here where i pick up the summaries so summaries equals i for i and os list are summaries if name is an i and the name is the name of the book so basically what i do here is i just grab the summaries for this book outline so that i open i open the outline of it and then we accumulate the summaries so then for each summary in summaries we're going to accumulate the summary chunk make it longer and longer last chunk equals so the last chunk is goes here so this would be like so alice in wonderland 15 so it'd be this one and then the next chunk would be what we want it to write next um so that would be chunk 16. um so we just iterate get the next chunk but the problem was even with these even with these short summaries you accumulate 100 of them it gets to be really long because 100 times 154 is 15 000 characters that's not going to fit into a gpt3 prompt so that's where we got stuck last time okay so class in review so what we need to do is we need to summarize this so the summary chunk if the summary chunk is too long we need to summarize the summary chunk um and i had an idea i could have sworn that i put in the code but so the idea that i had um was if summary chunk oops gotta do length summary chunk greater than fifteen hundred um then what we'll do is we will ah we need to grab this function gpt 3 prompt also that means we'll need those functions okay so just copy this this in here um and what we're gonna do let's make this a little bit longer just so that we don't run out of space is we will get yeah okay so prompt equals open file prompt summary yeah replace chunk with summary chunk so basically what we're going to do is we're going to ask it to summarize the summaries and so what this is going to do is wait did i already do that here prompt equals open file oh this is the full prompt okay if oops if summary is too long um summarize it okay so since we're sequentially adding summaries one after the other um what it'll do is the older the summary is the more like so like this one well here let me just show you this will be easier let me just show you keep it simple all right so we'll do summaries okay so let's just open the first 10 summaries okay so alice's adventures in wonderland is a story by lewis carroll so what will happen is each of these will get end up getting added in sequence and once this is over 1500 characters the this algorithm will summarize it and so then the oldest ones will end up getting summarized repeatedly see see what i'm doing here so since it gets summarized repeatedly the the older the information is the more it'll be summarized so the more it will have decayed um and uh since it since since it'll have this like kind of time decay i'm hoping that that will keep it keep it smaller okay so that's fine but this also means that for every summary where if the summary gets too long we're going to summarize the summary which means a lot more calls to openai so it will get a little bit more expensive another thing that i need to do that we're going to work on is we're going to um shorten these uh these outlines because like the outline for sherlock 2200 characters that's that's a lot but um pride and prejudice that's 3 400 characters that's too dang long um great gatsby 1800 that's probably okay frankenstein 2300 that's pretty long alice in wonderland 1400 that's a little bit better um because the thing is um with the with the with the outline or the premise all you need to do is kind of have a guiding star for how the story is going to go you don't need to give it too much detail okay so now that we are hypothetically oriented to the plan also let me make sure we're still getting audio yes good can you hear me now um where was i i accidentally closed the summarize okay because i was copying my old code and we needed to do this because there is something funky going on with the stuff downloaded from gutenberg it all should be in utf-8 but it didn't yeah so and what i did was last time i found out that i needed to encode it to ascii and then ignore errors and then decode so that'll just put it back into a normal text string that fixed it okay great and then response equals gpt3 completion response except what we'll do is we'll just do prompt okay so if the prompt or if the summary is too long we'll make it shorter so that should fix that part um let's save this um because you can see that this was two megabytes so we'll save it as novel two so it doesn't get overwritten because you see here we're just gonna overwrite it um yeah so there's that and then let's work on shortening the outlines okay um this one's fine we can still make it more more compact so we'll just put it all in a single chunk oh sorry my dog needs to be let in pause okay and we're back sorry about that um [Music] so we're basically trying to um reduce the the word count or the character count as much as possible but also it'll just be in a nice paragraph because if you look at the um if you look at the uh the which mccollum's that i made the plot uh plot synopses one two three one two three then um you know gpt3 just tends to not write in bullet points it also sometimes so here's the thing sometimes gpt3 handles lists really well i if you have a labeled list so what i mean by a labeled list is like um you know describe a company so a labeled list would be like location and you'd say like north dakota um name would be like joe's emporium or whatever and then you'd say like uh established um we'll say like 1848 or whatever so if you if you gpt3 is really good at ingesting information very quickly if you use a labeled list like this but if you use numbered lists gpt3 doesn't seem to understand it as well so numbered lists are more for human brains gpt3 doesn't really it doesn't really care about numbered lists and i'm not sure why um so but if you give it like a natural language paragraph it seems to understand that a little bit better um it might just be that that's a uh an artifact of the training data um like maybe like it just didn't ingest that many um uh examples so we we can we made this more concise by about 100 or 200 characters um but then what i wanted to do is show you the edit feature um let's see reduce word count word economy so let's see how that invalid url hmm i wonder if that's down text davinci edit okay it might be down that's fine um reduce the word count of the following passage without losing detail um apply word economy yes i know follow winga um and then um new version with fewer words um is the whole thing down right now status says operational no incidents i don't know what to do uh okay let's just save that real quick and we'll do a full refresh maybe if i log out and then back in and we go back to playground hey there we go you can tell i do technology have you tried turning it off and on again that's literally all i did okay this doesn't look like it really did much word economy the novel begins with a series of letters from robert walton like there are so many like ways that you can make this concise um i guess it removed a series of letters and just letters okay fine i know it might contain sensitive content okay so let's see frankenstein so 2236 down to 2181 okay that's not so bad so let's do this iteratively so let's then take that output and ask it to make it concise again um and so you see like we're down to like 434 tokens um so that means that uh that means that we're getting uh more more efficient because also since we're going to have to send this through repeatedly we want to be as efficient as possible okay so let's see if this gets any more concise and you can see how like if you iteratively if it does get shorter um then it will be it will be um kind of have that decaying summarization thing because like the further back in time something is in the book the less relevant it is but you might still need to know the key plot points because again gpt3 does not have long-term memory okay so this just reworded it it's literally the same same word count but not any shorter okay so let's see how can we make this any shorter well you know what let's let's just move on it's 2100 characters um it's shorter than it was okay so let's do this one mysterious millionaire who throws big parties it just seems like another parable about like how chasing women is bad like just be honest y'all i know that like some people think ah you see this is proof that women are all temptresses and you should you know we need to control women no this is a parable about how men can't control themselves come on man come on we're not animals grow up okay 1756 characters so let's run this through our compressor uh once and just see what happens new version with fewer words reflects on the futility of the human condition look at that okay down to 1600 characters happy with that pride and prejudice so this one's gonna be a little bit harder and also i'm i'm done doing this manually let me teach you guys some regex and i know some people say regex because it's regular expression but to me it looks like regex because if you have e-g-e it's the soft g sound so i'm just following english standards okay so carrots that means anything that starts at the beginning of a line followed by a digit and we'll do digit plus oops digit plus so that's one or more followed by a period followed by a space one or more okay cool so we want to replace all that with nothing and then we want to have space of two or more because at the end of the at the end of the rows it's um you see how it's uh the new line it's a new line and carriage return so it's r slash r n um and so we want to replace that with just a single space bam all right so now we have compacted it a little bit we're still at 3300 characters that's way too dang long okay new version with fewer words nick caraway so let's see how much we can summarize pride and prejudice even more um did we run out of tokens we did didn't we because this is long so because it's interesting right because if you're trying to summarize a book that's like 800 pages to 1500 words or 1500 characters versus one um that is like 100 pages or however long alice in wonderland is all right so let's see we're going from 33 66 to 29.78 not bad not bad okay um but yeah because it's almost not fair because like if a lot more happens in a book and you're trying to compress it down to the same window like that's not fair that's not fair to that book um but because it's too dang long we really need to get this you know get it down as much as possible excuse me i don't think it's doing anything different i'm fighting it come on can you tell that i'm getting more and more unfiltered yeah you if you look at the things that are underlined they're in the same positions long born netherfield netherfield netherfield longboard netherfield another for netherfield i doubt this is any shorter but that's okay we did our best okay so 2978 oh we did get a few more characters squeezed out of it so we went from what was it originally 33 66 to 28.70 so that's 500 characters shorter not bad control h we are going to replace no that's not right you're supposed to do that differently slash d carrot slash d okay slash period the reason you have to do backslash period is because period is like the uh the universal character for regex slash s one or more why did it is that not replace replace all it only did up to notepad you drunk that doesn't make any sense oh wait okay hold on maybe i'm the one who's drunk get some there we go okay replace all why is there still a leading space find leading space see there it is oh because i still have the space there i am the one who's drunk okay and then we do um space of two or more and so this also is good because like space of two or more oh slash s is white space so that's like new line carriage return normal space tab so any white space and just replace it with a single one okay so now we have compacted this solve the case okay so we're down to 2100 characters here so let's go ahead and summarize this as well because you know just the the fewer tokens we send through gpg3 the less money we spend but also because excuse me because gbt3 is really smart it can confabulate and fill in a lot of blanks so it's weird because in my first book natural language cognitive architecture one of the principles that i that i wrote about in that book is um you need all the relevant information and no superfluous information so like everything that you need and nothing that you don't okay so let's see this goes from 211 is it longer 21 2113 to 2131 they made it longer no okay we'll just leave it as it is then okay so we've made our premises a bit shorter it's still okay still not ideal but they're compacted they look prettier they're also in just a big paragraph which gpt3 tends to understand better so now that we're all caught up let's do this so for book in books um ah let's see let's add a little output say print summarizing the summaries just that way we can see what it's doing do i have any other output oh yeah there we go okay so print summary length prompt length next chunk yeah yeah okay prompts and completions oh that's where i've got them saved out to yes okay so the prompts are all saved here yeah okay so um the outline will be instead of a instead of a numbered list it'll be a chunk and then the summary of the summaries will be here and this will get shorter well it'll have a max of 1500 characters because it'll be it'll be shortened um and then the last chunk so this is the pros and then basically what we're asking it to do is then um so this is great gatsby 07. so let me open great gatsby 07 and the completions um let's see great gatsby 07. and so basically what we are training what we're training gpt3 to do is this so they spent a year blo in france for no particular reason so you see how it just continues it right along um ideally it will be you know like gpt3 will only spit out one whole paragraph at a time but let's see how it goes also let's not save that no um okay all those are all gonna get overwritten anyways before we go uh okay i think we're ready to test let me back up my work um what is this auto muse get status get add git commit am um saving work for part two i know i make the most awful git commits um get push okay save a work so now we've got no risks python is our python prepare json l um sleep is not defined we have insomnia i'm sorry i'm sorry for subjecting you all to myself right now okay actually no i'm not sorry you're watching this on purpose error communicating no api key provided what do you mean i need an api key guys i'm having a rough time of it today okay summarizing the summaries hey look at that so it's still getting too long hmm yeah it's still getting progressively longer interesting because so my rule of thumb my mental rule of thumb is that a gpt3 prompt is limited to about 6000 characters that that tends to be around 2 000 tokens 2048 which is the limit right now so this is not getting summarized accurately um it could be that i'm doing something wrong prompt equals oh that's the problem i'm updating the prompt not the summary chunk the summary chunk needs to be equal to that yeah guys the man okay let's try that again oh also this is not big enough sorry properties font 28 boom okay so now the summary should not get too much oh there we go look so yeah there we go perfect oh there yes yes yes yes summarizing summaries why is it taking so long the first one went by really fast 6 000 characters yep okay it's working it's working and let's see most of them the total length was let's the longest was 6 000 so that should still fit frankenstein is a little bit long 6 000 so that's this is why we wanted to um compact the story premise as much as possible so let me go ahead and axe this because i'm afraid you know if frankenstein is is going to be too long then um certainly pride and prejudice is going to be too long so let's try something else we will go to the prompt for summary summarize this passage very concisely so we're just going to go full send full yolo and summarize these things that are too the length is too damn high um the outlines that's the word words are hard yes okay so frankenstein is presently 2180 characters so let's using a different prompt make it shorter petty to fool okay hmm it looks like it didn't make it any shorter i don't think it worked yeah one for one output i might have to do this manually uh my brain okay what happens if we turn up the temperature like it's seen it's it sees that it's already a concise summary and it's like i can't what do you want me to do it's already super concise this looks like it might be shorter maybe not nope no shorter hmm this is uh this is a tough cookie all right so frankenstein is 21 um great gatsby's 1600. what was it alice in wonderland was like 1300 so we got to get these under like this one's got to be half as long as it is and sherlock is 21. oh man cheese it removed periods it's kind of interesting huh i didn't notice that oh man i'm gonna pause this and think about this for a second because this is this is tough give me just a minute okay gang whoops sorry sorry got a little bit over excited there okay so what we're going to do is we're going to just throw this out and start over so write a short plot synopsis of the great gatsby um there we go so this one is like just a couple hundred characters so that's not so bad so let's just save that because it tells it tells you the whole story and it's pretty concise i tried i tried uh changing the adjective detailed plot and it got real long let me just show you after a lengthy absence nick carraway returns to long island yeah wait the first time i did this it just kept going and going and going oh i've got the temperature too high turn the temperature down ah this looks like yeah known for lavish parties the object of his affection daisy buchanan yeah so here you can see it just keeps going and going and going and going and going and going is basically telling the whole story let's see how long it is though that's 2500 characters yeah okay so we'll stick with 500 characters we'll just see if this works it's short enough um and then we'll do write a short plot synopsis of mary shelley's frankenstein okay yeah because making making those those summaries that i had shorter just wasn't working wasn't working short plot synopsis of um is it jane austen jane austen's pride and prejudice that's the right book right yeah okay and we're down to 600 characters excellent um of arthur conan doyle's sir arthur conan doyle's the adventures of sherlock holmes no tell me the story it's never easy if it were easy someone else would have done it um of okay so now we will go back to here 400 characters 2100 characters okay um there we go and still okay list the names of the short stories in the adventures of sherlock holmes i don't believe you okay write an extremely concise summary of the following passage whoops extremely concise summary this is not working okay maybe sherlock holmes isn't a good example because it is a compendium maybe that's the problem let's go back to gutenberg gutenberg and go to book search most popular a tale of two cities let's do dracula get it in text yeah oh but that means i gotta start over with all the summaries and stuff you know what i'm just gonna we're just gonna do four i'm just gonna cut out sherlock holmes because this one this one is too difficult goodbye sherlock books sherlock completions and this is all saved in the um in the github repo so i'm not too worried like i can get it back if i need to those are the premises we'll use later the prompts that doesn't matter the summary is sherlock okay i know i keep moving the goal posts on myself all right so now we've got a fine-tuning um data set of 164 samples move on gracefully okay there we go so you see these prompts are shorter they're reliably under 5000 characters now that we've shortened the plot synopsis so i'm gonna let this run and we will be right back once it's done because after this it's just a matter of uploading the fine tune and then trying it and it's done and it looks like it worked so this video is already what 45 minutes long and this was this was much more of a troubleshooting session i thought i was just gonna come in run this cool idea that i had and it was gonna work um but looking at the lengths of the prompts and completions we've got everything from 3 700 characters 43 45 48 uh 47 49 so 5 000 5005 so this should all be well within the the correct size for fine tuning for gpt3 so with that said i'm going to take a break and we'll come back and actually test this in part three because i've got some some homework to do with uh with uh how to actually implement it in an automated fashion so that's what we'll do in the next one um is i'll get this fine tuned model going and then we will actually try and generate a story in part three so thanks for watching thanks for bearing with me and like and subscribe etc etc you know the drill talk later