Fixing "goldfish memory" with GPT-3 and external sources of information in a chatbot - part 1

Key Takeaways

morning everybody david shapiro here um i'm doing the rare recording two videos back to back because um i woke up with inspiration woke up at five in the morning and was ready to go okay so the reason that you're here um this dude uh ravi he asked a good question and he put it so succinctly that i need to borrow his language so thank you ravi for asking this question um he's not the first person to ask this question but he put it very succinctly so basically he says he wants school kids um wants to help school kids with a conversational app that is that is friendly um so that the chatbot we've got worked out um i have a previous video um where my tutor chat bot works perfectly fine here's an example fine-tuned model it's got 600 samples it can handle it can handle adversarial students it can handle emotional issues it's very kind it will adapt to whatever level the person needs if you ask it about quantum physics it'll answer quantum physics if you ask it about colors it'll talk about colors it can handle all adrian's all needs so i've got that part covered but um there are two primary issues um one is uh gpt3 is static um it was trained you know i think the the the update was it was updated with you know information circa 2021 already aging so how do you integrate new information big issue and a similar issue it has to do with memories and i love this term goldfish memory it doesn't remember anything long term goldfish have short memory um there is some science out there that goldfish actually have longer memories than you might think but the point is is that you know it's got a tiny little brain it can't remember long-term things so how do we solve these two problems that's what we're doing today so i have created a new repo um it's called long-term chat with external sources so let me just show you that repo real quick sorry i don't know why i'm so oh i've got some uh some ginger tea which will ow that's hot i just spilled hot tea on my leg i'm struggling this morning okay long-term chat external sources so we've got chat figured out that's not a big deal right you know you fine tune a model you've got a chatbot okay whatever but let's solve these other two problems where we've got um where we've got long-term memory and external sources so there's two primary ways that i'm going to tackle this um oh here's the here's the repo for the chat bot if you need to um see that um so this is how i how i created the the fine tune for that and then the this is the the multi-document answering so this is how do you how do you query information from an external source that's this um what i'm going to do is instead of querying a document is i'm going to actually query wikipedia because wikipedia is a live source of external information and it's searchable um so that's what we'll do and yes so without further ado let's jump in um i got started um i know that you guys usually like to see the whole process these are just copy pasted mostly from from the um from the from from this one from answer questions dot pi because you know i i solved some problems here um like search indexing that sort of thing um so i just copy pasted the functions so if you want to see how i did all that go watch the other video let's get to new stuff though um okay so first of all if we're if we're doing a chat um if name equals main um so we'll do uh converse uh let's see yeah conversation so we'll do make sure i'm not using that variable anywhere else okay good so we'll have one list to accumulate the entire conversation um that's probably the only global thing while true let's see input is a function i want to say like user says input we'll just say user so that way it'll say like okay this is what you say okay so line in equals user and then we'll do percent user says okay so this will this will be basically populating this so user says hey tim can you teach me about ancient rome so i just i brought this up one thing i noticed and this is kudos to openai this is something i've been complaining about for a while is that it tends to unload your fine-tuned models this has been sitting here for 10 minutes and look did matrons ever have sex with gladiators this is a real thing that happened in ancient rome let's see how it handles it oh yeah it so i trained it not to engage with sex um so let's see if we can get it to yeah he's not gonna talk about it okay um that's fine because this is this is meant to be pg or g rated um did gladiators ever fight lions he should answer that yes okay they're called venetiones oh cool um so this has been sitting here for like 10 minutes and you saw it was ready to go instantly so good job openai thank you for fixing that that was the most annoying thing ever was that our fine-tuned models were not ready to go okay so fine tuned model's ready to go at all times wonderful um my spanish teacher would used to say todo el mundo so all the world i guess that's a that's a turn of phrase in spain where it's just like everything is good let's go so todo el mundo um or maybe that was just something she said i don't know okay brain reset what are we doing okay so we have this and then what do we need to do well it's not just that simple once we have a conversation going so what we'll do is we'll say conversation dot append line in okay so we'll just start this conversation starts we put something in excuse me um and then what do we do well first we need to search the rest of the conversation so what we're going to do is search indexed so we're going to search let's cut this down a little um okay so count 5 that's going to error out the first time um because it's going to be too short but oh here's here's how we'll do this yeah yeah gotcha gotcha gotcha okay so search index the data is going to be the conversation um oh i need to actually keep track of the whole the whole thing hmm okay okay so the conversation will actually need to have the embeddings with it yeah vector equals gpt 3 embedding and then we'll do line in and so then we'll say info equals um no that needs to be a dictionary so line of conversation equals line in and then the vector equals the vector okay and uh oh in case you in case it's not obvious i prefer the term vector because that is mathematically what it returns but in the context of a large language model this vector is technically an embedding or a semantic vector so an embedding is a vector that has semantic meaning but a vector is just mathematically this is what it is um okay so we will we'll store so the reason that we're doing this is that every line of conversation so let me come back over here every line of conversation is going to have an associated embedding or vector which means that this whole thing will be searchable um okay so conversation append dot info so we're just going to accumulate the entire conversation right here and then what we do is we i guess we actually won't need this because it's gonna well yeah okay so we'll search we'll search for the most for for similar so i'll need to modify the index also i need to move my microphone because it's blocking a little bit of my field of vision hope you can still hear me if not deal with it okay so we get our line in and then what do we do we search for we search we search we search um old lines equals search index and so we'll do we'll pass info so that's our most recent one we'll search we'll pass conversation and we'll leave it at that i'm not going to pass a variable for how many to return um we'll just take that out we'll hard code that doesn't matter um so we'll do recent and then we'll do um all lines so what you're supposed to do is if you if you use a variable in it in a function especially if it's a list you're supposed to use a different name um otherwise because gpt3 or not gpt3 python python automatically has um lists as global variables i'm probably saying this wrong but like you can modify the contents of a list from any function but if it's local to um to a function it won't be modified so like if i have a different name even though it's the same um conversation like i'm not going to be modifying and i'm just going to be reading from it but anyways this is best practices i think okay so what we're going to do is where did i go i lost my search index there we go okay we don't need the vector because we're already going to have that so for i in all lines score is similarity so we're going to do this vector so we're going to do recent vector and actually what we'll do is if yeah so if these are the same then we skip our current line so if i vector equals recent vector continue and that should be the last one um technically you could probably uh you could probably just skip the last one but i don't want to make that assumption because what if what if we're searching for whatever reason we we reuse this and we're searching elsewhere in the conversation i don't want to assume that the one that i'm searching is always going to be the last one because i can just imagine someone saying why don't you just skip the last one because i don't want to make that assumption we might be searching based on an older memory okay so score we get the score and then um what do we do yeah okay so we can just do um the content i'll change this to line equals and then we'll do i um line i think that's what i called it right yeah we've got line and vector okay so the line of dialog and the vector search index okay so line equals line and the score equals score okay sorted scores sort by score um reverse is true so it'll be top down and then um [Music] i only want to return the top five so what we'll do is um when we first start the dialog it's going to be shorter than five so we'll just return um you know we'll try to return the top five but if that doesn't work actually here's here here's what we'll do we'll do ordered equals ordered and then return ordered because i'm afraid that if it if with the previous one if i if if it has the return at the same time that it's trying to slice up the list it might just break entirely um but then accept will just return ordered um so because if it if this doesn't work it'll air it'll bomb right there and then we'll just skip to this and just return it as is um so this is another paradigm in um in python where it's it's better to ask forgiveness than permission so rather than test something you just try accept um you do need to be careful oops one of my i put up you probably see those hexagons in the back i put up um sound dampening things let me put that back up it's going to bug me um but yeah i put up sound dampening because you probably noticed it sounds better there's less echoes okay we're back sorry about that um someone online said that the the best way to stick stick those panels up was um was blue painters tape which seems to be working because it won't it won't peel any paint off but one just fell off so i might need to get some better better tape or some two-sided tape anyways where were we search index okay yeah so better to ask forgiveness than permission you do need to be careful um if you do a try accept loop sometimes uh it might not have the results that you think and you might gloss over problems so that's why when i do it when i do the tri-accept loop here i actually give it output because i you know sometimes the assumptions you're making might be wrong and you need to see see why but in this case there's like only one thing that could go wrong here so this is probably a little bit this is more of a sure thing right because there's only one one action hat happening okay so we've got that old lines equal search index info conversation um you know honestly i don't even need to pass the scores back i don't care um but whatever okay so we get an input we get the old conversation so this will find anything that's most relevant we'll find the most relevant five lines of conversation this number is arbitrary um [Music] where did it go actually you know what i'll do i'll do um i'll add count back in count equals five um and then we'll just do that so that way if you reuse this code you can just pass however much you want um because imagine that this conversation gets much longer and you wanna return like 50 right because you can easily fit that into actually yeah let's let's do a little bit longer let's do let's do 10. um so we'll get the 10 most relevant lines and then okay so the whole conversation is going to be accumulated here the old lines will be there um and then we'll just pass in 10. um let's cut off um let's see latest conversation equals uh let's see actually no here if len conversation [Music] no um recent conversation equals uh here do if there we go if len conversation is greater than 30 so we'll we'll say recent conversation equals um equals conversation and i believe the thing is negative 30 that let me make sure i'm doing that right python l equals list range 0 to 100 l okay and then l if we do negative 30 colon that's just the last 30. okay perfect um and then do laying just to make sure that that's right yes okay um yeah so what this does is if our conversation is getting too long we don't want to pass the entire conversation right we just the last 30 lines that's more than enough because you look at this this is 1 2 3 4 5 6 7 8 9 10 11 12 13 14. so this is 14 so this is less than half of that and obviously this is enough context to know what we're talking about so 14 lines of conversation is plenty 30 will be way more than enough that is a pretty long conversation and then on top of that we're going to have the 10 oldest lines right or the 10 most relevant older lines um okay so now what we've got we're searching our index we're searching the conversation we're getting the most recent conversation so this is 40 lines so now what what are we missing we're missing external data okay so um convo block equals so we'll do um n so this is newline dot join and then we'll do old lines so this is going to just be the old old bits of the conversation plus um i probably need to do a new line between there and then plus and we'll do the same thing and we'll join recent conversation so basically what this will do is it'll combine it into one chat block right so it's like okay we've got this um so now what we've got our convo block what we need to do now is we need to do a prompt um so prompt and just copy a line so prompt equals open file and we'll just do prompt um keyword replace block with convo block okay so now we need to do some prompt engineering so basically what we're going to do is we're going to i'll use this as an example so we're going to say read the following conversation and generate um actually and let's see recommend a single relevant wikipedia article title okay so then we'll do um most relevant wikipedia article title so let's see if this works um i want to keep this up so i'm just going to duplicate this tab and we'll go to davinci instruct okay let's see what it says romulus augustus cool and then let's go to wikipedia wikipedia why is my uh i've already had coffee romulus augustus bam got it okay um so what we can do then is uh we'll we'll query this we'll get it from wikipedia um so this worked really well it's always nice when when your intuition is like spot on um okay so we've got our prompt uh let's go ahead and just do block and we'll call this prompt what did i call it keyword actually let's call this wikipedia because that's it's very specific it's specifically let's write a prompt to get wikipedia prompt wikipedia so this presupposes that that gpt3 will know the correct wikipedia article gpt3 was trained in part on wikipedia so i don't see any problem there's another way you can do this so let me show you this is if you want something that's more open source or more open-ended what you can do is actually you could feed this into a um into a question generator um so let's see where was it my nowka question generator wow is it really that old um questions yeah so i have a question generator which basically what it does do i have an example yeah so i have an example where you um you feed it anything and it'll generate a list of questions so natural language questions are going to be much more open-ended in terms of what questions should i ask about this you could do that but gpt3 is also just really great at like [Music] let's see read the following conversation and um recommend let's see uh a list of google search queries to learn more okay so and then we'll say like list of google search queries um so you do dis and it like so let's say you want to query google instead of wikipedia um you could do this and then just do google searches i'll go ahead and save this prompt just so that way you all can see it so um we'll do uh do block and then we'll remove that and save this as prompt google okay so now you guys can see um you can experiment with this if you want to use the google api instead of wikipedia um either way both serve as an external source um it's pretty trivial as you see like you just ask gbd3 to generate a list of google queries and you're done okay um answer gp oh wait no no no no no um oh right yeah so then we say um title equals gpt 3 completion prompt i think that's all i need to pass that's a chat completion gpt3 completion yes we do not need that many tokens let's say like i'll say 500 because we'll use we might use this for other things now let's say 200 nice and short [Music] max tokens for the chat let's say 100 we don't want super long conversations temperature of 0.7 that's fine frequency penalty let's take that out um that should be fine okay uh yeah so we've got our conversation set um we've got a title and then let's take a look at the wikipedia pi pi so if you want to get the article you just say here's our page so i'm going to assume that it works actually no we'll say um wiki equals let's do let's let's not make any bad assumptions let's do fetch wiki um and we'll do title okay so in this function because when you're when you're searching for something you don't always know if it's going to be if it's going to work so def fetch wiki title and so what we'll do is we'll do try page equals make sure i get the syntax right wikipedia.page title so if it works we'll just pass this back return page if not this will error out and it'll say i don't know what you're talking about it'll say it exactly like that too except if it doesn't work we'll do search equals wikipedia dot search title and so you see the syntax for that here where you can do a search and it'll return a list of possible um articles and so like in this case it's like barack so it's like do you mean barack obama presidency of barack obama barack the brandy barack the given name um i have no idea how i'll probably just pick the first one so then we'll do page equals wikipedia dot page and we'll do search zero so i'll just assume that like return page um i'll just assume that it it it returns you know the the first one is going to be most relevant like i said this is i'm just showing you the proof of concept um okay so then so we've got the wiki that will return it oh i also need to import wikipedia import wikipedia and then do i use os anywhere import os i probably do okay fetch wiki okay so we get our relevant wiki so then what do we do with it well we found romulus augustus this is way too much text right this is way too much to um to uh uh put into a gpt3 prompt so what do we do well i'm glad you asked that's why we have this the the the document answering thing um so you know what let's just go all out let's let's let's hit this um i was gonna do something simple but no let's uh let's go let's go whole hog um all right so we've got the uh the wiki content um let's see what is it if we go here we do if we want to look at the whole content where was it summary search page content that is okay so we just do dot content so that'll give us just all the text from that particular article and so then what do we do well i'm glad you asked so let's go back to our playground and we'll say um read the following conversation and um and and write a list of uh of well we're asking questions because basically what i'm thinking is we want to we want to query this document right um for interesting facts right life in exile um actually like the questions we should ask are not necessarily obvious from this so we get romulus augustus um how should we handle this because if we let's say this conversation leads to this and i don't know what i'm asking so federation um because if we if i use my multi-document answering thing and we query it like we got it we got it we basically we have to like we have to be there has to be something we're looking for unless i just want to summarize it but i don't know if i want to do that because i want to summarizing is good and all but if you get if you answer a question you get much better but but the problem is if this is a this is a chicken or the egg problem or putting the cart before the horse because if you don't read the thing and you don't know what the information is you don't necessarily so there's there's what i call a naive question so like if if you don't know anything about something you ask naive questions right and naive question is just like um well here's a here's a perfect example um who was the last emperor right i had to i had to already know that rome had emperors right so that's that's a that's a semi-naive question um it's like okay so romulus augustus whereas if i ask so this this is a super not naive question right um because i knew that there was a virtuous roman man that there was this concept in ancient rome but how how do you ask the question to even get to the right question so we need like meta questions but gpt3 is trained on all this stuff so maybe it could i don't know i wonder i wonder i wonder um follow-up questions to search the internet um before yeah that's fine we'll just do search the internet because it's like um list let's see list of follow-up questions to add to the conversation and then let's just see what it does what caused the fall of the western empire oh wow okay what was life like for the average person in rome this is great um it's not really staying on topic though because if you ask these questions it's not that's not going to be information whoops that is that is necessarily here let's see given the following conversation and the selected topic so we'll do romulus augustus write a follow-up question um to ask uh no because if we look at this like what is the kind of information that i want to extract from this right like the summary of of this dude's life or what did he do write a follow-up question to ask follow up question to ask about romulus augustus let's just see what it does that's not a bad question okay i have to run to the i have to take a quick bio break so i'm gonna pause and then we'll be right back i think this is what we'll go with all right and we're back where were we oh yeah okay so we've got the follow-up question so because we we we have identified where do we go um so we look in here um we identified we use we used our wikipedia prompt to to get a particular topic right so we get that um we then we get we we i wrote a quick thing just to go fetch an article so we'll get it and then what we'll do is we will do a follow-up question so we've got a topic given the following conversation and the selected topic so we'll do [Music] topic and then we'll do block and we'll do one follow-up question to ask about and we'll do topic okay and we'll do prompt follow-up okay so then what we do is we go to here and so we say control d for duplicate and so then we'll do prompt follow up and we'll replace the block with convo block and we'll also do replace topic with the title and so that what that'll do is that'll take this and populate it with the way that it looked up here right so it'll just say okay let's populate in the conversation block and ask a follow-up question so then once we have the follow-up question we can then use the same logic that we had here to um to ch brain come on we're going to index the content of this wikipedia article and then we're going to um we're going to search it okay and answer the question so then what we'll do is we'll do answer equals um we'll do answer question and we'll pass in the wikipedia article oh wait we actually need to do question and the question and i think that's all we need to do because the question will be generated here um generate a specific follow-up question [Music] to to um to use to query the external information and then we'll do um generate a uh a search query to find external article in the wide world okay add a couple comments just because this is becoming a really big block let's see get user input and vector and then we'll say search conversation for um for previous relevant lines of dialogue okay that's good so we'll get the answer for answer question answer question and so then we'll pass in the article and the question so then what we'll do is we need we've already got text wrap so we'll basically do the same thing we did here but i also need to open whoops no that was that was the correct thing build index so this is super super simple um basically all we do is um we cut this into chunks so when we come in here we'll say chunks equals text wrap and instead of alt text we'll do article and then we'll do it in let's do smaller so we'll do 2 000 characters and then for actually i guess we don't even need to index it because we're just asking the question so this will be a little bit simpler because the reason so in um in this one the reason that i did it the way that i did is because i was basically saying like okay let's go ahead and index an arbitrarily large thing because if if you're if you have like 150 entries or 150 000 entries you need to be able to search that database um and then answer the question but in this case we're already zeroed in on one article so we don't you know we don't need to answer the whole thing um so in that case what we'll do is we'll just go ahead and skip ahead to answering the question [Music] and we'll probably just copy this this prompt let's see which was it prompt answer yeah use the following passage to give a detailed answer to the question that's great so we'll just save as and we'll come back to our long term chat and we'll call prompt answer okay and so now that should appear here prompt answer okay and close that because we don't need it and then 4 chunk in chunks prompt equals open file replace passage with we're going to replace that with chunk and we're going to replace query with question okay and to keep it consistent let's name it the same thing so we're gonna do question is question um so basically what we're gonna do is we're gonna cut up um yeah use the following passage to give a detailed answer to the question oh actually i guess i probably should search it because um the answer might not be right like fall of yeah see also fall of the western empire yeah okay because the answer might not be in it um so let's add something use the following passage to give a detailed answer to the question if the answer is not present um in the passage just write not present okay detailed answer or not present all right so let's test this real quick so what we'll do is we'll come here and we'll say okay so the question is and um oh darn hang on i need this question wait give me the question okay so we copy the question and then let's go here so we'll say okay and question okay so this is our question what made romulus augustus the last emperor of the roman empire so the passage we're going to have a small chunk and actually i think i think i was wrong i think we should go we should make this a little bit bigger because many wikipedia articles will fit and i'll say 4000 tokens is 16 000 characters so let's say like 10 thousand because if the whole thing fits great if not we'll skip it and then let's go back here and just grab a chunk um so like name here we'll just grab the summary and see how it handles this okay passage detailed answer or not present okay hey is that there yeah all right so the answer was there let's see if there is a section that it doesn't have it ancestry and family so let's just grab this section because it looks like the answer will not be okay because we wanted to be able to say not present he was the last emperor of the roman western roman empire because he was the last emperor of the western roman empire wow the floor is made of floor thank you okay so it didn't really work there um but at least it didn't it didn't give me too much information um so that that's not that's not too bad okay so then we'll say answers equals list because here's here's a fun here's a fun trick so then you say answers dot append list whoops no not list answer and then what you do is you have one final summarization step [Music] so let's get this and i'll show you what this does [Music] all right so we got that answer and then because oh darn okay that's fine um i like i liked it's uh it's original one better let's see if i can do was it ctrl shift z there we go okay so we'll take passage and we'll take um what was it his legacy or no it gave me a good answer from just the introduction okay so we'll do that and then okay so it gave me a shorter answer that's fine so we'll accumulate this here so let's let's ask this question of a couple more spots so we'll do ancestry and family just so that way we can i can show you that like if you have different answers then or or different things the gpt3 is still pretty good at like summarizing it together yeah so that's not even remotely relevant okay so that's fine arrestees question what made a romulus yep okay so then we'll do last one in rain so we'll just basically what i'm doing is i'm pretending that um that this article is much longer and that it's giving me that i needed to split it up into like four different chunks in order to get this answer okay this made the last emperor okay okay so there is relevant information in each of these sections that's great okay so then what we do is we take all this and we say merge the following answers into a single paragraph um the question asked was and that should still be oh darn let me just copy paste this whole thing okay merge the following answers into a single coherent paragraph the question asked was so then we'll do oh i have another thing falling off let me pause real quick okay and we're back all right the question asked was okay and then um various answers so then we'll come back to here and grab wait where did i accumulate those answers here we are various answers and then let's do this all caps various answers and then um merged answer actually let's say merged paragraph see how that does and gaul however rsd's incident marched on rivena the capital of western proclaimed romness and napostead was only a figurehead with orestes running much of the imperial administration interesting okay but it doesn't mention auto asser okay well whatever that's fine let's go whoops let's go to the wikipedia article and see lester captured rivena killing arrestee's deputy okay all right whatever okay so we've got it broken up into a couple sections we summarize it um here i wonder what happens if um that should be fine yeah that'll be fine okay so we'll do one last thing we'll save this as answers question and then we'll do prompt and we'll call this prompt merge okay so then we're closing in on complete i think all right so we got all of our answers and so um answers equals uh no answer equals we'll say space.join answers so that just merges it all into one block and we'll say prompt equals well here we'll just prompt merge and it's we have answers and question so replace question with question i already had that um and then we'll replace uh answers with answer i think that's how i did it yes various answers into a single coherent paragraph the question asked was okay that should work and so then we get that gpt3 completion we do the encoding here so um this was a thing that i started adding because when especially when you use external sources sometimes there's characters that gpt3 does not like but if you convert it to ascii and ignore errors and then decode it back then gpt3 seems to be happy and then return answer okay so we've got we pass it a wikipedia article we ask a question we break it into chunks if necessary might not be necessary we ask questions and then pass it back okay so now that we've got this it's like okay wow we've done a lot now what um so let's copy this and we'll copy this out to here and let's go back here and we'll say background info user um what caused the fall of rome time why didn't you give me an answer there we go but you got to include this hmm because we've got this information which is good so part of the problem is and this this might not work i might have to exclude this part is because this model was not fine-tuned to um was not fine-tuned to incorporate other information ah let's see um oops user i wonder if instead of using the fine-tuned model i could just what if i just exclude the fine-tune model and just have this information it would require it would require um reworking this fine-tuned model so i might have to stop here and and split this up because this fine-tuned model does not know how to incorporate additional information darn but that's okay we're already at almost an hour so it happens we'll stop here we'll leave it off where we need i need to create another model that will incorporate background information external information into the chat because this chat model is already pretty good but it's missing a couple things all right thanks for watching