Fixing "goldfish memory" with GPT-3 and external sources of information in a chatbot - part 2

Key Takeaways

morning everybody david shapiro here we are going to finish our long-term chat bot uh long-term chat with external sources um i was working on it and uh i was just cleaning up a couple things and i realized i was getting carried away so i was like let me just start recording so let me show you what i did before i started recording um just so that you're up to speed uh one thing that i did was i realized that my old lines um and my recent conversation i was passing the whole object which which was a list of dictionaries that included the line and the vector and that's no good in order to populate the actual text block the convo block i needed just the lines so i fixed that right here and then i also fixed it up here where instead of just returning the whole ordered item i returned i returned you know i've i lined for i and ordered so that's a pythons list comprehension um okay so that's what i did there and then i started working on a prompt because what i what i realized last time is that my fine-tuned model so let me show you the fine-tuned model that i use for tutor unfortunately i don't think is going to work as is because it can't really integrate new information so without any without any extra prompting so you see like okay this works just fine user um who was the last emperor oh wait yeah tim that would be marcus aurelius i don't think that's true there we go um okay so then what i did was i wanted i wanted to have the ability to search um an external source so in this case romulus augustus to get information right and so um i wanted i wanted this to i wanted to have the ability to pull from external data sources because sometimes um you know like gpt3 doesn't know what it's talking about like in in this case it just gave me like marcus aurelius because that's a popular roman emperor um so i was just like i probably shouldn't have deleted that because it's like uh no that's actually not true so if we go to here and say like last emperor of rome last emperor of roman so yeah uh yeah so we can't rely on it so this is actually a prime example um of of exactly what i'm trying to fix or account for so what we would do is uh we would take this and so let me just copy and paste this save that and um so then we say uh we go back to my prompt here so i had a few prompts in not the tutor chat but the long term um prompt wikipedia read the following conversation recommend a single relevant wikipedia article so basically what's going to happen is let me move this it's blocking a little bit of my site i hope you can still hear me just fine so we'll pass in our conversation and then we'll ask gpd3 to recommend a wikipedia article um that's supposed to only give one but that's okay um so if we go and search wikipedia for this list of roman emperors so it'll probably just return the first one that's fine um and then if we take this and ask like who's the last one there are quite a few aren't there um constantine i guess that's the eastern roman empire i like how their portraits change very persian um macedonian dynasty yeah oh that goes way up that's the end of the 11th century 80 or ce jesus modern heraklion dynasty okay anyways sorry i'm getting distracted um point being is we like we ask uh um let's do single most relevant wikipedia article list of roman emperors perfect okay so let's actually update our uh our prompt here okay so then that will allow it to answer answer our questions a little bit better right because it gave us the western roman empire but it didn't really differentiate um so it's like okay well is that really the best that we can do and the answer is no all right so we uh we have our conversation um and so this is just relying on gpt3 to just spit out information without thinking about it so in human terms this is like you're relying on your knee-jerk reaction or your intuition to just spit out an answer without really thinking about it but that's not good enough right i'm working on cognitive architectures so how do we make a machine that'll actually think about the answers that it's giving you that means we need recall we need so recall has two primary sources so there's episodic recall so episodic memory or episodic recall is my story of my life if i remember what i did and said that is episodic recall if i remember hey you said x y z to me that is episodic memory um and in many respects uh humans we trust our episodic memory more than anything else because that is the experience that we have lived and so episodic memory is what we're doing when we recall the most recent conversations uh most uh um the most uh relevant uh lines of dialogue so this is an this is episodic recall but then there's um what's called declarative knowledge or declarative memory so declarative memory is i learned this fact right like i've never been to rome i never saw ancient rome i've never met any roman emperors but i know or i think i know that um you know like marcus aurelius was a roman empire that is declarative knowledge so what we're doing here is we're trying to bring episodic memory and declarative knowledge into this cognitive architecture into this chatbot okay now that i've said that i think you understand um why i'm doing this so now that we're all oriented and maybe you knew this all along and that's fine so i apologize um oh darn okay so the fine tune model can't integrate this external information so i'm basically going to need to start over so i just copy the same prompt that i used to create the original um uh fine-tuned data for for tim the tutor um so just imagine a long text message uh chat log between a tutor and a user the tutor is tim who encourages curiosity and uses compassionate listening tim answers questions with thorough academic responses and offers follow-up questions or suggestions to spur curiosity tim maintains a cool professional tone so um basically what i want to do is is figure out how do we integrate um past converse whoops i did not mean to open my spotify how do we integrate past conversations into this um so let's just copy paste the thing that was already started and i'll say um was that east or west rome and then we'll say what we'll see what tim says tim is short-circuiting um oh that was east rome west rome fell in 479 haiti i think tim's wrong anyways i realized that i also need to get out of the fine-tuned model okay so let's go back to just plain vanilla that was the western roman empire west yes okay so clearly the fine-tuned model is not perfect but we also see that it uh it it spit out the answer and it didn't so this was one advantage of fine tuning is that you get the format that you want um the instruct series tends to do the double new line after an answer which i find kind of irritating but whatever we can there's ways to compensate for that okay so um what we want to do is figure out a way to incorporate this kind of information into this so like how do we do that um let's see there are hints added into the chat logs that tim can use to give correct answers tim should use um oh hang on my dog's barking gotta let him in morning ritual okay i'm back sorry about that um okay so tim should use the hints which are not seen by the user to give accurate answers okay so let's see if this works um because if this does work then we can integrate like all kinds of stuff from from the outside world you know obviously right now i'm just searching wikipedia but there's no reason that you couldn't search google or actually integrate other apis so i set up my discord server by the way you should join if if you'd like there's going to be a link in the description and one of the guys in my discord server is working on doing language to api calls which means it if you can if you can integrate if you can seamlessly integrate any api call into any prompt then like the sky's the limit right you can control robots you can control computer games you can search any information you need once you get api calls working so his work is very impressive yeah very smart people in in the discord so but for now i'm just doing it manually by calling wikipedia in the back end um okay so let's change topics a little bit um who is the last emperor let's actually just see what it says here romulus augustus who was over thrown in 476 80. okay so he got that right but i wonder what happens if we because this is this is going to be a mess legitimacy yeah inclusion criteria [Music] okay so it's all here eastern emperor zeno ended yeah all right so let's let's copy this because this actually just has the answer in it so then we'll duplicate that and we'll say okay so this is um and if this was our question whoops that's not what i meant to do there we go so our question and then i think i said passage um for this so tutor chat bot stop opening for the tutor chatbot we need our long term um merge oh that's merging that's not what i needed i needed answer yeah okay so use the following passage to answer to provide a detailed answer to the question so then we'll just say we'll copy and paste this up here [Music] and this is the prompt that i'm going to use in this in this particular cognitive architecture okay so in this case um i this is not a detailed answer um i don't like it i mean it does answer the question but it doesn't provide any context um but let's heck with it let's just integrate it right now because um doing just prompt engineering is going to be like good enough okay so hint and then let's see if that modifies what tim says okay and just spit it out um user okay weird i thought it was someone else tim why do you think that you tell me who was constantine oh okay user what about the western empire gotcha okay so that's not so bad um so what we can do is um by pulling this by by by having this hint integrated um every now and then i think we can get a little bit more interesting results um okay so that works let me let me just copy this and this will be our new chat prompt so we'll start here so double new line prompt chat we'll see if this works okay so now we gotta we we've got we've got our wiki we've got our answer um so that we we generate we generate a question right um generate a specific question um and then we get the answer so answer question which does the same thing and this is actually several steps because we're assuming that we're going to have a really long answer or several answers right because if you look at how long this is this might take several prompts to get to get an accurate answer and so then we'll put the hint in and see if tim can integrate that and this is all just prompt engineering this isn't even fine tuning once i get some more chat data um this could probably be fine-tuned and that actually makes me think i wonder if i should if i should um save it all separately uh well no because i save it all out to a uh to a log file long-term chat so we'll say gpt-3 logs and then yeah that'll be fine because two of these functions see we write it out to gpt3 logs for the completion um so for the chat completion and actually i will probably just get rid of that because we're not even going to use it i don't think yeah all right gpt 3 logs make sure that's the only two places we use it yeah all right so we're not using the um we're not using the fine-tuned model so we can just delete that function that's fine okay uh tokens200 we'll increase this token just in case he gives a little bit longer of a response oh let's do 400 that's fine okay so completion create engine engine i think they're changing this to model actually so that'll be deprecated before too long uh okay so now we've got to integrate all this together so we've got um populate chat prompt so prompt equals all right um convo and then we will also need the hint hint and then it'll end with tim okay uh where did i go here it is all right so prompt chat replace oh we'll do we'll use the same nomenclature block all right so block with convo block there it is and then we'll replace hint with answer and all this will be obfuscated in the back end so that's fine so then we'll say response equals gpthree completion prompt so that will give us tim's output and let me make sure gpt3 completion okay so this line right here what i do is i do two things is i i strip it so that means that any white space that's padding it it removes that so if there's new lines extra spaces whatever it removes that but then i also have this line here which for any um any interior white space so let's say that the output is like here's one sentence and then it gives a double new line and then another sentence i don't like that especially for chat so i say let's replace those double new lines with just one space so it'll kind of compress it all into a single single what am i trying to say paragraph okay so we get the tim's response and then we have to get the vector for tim's response and add it to well i guess first we can just we can just output it but yeah so we'll do vector equals gbt3 embedding response um so then we'll say uh save the output um line in looks like that yeah okay so we get the vector and then um info equals let's see line out equals tim and then we'll do response so that should be good and so then we say info equals excuse me line out and vector and then we save it so conversation pin info so that means everything is saved and then we just print line out and that should be good is this done i think it's ready for testing i am so scared of all the bugs that this is gonna happen okay so yeah we only have two places where it prints um there's a lot of steps in here a lot of places that can fail oh i'm nervous um yeah we do need some debug um so we'll do we'll do that because there's the search part that where it can fail old line so when i put in the first one it's just going to return the original line and that'll be actually no i think i've got it so it'll skip that yeah if recent vector equals that it'll skip it okay so for i and all lines um this will actually fail uh okay so if laying all lines equals um actually no we'll just say less than or equal to we'll just say if it's less than or equal to count return [Music] list yeah that makes sense logically that makes sense so we'll just say we'll just return an empty and empty list and that'll solve a lot of problems um okay so that's fine so we won't have to worry about that so old lines will be empty so then we'll just join old lines that's an empty list that's fine it'll skip over it convo block and then we'll add convo block equals convo block dot strip so that it doesn't have any excess padding um okay so that's where that could go wrong the gpd3 embedding i've used that plenty of times let me make sure i'm using the right engine text similarity ada okay i think that's right because we're just we're looking for similar lines of dialogue we're not doing querying for a document because there are different um let's see open ai embeddings um there are a bunch of different engines that you can use so classification search topic clustering recommendations so yeah clustering regressions anomaly detection visualization so this is what i'm using text search so this is where you're saying like these models help measure whether long documents are relevant to a short query so this is if you want to search like if you have a um if you're if you have a question and you're looking for it for the right document this is going to be a different embedding and that's not what i'm doing i'm just comparing um like apples to apples so that should be fine all right all right so we get our convo block generate a search query um so we've got this the prompt wikipedia this is here single most relevant wikipedia article that seems like it worked um i had to fix it because it wanted to give me a list of stuff that i didn't want title equals gpth3 completion prompt wiki equals fetch wiki um we pass it excuse me the title fetch wiki so we try page wikipedia um uh title so if it does if it if it gives us the right title it's fine otherwise we'll do a search and then we'll just grab whatever wikipedia thinks is the most relevant article great um okay so this will give us one article and then let's see so then we say generate a specific follow-up question to use to query the external information so we say okay so we've got our follow-up prompt um which is this guy so it says given the following conversation and the topic and the topic is is populated by the title okay so we're going to say that the title is the the wikipedia article so we say okay so generate one follow-up topic question to ask about this and so that means that it'll say like okay rather than just say like because like for instance if you arrive at the list of roman emperors okay we're not gonna stuff this this whole thing we want like an actual like last emperor emperor there we go and so like if we say like who was the last emperor um it should like zero in and i think it did actually um so constantine the sixth let me go back to this is that who it said constantine the eleventh interesting okay anyways point being we wanna we wanna actually ask a question of this external information we get the answer to that and that will be embedded in the prompt as a hint and then tim can use that to answer oh boy okay i don't know if this is going to work but yolo cmd cd long term python chat no such oh whoops open ai api key what do you mean i need an api key i didn't copy it over so we'll copy my api key from somewhere else and actually i need a bio break i apologize and then we'll get going okay i'm back um let's see where were we yes all right um i'm actually going to add a debug function def save debug and so then we're going to do label content and we're going to basically just copy the copy this so but instead we're going to do debug and the file name will have label in it and then debug out and we'll just do content okay so what i'm going to do with this debug function label and content i'm going to basically save each of these each of these bits or whatever save it out so that i can see what it's doing all right so we'll do save debug um we'll say convo uh we'll just say convo and we'll do convo block um so that'll be there and then [Music] the title here we'll do save debug wiki title and we'll say title so we can see what it's getting save debug wiki article and we'll do quickie so let's just save what it got and then [Music] save debug we'll say question and then save debug answer um yeah that should be good okay so we've got a few few inter intermediary steps um and i might add some more debug output but that should be fine for now user um hey tim can you teach me about particle physics and this might take a second let me open the debug folder oh nope it blew up save debug wiki article right content it really does not like something that's going on in here because it tried to save the wiki article and it blew up all right so we will borrow our ascii code thing from here it's just something i don't know what i don't know if it's something i'm doing or if i need to do encoding something or other but whatever all right so for the wiki article um fetch wiki so we will do um when it fetches it wiki.content and then we'll do encode and then decode all right but let's see so it got it got that far before blowing up it got down to here where it tried to save the wiki article so let's take a quick look in debug convo okay let's save the convo particle physics and then that blew up that makes sense all right let's close some of these things because they are all excess noise um all right let's go back here hey tim not time can you teach me about particle physics and then it might take a second before it blows up well let's see what happens and obviously this is too slow because that fine-tuned model was much better one of the most fundamental oh that's a good question most fundamental particles in the universe presumably invisible okay it doesn't say quarks does it okay oh hey look at this look at this oh man this is perfect oh oh whoops i forgot to add the the uh particle six is the study of the fundamental subatomic level it's a branch physics that explores okay cool um i needed to add the stop but look at that that was good that was really good okay let's pause real quick because i want to look at the debug and um okay so it got that far and then let's look at the gbt3 chat logs um yeah so the last one prompt imagine a long text message user and so then it added and what i needed to do is add these as stops um it didn't add the hint though did i forget to populate in the hint because that should have been there various answers okay that's fine these include leptons quarks oh it did have it but then there was only one okay i need to actually so first let's skip this if there is only one so let's go back here um so answer question so merge um because we assumed that there would be a chunk um but there might not be right um so if length of answers equals one then we just return answers zero that should be fine okay so we can short circuit it because we only we only need to to split this up um and answer multiple if the document is too long okay so let's fix that and then it did not like it didn't replace the hint so prompt chat hint is there oops come back okay so let's fix that um so that's the chat that's the chat prompt and then if we go into gpt3 logs this is should be the last one and it just goes straight from user to tim this is really weird how is that even possible because that made it here that's right there but it completely erased all right well first let's fix this other thing because the stop is going to need to include user and tim alright so we got to fix that um prompt chat text replace block replace hint with answer and i saved the answer right so let's open debug answer oh i know what i did wrong oh that's what i did okay so the chat block but then i need to have the hint okay and then tim's output yeah because this is okay so the hint was here it just it wasn't labeled correctly i clearly have not had enough coffee yet today but then tim's answer was really good okay i think i fixed all the bugs fix the bug fix the bugs okay so let's control z out of that clear that make sure everything is saved actually let's do uh get status um i want to exclude my api key did i not have a git ignore in this i don't um new and so we'll do open ai api key dot text and save this as dot git ignore do not save it into the debug folder save it here hit add dot get ignore get status okay get add all get commit am should be working get push it's always like i don't trust it when it works the first time i guess it didn't quite work the first time but like i saved all the debug and you saw like right i'm not imagining things like particle physics it gave me a wiki title here is the actual article and here's a question what are the most fundamental particles in the universe which is great and here's the answer um which you know it i inappropriately summarized it but like that worked that worked okay fix the prompt chat um print line out and then we just go back to here so user says input user let's add a space there um that should be good because that's tim and there is the space okay good good good good um cls chat um can you teach me about particle physics give it a second it'll go through and churn everything in the background we'll wait for that to finish and then we'll check on um you can see here it's going through um particle physics is the studies of the properties of particles such as electrons and photons cool okay so let's see what is the difference between a particle and a wave that's a good one a particle is a small object of individual mass that's fine it did not make it into the final output but let's take a look can you teach me about particle physics so it was in there but tim decided not to use it interactions between particles such as the force of gravity so i'm a little disappointed because tim was supposed to use this information um but that's fine [Music] um hmm yeah i'm actually i'm actually kind of not happy about this because gpt3 absolutely should use this information so let's do a little bit of prompt engineering okay so we have this and we say tim should use the hints let's see so let's just say tim is an academic tutor well here the following is a chat log between tim and user tim is an academic tutor [Music] who uses the hint to provide robust answers to um to the user excuse me why do i have hiccups um and then we can uh then we can copy the rest of this um let's see tim encourages curiosity and uses ask follow-up questions to spirits to spur curiosity and maintains a cool professional tone let's see tim encourages curiosity and uses compassionate listening um yes tim maintains a cool professional tone okay so if we have the hints part up front let's see if this does a little bit better a lot of electrons the nucleus is held together by strong nuclear force electrodynamics well okay still didn't use the information from the hint but that was a much better response i like that let's run it again just to see what happens i almost think we just need to like just spit this out like what if what if the hint is the actual response um and then we don't actually use the chat at all like why add another step when we've got a good answer um okay let's do this again i might each of these answers is very very different which is interesting that's fine um okay okay yeah yeah this is fine um oh i well you know maybe i shouldn't be so unhappy about this because when i look at what this answer is so let's take a look at what the actual question was right so the um the wiki title was particle physics the question was what is the difference between a particle and a wave so that's kind of inferring a question that we might ask but that's not actually what was asked right and so then gbd3 maybe gpt3 is smarter than me this morning so gpd3 is actually answering this question and kind of ignoring the bit about like a particle but that is still priming it so this is i think is actually really good okay so let's just update the chat prompt with this because i like this i like this better whoops that's the script not the actual prompt prompt chat um so let's save that and i shouldn't need to restart it because i didn't change any code i just changed the prompt and it's going to be reloaded each time um okay there's a study of particles such as electrons and photons does gravity affect particles it's thinking obviously this is not fast enough to be useful yet but this is just proving out the idea that um that you can pull from multiple sources such as long-term memory which we don't even have the conversation long enough to do that yet and as slow as this is it might not work gravity is a force that affects all matter um okay cool cool cool is also what keeps the planets around the sun particles are affected by the force of gravity just like everything else good answer let's see what what that looked like in the back end so let's see wiki title particle physics can you teach me particle physics we already pulled that question what are some of those most famous findings of particle physics okay subatomic particles does it mention gravity no it doesn't okay so none of that made it in which is that's fine um and then let's look at the gpt3 logs for the final output you know i'm almost i'm almost more impressed with this output than than the actual chat output how long is this video we're already at 45 minutes all right there's gonna be a part three because there's something that i did not anticipate which is that all the thinking going on in the background is actually better than the chat output um so i'm going to stop here because i'm clearly on to something it's just not quite it's not quite there because i look at the i look at the precision of this and i'm like this is good and then i look at like how watered down the actual output is and it's it's not bad but it's also not the rigorous academic output that this is so i think just be by the fact that this prompt says like oh hey this is this is a chat um gpt3 this model might be overly aligned and so that it's watering down the responses and not giving me what i want so i'm going to stop here i'm going to save this code and then we'll come back to it because something is interesting is going on and i'm really impressed with what's going on in the back end and i'm glad i saved that debug but we're not quite there yet so anyways thanks for watching um i did want to say that i have a new patreon a bunch of people suggested that i should do discord and it was immediately popular a bunch of people have now suggested that i do patreon so that that way y'all can support me and who knows maybe one day i'll have enough supporters that i can do this full time but yeah so jump in on the discord jump in on patreon like and subscribe and let me know what you guys want to see next thanks for watching i'll talk to you later