FOCI LLM Users Group: "Beyond Autocomplete: Instruction Following & CoT Reasoning in LLM Agents"

Key Takeaways

brief introdu for you all right good evening everybody uh welcome to the third installment of our bosai llm user group meeting uh this will be the final one of this semester um today our speaker is Abraham Sanders he's a thirdy year PhD student with the cognitive science department who is I believe his research is about creating dialogue agents um and today he's going to be talking to us about one of the most important um things that makes llms like chat GPT actually good which is instruction tuning and also talking about Chain of Thought reasoning Nows awesome thank you so uh thank everybody for coming um and I'm glad we have enough pizza for everybody this time um can everybody hear me okay to am I close enough to the mic yeah y all right so today we're going to talk about um llms right we're going to talk about how or what makes them like Aaron said what makes them effective at um at doing useful things such as um answering questions being a personal assistant all the things that we sort of take for granted in a post chat GPT World um this presentation sort of goes through a little bit of History of how we got to where we are so um but there are for the more you know if you came here for for code or for practical things there's there's a list of resources at the end that you can U once the the siid are shared that you will uh be able to use so okay so to start off um this is a tree of sort of a family tree of llms going back to 2018 and going going towards the beginning of 2023 and you can see that there has been a tremendous amount of work gone into improving these systems both from the commercial and the open source Community um and the if you could see the second half of 2023 which is absent for this tree I think it would be just as big as the rest of the rest of this entire tree by itself so what makes these llms so useful that everybody wants to work on them and everybody wants to build new ones and and and uh everyone wants to fund them um let's you know take take a look a little bit back where where all this came from so llms were originally born from the natural language processing research community and their capabilities have continuously evolve as such so in 2013 we were we we were investigating the question we as in the the NLP research Community we were investigating the question of how do we represent text prior to then tech text representation methods were restricted to things like um NRS and uh and and um word frequency vectors in 2014 the question shifted to um how to generate text now that we had efficient dense representation such as word Toc once we figured out how to generate text um using using neural networks the focus was on how to excel at specific applications such as machine translation natur language inference summarization and so on 2017 2019 sort of was an inflection point there there was the introduction of contextual representations of language um and um scalable transfer learning which which we'll talk about and that enabled um our our language models to be able to tackle General MLP applications all within the single same model and moving forward we're now unlocking General multimodal applications so let's take a look at more specifically what some of this stuff means so how to represent text um means how can we create a numeric representation that we can feed into a machine learning model such as a deep neural M represents what a word means and this image over here is a very classic image um that is uh that that illustrates what word Toc does word Toc encodes a word into a vector space a high-dimensional dense Vector space such that words are compositional like if you take the definition of King and subtract the definition of man and add the definition of woman you might get somewhere close to the definition of Queen so the this ability to to create numeric dense representations and feed them into neural networks allowed um text generation to happen through repeated classification classify what is the the next word that should come next after a sequence of um of of previous embeddings of of words in your context and this was done in 2014 originally using what we call the sequence to sequence framework um generally built with recurrent neural networks such as lstms and grus uh which we will not be talking about today so originally these language models built on top of rnns were uh able to tackle specific you could fine-tune them to tackle specific problems and in this case you have the problem of machine translation where you have let's say the translation of a sentence in English to a sentence in French and the model will learn to attend or to um to align the vectors representing the English words to those corresponding to the French words and you can see here that in English the European economic area is sort of in a different order that in France laone economic sorry if I'm butchering this he European and um you can see here that the alignment graph is is uh is off and this is something that the the the neural network had to learn from example by by by doing this language modeling process then with uh Transformers We Came to have this uh scalable transfer learning um we would retrain one of these Transformer language models from web scale text so um gpt2 and Bert are the great examples of that and then um you would be able to fine-tune them for specific tasks with a tiny fraction of the amount of data required previously to train these the RNN and then um gpt3 gp4 llama 2 these are examples of modern large language models which are capable of M many different tasks and um and they sort of they excel at all of them and we're going to investigate in this talk how that works why do these models excel at all of these tasks whereas previously model um language models were only capable of single tasks that they would be fine-tuned for specifically at a time um and now moving forward um we have the ability to to model also images audio and video and to generate them using language models this is something that is I'm my my own research and my my focus of my PhD is is focused in this area but we won't be focusing on that today and and since chat gpt's release it there there's been a huge explosion of of development ecosystem around these large language models um we have Lang chain we have llama index we have the openi developer platform hugging face um we have projects such as Auto GPT which um aim to take the um these very capable General language agents and compose them to create Crees solutions to problems that would otherwise be impossible to tackle with AI so we're going to look at like what was the fundamental jump in the llm capability that led us from where we were to where we are and there are several contributing factors that we're going to focus on for the next uh 50 minutes are um the the fact that we now have scalable models so we have very large very very uh very large models 175 billion parameters that was gpt3 people speculate what GPT 4 has that has over a trillion parameters um we have instruction tuning we have alignment through rhf we have um Advanced Pro prompting or prompt engineering that's um that that allowed us to elicit behaviors in language models such as reasoning and we have retrieval augmented generation which allows we us to factually ground llms in data yes alignment is I'll talk about it in a little bit but it's basically we we want to align the behavior of a language model with expectations that humans have about how the language model should behave yeah um we have then General generalization of of knowledge grounding which we call tool use um and we and to bring it all together with a perception action Loop that allows us to take all of these techniques and unleash them on complex compositional problems so before getting into those things we'll do a very quick recap on how llms work um if if you are doing research in this area or even if you're a proficient user you probably know all of this but um I put it in here because um the the talk was geared towards a more General audience um so you know be bear with me if if this part uh is something you already know so llms are essentially causal language models their job is to take a piece of text and to predict the probability distribution over what word comes next so for example here there's a story um that involves um the phrase he opened the door and slowly stepped inside immediately he felt a sharp pain stabbing through his foot for the floor was covered with and then the next word could be pepperoni right but that wouldn't make too much sense because pepperoni wouldn't hurt your foot if you stepped on it so the model has to um come up with a probability distribution conditioning on the context of what's going to come next so for example the word burning then it will take that prediction and append it to the text rinse and repeat so the next word might be hot as opposed to burning pizza or burning coal right or it could could be coal but burning hot um burning hot dogs burning hot ice no maybe burning hot lava is something that would that would be on the floor and hurt your feet right so keep on going taking the predictions moving them into the end of the context and rinse and repeat and that's what an llm does fundamentally this is called Auto regressive language modeling because we are um predicting we're using our own predictions as input as we go to do this input text is first tokenized into words and subwords so for example the the lauram ipsum over here would be potentially tokenized into something that looks like this um low REM then um underscore I IND indicating that a space comes before it sum indicating there's no space between I and some because it's the same word right so we break words down into smaller smaller linguistic units that may or may not correspond with an actual word but represent um ways that that we can we can we can assemble language out of um out of out of these you know smaller pieces and the reason it does this rather than just using words like word Toc is because it addresses the out of vocabulary issue what happens if I make up a new word and I want to feed that into the llm um if that word isn't in the vocabulary it would previously just be mapped to a general unknown token but here the smallest units of a tokenizer will typically be letters and unicode characters so it can tokenize anything um these are mapped then to um indices in an embedding Matrix which map to dense vectors which are fed into the model this is how we take language and we convert it into our numeric word Tove like representations and feed them into the llm just like WTOC these representations are also have compositional prod properties so these representations are at the at the base level of the at the lowest level of the neural network um are fed in such that they have context free semantic meaning to the point where things like trains buses taxis cars and bicycles would be clustered together in the in the vector space um tutor student school and University might be clustered together in in the vector space they don't these representations are are context free because they don't depend on the context in which they're used which is what we'll see next um so Auto regressive language models would come in encoder decoder or decoder only setups um an early early work would do um basically would basically take an RNN and um and they would such as this example of an lstm network over here and it would feed in just like we just described the words one at a time and predict the probability dist distribution over each subsequent word given past formally it's the probability of the entire sequence given its prompt which is the the um product of their conditional probabilities modern autor regressive language models are still doing this but instead they use the Transformer architecture um I know um Aaron had uh had introduced this in his first speech Transformer architecture um has benefits over RN because they're parallelizable and scalable um Beyond what's possible with r with uh with rnns um and specifically the recurrences are are replaced with positional encodings and self attention what this means is that instead of feeding the hidden state from the previous time step into the current time step you would feed in a another Vector that indicates what the time step is um and then and then uh information would be shared across time steps through an attention mechanism which is essentially a query mechanism querying what part of the past history is most relevant to the G to the representation of the current word and it looks like this so here is a a sequence in a dialogue model that says um I need um a Japanese restaurant in Morgan Hill um and then the system responds with I found three options and one of the attention heads in this is gpt2 actually one of the attention heads um was mapped here and the options token actually combines the representations of the words Japanese restaurant and Morgan Hill because when the system says I found you three options the options it's talking about are those restaurants that you asked for so this is what we call a contextual representation or a contextual embedding we are embedding the meaning of the word options in context that it appears which would be completely different if it appeared in a in a different context and this is just the the way it's formulated you have a query vectors you dotproduct them against key vectors gives you scores of which one of these past words in your uh which I if I can I have a pointer here I guess I'll just use the Mouse um so which ones of these past words over here are most relevant to the to the representation of the word options that's your query. key scores and then those are are multiplied against the the values of the uh context free vectors that are fed into the network all right so so now we've seen that llms are just over parameterized autocomplete models that have attention mechanisms and a lot of other interesting nice properties so if you if you feed into to gpt3 how do you say thank you in French it might generate I'm visiting France and I want to make sure I know basic eate because it's just coming up with something that might come next probabilistically it's this this looks legitimate right it also might generate something like uh how do you generate or how do you say thank you in Spanish how do you say Thank you in Italian how do you say thank you in German how do you say thank you in Portuguese etc etc etc because it's again it's just autoc completing what you put in it has no idea what you actually want so prompt engineering is needed to get desired results prompt engineering would in the past before instruction would come up with templates templates would would elicit the language model to um to Output the desired Behavior by Framing the input in the context in which it would have appeared in the pre-training so for example um documents full of question answer pairs were're in the pre-training so we would stick a q in front of your question um you stick a q in front of your question here and stick an a at the end and then let it autocomplete from the a so now it will generate the translation of thank you in French you can say Meri or meru um f shot learning would then be often needed to teach the such an L new task in context and this is simply done by giving it multiple examples question answer question answer question answer and now here's the actual question prompted for the for answer and then let it autocomplete so if you had a couple thousand examples like that you wouldn't need to do F shot learning you could then just fine-tune the weights directly for the desired template and and and then you wouldn't need to use the template you just feed in what you want and it would do what what you expect because you you updated the weights to to uh support that behavior and the critical thing to to think about now is that when we fine-tune a language model all we're doing is making the most likely autoc completion become a natural response as opposed to whatever the llm wants this is what we mean by alignment we want to align the behavior of the language model with what is people would expect from it such as a random user who's never seen an llm before who opens chat GPT they would expect their question to be answered they wouldn't expect some list of other questions to be generated so yes oh yeah do elaborate again most natural response isn't LM always to complete I'm confused about the phas of natural so what I'm trying to say here is that um the natural response to the question would it would be an answer right and what what we do is we say that when the when the llm is autoc completing we're going to boost the probability of the word you here yeah and and then the rest of the sequence can say mer C Etc we're going to boost those probabilities so that they are the the what comes next in the autocomplete the most likely we're just ping the probability right right yeah so it's still autocom completing it's just we've we've we've changed the autocomplete so that it autocompletes things we want it to now um such a fine-tuned llm would become specialized to that one task like translating English to French and would be incapable of others so if I then take that l and I say what's 3 + 4 it might output you can say comn font 3 + 4 right because because I'm overfitting that's why but the solution here is instruction tuning instruction tuning is the process of fine-tuning on a mixture of tasks that are all prefixed with natural language instruction such as answer the question what is 3+ 4 translate the following sentence into French thank you and then um training it to Output the appropriate answer for the appropriate um for the appropriate instruction in doing so the the language model will not only learn the tasks but they'll learn what the instructions mean and how to derive the meaning of the instructions and be able to potentially follow instructions that you didn't train it to do and this was originally explored in the um in this paper um from 2022 called flan this was out of Google um fla would essentially um built a huge data set of a whole boatload of tasks and um prefixed them with natural language instructions and then fine-tuned a uh decoder only model kind of like gpt3 um on that that uh task and then compared the zero shot performance with the original base gpt3 and they found that the um that after doing this they they could do zero shot prompting of their model FLA sorry my mouse is over here okay they would do zero shot prompting of of fla and they would outperform the um the fuse shot prompting of gpt3 in the case of um I'm sorry I'll perform the zero shot prompting of gpt3 and only and and come close to the the F shot uh performance of gpt3 so what this means is that you can get this same result by by as giving a list of instructions to a an untuned language model you can get that same result by fine-tuning it to follow instructions and then describing the task in natural language which is great because that's what users are going to do they're not going to sit down and craft five or six examples of what you wanted to do unless unless it's a fourth grade teacher maybe they would do that but anybody else is not so so they they measured their um they they measured measured um human preference ratings against gpt3 and found that um that their instruction tuning model actually I'm sorry I skipped ahead here um this is no longer fla this is now instruct GPT which is uh the next major work around instruction tuning but they basically did the same thing that flon did except it came from open AI not from Google so here they they built another data set uh instruct GPT they built another huge data set of um natural language instructions and they compared it against Bas based um gpt3 they found that when they um when they use the instruction data set the um humans would rank the preference of the instruction tune data sets significantly higher than that of base gpt3 and that's what you can see here in this table so this over here is the um the instruction tune model and down here are non-instruction tune models and these are percentage Point win rates so the so 60% versus 20% win rate for base GPT versus a uh um instruct GPT so that's pretty significant here are some examples from the flond data set of what these tasks look like after they've been formatted so um hopefully you can see these I don't know I don't know if you can I'll read a few of them in case you can so the first one over here um is zero shot rule based manipulations so the input is John is reading the paper fine tuned language models are zero shot Learners change this to past tense and the output is John was reading the paper find find fine language zero shot Learners um we have here the last one on the on the left says John is reading the paper fine tune language models are zero shot Learners change the verb to eat and the subject to Noam chsky and the output is Noam chsky is eating the paper fine tune language models are zero shot Learners so these types of natural language descriptions are only uh are only only a um followable if this instruction tuning process has been carried out otherwise it's just an autocomplete right um on the right we have sentiment analysis the inputs a movie review since this movie was the best Ro this movie was the best romcom since Pretty Woman did the dis the critic dislike the movie options yes no um and then the uh model would output no um here we have we have some other examples of tasks that are instruction tune formats um for the sake of time I'm not going to read these but um once the slides are distributed if you're interested take a look or take a look at the original flond paper um but the point is you can format pretty much any natural language task as a natural instruction so in addition to instruction tuning the instruct EPT uh research involved a process We call we now know as rhf and um this became famous after it was used to align chat GPT as an assistant rlf is the process of um doing reinforcement learning from Human feedback and what they would do is they would collect a data set of instructions and human provided demonstrations that's step one they would train a reward model to predict um whether or not a particular demonstr is a good demonstration of a given instruction that's stage two and then stage three is they would allow the um the large language model that they want to align to generate multiple possible um demonstrations of a given instruction and then use the reward model to rank them in order and use that ranking as a reinforcement signal using po that's our lhf and this is what they did to improve the performance of instruct GPT and ultimately chat GPT is just instruct GPT except they formatted the data set in conversational format so the takeaway from instruction tuning is that pretty much any off-the-shelf llm that you're going to use these days is going to be instruction tuned and also likely aligned with r lhf however if you're using open source models you need to be careful because you might come across models that come in both Bas instruct or chat variants um like for example hugging face here is llama 2 chat which is an instruction tuned rhf ready to serve you personal assistant and you have llama 270 billion no chat it's just a wild Untamed autocomplete engine yes so for ch um how does the propy or yeah the reinforcement running model it you gener the model the decoder generates the entire output and then it makes it video b or does it come in at a to level so the uh the scoring model is going to is going to rank um multiple samples from the model that you're trying to do rhf on and it will it those scores will be or the ranking will become the scores essentially where the the the first ranked example is the highest score those scores would then um be applied as token level feedback for the entire for every token in that particular trajectory that sequence so the highest ranked sequence would get the highest score applied evenly to all of its tokens um additionally po does some some um some uh anchoring so it will it'll also minimize the kale Divergence between um between the model as it trains and a frozen copy of it before so that it doesn't sort of go off the rails and forget everything that it new yes for the r can assume that gbt right if two human asking a question and the gbt gives the same Poli answers as a human to evaluate uh the the feedback from the chat the answer from the CH is or not uh it would be two few giv exact opposite uh feedbacks that will be a complete how they manage so you're talking about step one where they're collecting demonstration data from humans um so if if they if they gave the same prompt to multiple humans I forget if they did um sometimes sometimes they do that um but uh if they did then I would imagine that such conflicts would be so rare that they would just they would just get uh um well because the scale of the data set that they have is so large and those conflicts would be rare they would just get sort of come out in the wash the model would learn what it needs to learn anyway because the frequency of non-conflicting instructions is much higher than the frequency of conflicting instructions um it is also possible that they filtered for that ahead of time but I don't I don't recall if they did I'd have to check the paper [Music] %% if the data set had 50% of um yes if if that was the case and and I know that wasn't for the original um um instruct GPT but let's say you're building your own alignment data set and you hire people and it turns out that you have that 50/50 split on all of your instructions if that's the case then you got to go back and then change your instruction because the the model isn't going to get the signal that it needs in order to learn with the preference ranking so on his original question seems to me coming out in the wash is the right answer right if if you've got uh human humans that are in Conflict then the end result of the learning is going to properly reflect the human respon right yeah is if the human response is in determined the models not be right except the tasks that they were training it on things like translate from English to French um it would be you'd have a much larger problem if the if if you had a 50/50 split between all of the same instances of translations from one to the other oh yeah I'm not talking about question 50/50 yeah talk about those yeah so they'd be relatively rare they would just come out in the wash because this because the scale of the data set is so large um I about that they just twak a little bit for chat GV but like it seems like they got a lot of safety stuff in there and stuff what is exactly is it making that model and making it chatty or do they find food to be chatty and where does all the safety stuff come in so the the base version of chat jpt that was originally released was um a a a conversationally formatted instruction tuning data set so the instruction tuning wasn't just instruction task instruction task it was you it was speaker colon instruction um you know uh agent or assistant colon response speaker colon no that's not right um agent colon I'm sorry you know things like that so so they would fit their instruction following trajectories into a dialogue as opposed to just having it floating out there as for the safety stuff um that that that's part of the rla Jeff so if they would have what they call Red teaming you have teams of of people that they hire and those people would um would elicit um unsafe responses from the model and then down down rank them so that the reward model would know the difference between safe and unsafe responses before they perform RF so maybe a silly question but Inu GPT was it owned by openi yes it was a it was a modification they made on gpt3 I think it was D vin2 um actually we I still have quite a bit to go and uh yeah I don't want to I want to make sure we have time to get through it um but we I'll take more questions um at the end so now we have um Chain of Thought So that's the next piece we're going to look at so um so we know now that llms are not just overized Auto overparameterized autocomplete models but they can follow instructions um but they're pretty awful at math and reasoning tasks so for example um if you ask it a question like um the cafeteria had 23 apples if they used 20 to make lunch and bought six more how many apples would they have that might generate the answer was 27 which is wrong right but the problem um can be broken down into simple steps and make it tractable for the language model this is called Chain of Thought or coot prompting Chain of Thought prompting will elicit step-by-step reasoning and language models and the way it was originally explored by wey and colleagues in 2020 uh 2022 was as a a method of form of forming few shot examples so that it would follow the step-by-step breakdown as opposed to trying to answer it directly so here if the question was Roger started with um sorry if the question was Roger has five tennis balls buy two more cans of tennis balls how each can has three tennis balls how many tennis balls does he have now they would give a demonstration of the stepbystep breakdown to the language model here in the F shot over here in the F shot example Roger started with five balls two cans of three tennis balls each is six tennis balls 5 plus 6 is 11 so thus the answer is 11 now they give the next question and the model has already seen in the F shot example how how um how the uh human had broken it down step by step so it then elicits that same step-by-step breakdown behavior in the response and it turns out um that that this can make the um turns out that this can make these models a lot more accurate at specific types of reason in tasks and it was discovered actually um just a year later that the largest llm such as gpt3 can do this zero shot if you put the right incantation into the prompt and the incantation that they discovered I remember seeing a tweet out that was um expressing absolute incred incredulity what's the word incred incredulity at the fact that there was a NPS paper that was accepted around the the entire technical contribution was an incantation that they feed into the language model so if they if they would do if they would affix their prompt with let's think step by step then the largest language models would then break down the prompt into the co or break down the problem into the coot format whereas before you would have to do fuse shot prompting or um or or something like that and they found that this Behavior emerges at a specific scale so they did they compared different scales of gpt3 and they found that only at the um only at a specific scale um I forget exactly how many parameters L has oh over here's uh that's right seven billion parameters so um only when you hit 7 billion parameters after that point does um coot start to emerge before that it doesn't make a difference but once we get up to the Excel the 175 billion parameters that's when you really see this huge difference between performance on a task so what you see here is that um there's 177% accuracy on a mathematical reasoning data set just by putting let's think step by step after all the prompts in the data set the model now magically ass um achieves almost 80% accuracy actually the SC the bot is like 62 to 540 right um yeah this is a different data set um this is the first part is this m multi arith data set this is GMS 8K which is I think e8th grade math problems yeah and it's also a different model this is Palm Google's Palm as opposed to open GPT um oh so critically the um the technique was dramatically more effective on instruct GPT than base GPT so you can actually see here um on the on the left here there's this is the original gpt3 versus instruct gpt3 the original gpt3 did not really have much of a of a significant boost like that at the 175 billion parameter scale but the instruct version did so instruction tuning plus scale allowed this coot zero shot co phenomena to exist and other similar phrases also yielded similar results um in case you can't see the table they triy stuff like let's think about this logically let's think solve the problem by splitting it into steps um let's think like a detective um the answer was just after the proof um things like this and they found that some of these prompts actually had similar accuracy boost although less thing step byep um yielded the best one in their experiments and this technique would work for a variety of reasoning tasks not just for math they tried it on also um on on um common sense question answering strategy based question answering date understanding some task involving um Shuffle shuffling or unshuffle object descriptions um coin flip task if you're interested in these task descriptions uh then then you can see the the uh experimental details in their paper but the point is that it works for a variety of reasoning tasks rather than just where the focus happened to be with coot originally was was with mathematical um word problems so here's some examples um from the common sense QA um if there is a pond with trees around it where will it likely be located answer choices a the ground B the bathroom C the forest D the country side e the rural area and the prompt was let's think step by step and then it generated a pond is likely to be located near trees because trees near water to survive therefore the most likely location for a pond with trees around it is in a forest therefore among a through e the answer is c um here's a comparison of zero shot versus zero shot with coot for question is for Halloween Megan received 11 pieces of candy from Neighbors and five um pieces from old her sister if she only ate eight pieces a day how long would the candy last her zero shot um would just said three days whereas coot said let's think step by step Megan received 11 pieces of candy from Neighbors and five from her older sister that means she has a total of 16 pieces of candy if she only ate eight pieces a day she would have to divide the candy into two days that means she would have eight pieces of candy for day one and eight pieces of candy for day two therefore the answer is two so we can see here what the effect is that this has recently authors have proposed improvements to co so now uh inan to your question is this still popular um it's popular in that people have moved past just coot and are now building on it to build more complex types of reasoning behaviors and language models um here we'll look at two of them um one of them is called um one of them is called self-consistency and the idea is simple here instead of just doing one chain of of reasoning they will do multiple chains of reasoning by sampling from the language model and then they will do a majority vote to see which which answer did did the majority of the sampled reasoning chains arrive to um which you can see here here uh two reasoning chains led to the answer is 18 whereas one led to the answer is 26 so it selected the answer is 18 and it turns out in evaluation that that boosts performance um over base Co with 3D decoding uh for example for um on on that uh eighth grade reasoning math problems data set from 56% to 74% accuracy um and for and on multi-e from 94 to 99% accuracy um and the same types of results for some other data sets as well so when when we do we've done a lot of experimenting Chain of Thought with chat BS SE before and we find a lot of different paths that it comes out will bot out 10 results at a time and they'll be different but all correct you know the the the chains of thought that are indicated is is is that what's happening in G4 we're giving it it's zero shot only explain step by step it's doing this so what the fact that it's coming up with all these different R all all plausible paths what's going on well how does that relate to this it wouldn't necessarily be um Chain of Thought with consistency with self-consistency unless unless you implemented the self-consistency checking um if if this just happens to come up with correct answers then that's just um a result of of the quality of its training so I guess what the maybe the question should what's what what is what's I work for open AI they probably not doing this at all this is just a uh this is a post-processing step that you can do if you if you want to yes if you want to improve the performance of coot for whatever mod you happen to be using so if you took if you implemented this on top of gp4 you might even get even better results quick question is when they're evaluating these are always natural language answer so like say you're doing math task it's going to say like a paragraph of stuff somewhere there's going to be the answer are these guys manually reading the stated that kind of was a problem I had just the number um you're talking about the answer is 18 how do they know that that's correct or no like so how do they like do the mention how they're evaluating so like the when you give it the um problem you just want number 18 a whole lot of you have to to get answer they um I don't recall that exact detail I can I can get back to you on that yeah yeah so um it's currently we currently have 15 minutes left I'm we might go a little bit over um if you got to go then you know feel free but uh so Chain of Thought um next version of chain of thought that that I looked at here is also called tree of thought tree of thought is an extension on on self-consistency that self-consistency is only considering different branches of reasoning in isolation here tree of thought actually does tree exploration either bread first breads first or depth first by at each step of the reasoning process considering potential Alternatives and then reranking them using a um using using a vote system so for example here there's a a writing task where where it's broken down into two steps first planning a skeleton of what the the final uh uh product is going to look like and then step two would be actually writing the passages of that final product and before selecting a reasoning path to use as the conditioning for the writing part they first generate a bunch of plans over here and they use a voting system and in this case they just feed the the plan back into the llm and prompt it to to to to say whether it makes sense or not and they will um select the the best plan and then go with that before moving on to the next type of Co which is then to generate the uh the passages based on the plan so they go and they're able to explore a whole different possible tra trajectories of reasoning maybe you know coming down here this one doesn't work so so go that way then condition the entire next row of options based on the selected one so it's more of like a a more of like a a depth first or breath first search type of thing um where where reasoning is broken down the almost like the the chain of thought process itself has broken down into a step-by-step process so um so this is really interesting and um it a lot my n collaborate because it's trying all these possibilities it doesn't do greedy um yeah I think that's the the the decoding method that that chat GPT uses I believe is called speculative decoding which is unrelated to this um this again is is a prompt is like a it's it's more of like a control mechanism how you how you feed your prompt in and what you do with the response it wouldn it's all external yeah so um the last thing we're going to talk about before talking about agents is retrieval augmentation and the motivation here is is um is in terms of factuality so if I were to ask anel I'm I'm planning to visit the Palm Island in the Grenadines can I drive there you can see on this little map here that uh if you wanted to drive there you would need a submarine so the llm would say yes you can drive over the Palm Island Bridge from Clifton to Palm Island St Vincent and grenadians because that sounds like a pretty nice autocomplete right what if instead the llm would first query a knowledge Source like Wikipedia extract something from it append it to the context such as the phrase Palm Island in the Grenadines a small island one mile from Union Island only accessible by boat and then you because that information from an external knowledge source is appended to the context the tokens and the correct information become more likely in the autocomplete process so the llm becomes more factual and this is a process that we call retrieval augmented generation or rag um I I just want to ask people please at this point hold your questions to the end otherwise we're never going to get there so um so approaches to to retrieval augmented generation include um realm rag dense Passage retrieval fusion and decoder and retro these are some of the papers that have have worked on this um but they all sort of share the same idea which is you encode parts of the text use a vector space kind of like what we're talking about with word to V where similar words are densed or are put into similar neighborhoods of that space But here we're talking about entire queries like um can I drive to to Palm Island that would be embedded we would find relevant documents in a document store such as um heard of milis or um or or we8 you know these these Vector databases that you can do approximate nearest Neighbors on and get back documents that are relevant to your query and then um it would inject it into the context to come up with a better answer and um the original paper that was titled retrieval augmented Generation Um they actually did something a little bit more sophisticated they would actually um use they would they would use a fully end to-end trained query retrieval model called myips um to generate their queries and then they would jointly tune their query and their generator by marginalizing the probabilities over all over the a set of responses given the retriev documents um so so the probability of a particular sequence given its input would be the breakdown of the probability of each of the retrieved documents the probability of the document given the input times the pro probability of the answer given the document and the input and uh this would be marginalized out and the the best or the most probable combination of all of that would be what the model would optimize for so um recent more capable instruction tuning models made that sort of not necessary because these instruction tune models could do a very much simpler form of rag which we now call tool use basically it would this generalization of of rag would basically allow a model to predict the name and parameters of a tool such as a a python Ripple or a search engine or a calculator or a calendar or a uh external Knowledge Graph get the response back from that and and prepend it to its context so no need for for joint training no need for myips or DPR it's just instruction tuning allows this to work and here are some papers that that explored this in in um 2022 so bringing this all together now we looked at the scale effect of scale instruction tuning alignment Advanced prompting with reasoning retrieval augmented generation for for knowledge grounding and Tool use which is just a generalization of that um now what's missing is a perception action Loop that would allow this such an agent to act in its environment and and handle much more complex types of problems this brings together everything that we've talked about so far so the most famous uh version of this is called react it was a paper uh from this year actually was from late last year that um called synergizing reasoning and agents uh and acting in in language models and what it would do is combine um it would combine Co tool use and um put that in a loop so we can look at some examples of that over here you have originally um a question aside from the Apple remote what other device can control the program's Apple remote control the program Apple remote was originally designed to interact with so in order to answer this question it has to First figure out what was Apple remote originally designed to interact with then it has to figure out um what other device might be able to control that thing so it's a multi-step or multihop reasoning problem which is a lot more complicated than um what C co uh was even originally doing so what they what they did was they they put they they took um they considered reasoning only models so let's think step by step and come to an answer they considered Acton models which would do tool use they would use a search engine to search first for the Apple remote then for um then for front row I guess that's the name of the software um and then they combined that with coot so what they call um they call react reason and act so first it would prompt it for a thought let's thing step by step right I need to search Apple's remote and find the program was originally designed to interact with the first action would be then as a result of that searching for Apple remotes observation is then returned from the search engine and it Loops thought two apple remote was originally designed to control the front row media program I need to search front row next and find what other devices can control it act two action two is search for front row observation two comes back from the search engine thought three and Now We're looping again front row it was not found I need to search front row software okay so it searches for front row software observation comes back from the search engine and we're loop again thought four front row software is controlled by an apple remote or keyboard function keys so the answer is keyboard function keys so you see here's a four-step reasoning problem involving multiple uses of a search engine with different parameters and react um combines instruction file following Rag and Co to make that happen here's another example I'm not going to read this one um but um it shows a react agent similarly navigating a text game called Alor um and they did some evaluation here on various data sets and found um as we would expect um significant improvements on various data sets that that require multihop reasoning so hot pop Q way they went from um 29% using standard uh Co to uh to 35.1% using um using um react and they compared this also to basically the state of the art using supervised learning that specialize models for those tasks um since then there's been a um a surge of agents that use this type of of uh of um control flow and particularly there's a data set that was released Called Agent instruct um data set this is a a a data set of full trajectories that go from a user's instruction to multiple Loops of of react all the way to An Answer full with specification of tool use and um you can fine-tune an open source model on such a task and you'll find that um if you reproduce the results in the paper the um the open source model llama 2 can now approximate the behavior the zero shot behavior of chat GPT GPT 3.5 on such long long Horizon reasoning tasks in a react framework which is pretty cool um I'm actually using this in in my own research um and there are challenges but uh it definitely has some some advantages on loopin tool use and and and reasoning over what we saw with base open source models so takeaways on agents um is that they can tackle far more complex problems than previously thought possible they combine instruction following reasoning and Tool use step by step the um the short-term memory of one of these agents because you would think if it's taking multiple steps it needs to remember what happened in the past that's its context so if you have a long context language model then that becomes its working or short-term memory it can remember things that happened um in the intermediate steps however the long-term memory of such an agent is the knowledge that it retrieves via rag or through tools so um there have been already efforts to formalize the components of a cognitive architecture around llm agents for example um you could use rag to allow the the agent to access a a semantic memory through a knowledge graph an episodic memory by doing retrieval augmented generation on a list of its past conversations um and then procedural memory which is essentially instruction retrieval or um or or um direct modifications to the to the um Loop of the the code that governs the loop of the agent um and then this is called the koala cognitive architecture so it's sort of attempting to marry large language models with traditional cognitive science and production systems and um if you came to this talk hoping to see some practical resources here they are we have Lang chain um which has a whole set of documentation on building Agents from from scratch using open source language models in python or using open AI hugging phas has its own agents framework which is still sort of in experimental phase I wouldn't recommend using it now but it's there in case you want to look at it you also have the open AI assistance API which is um just dropped from open AI a couple weeks ago and um it seems to to Really um facilitate dead simple development of tool using agents um even swarms of agents that each have specialized tasks and work together which is um a even more sophisticated form of of um of of intelligence that that's sort of approaching swarm intelligence right you don't have just one agent but you have a collection of them collaborating on a larger scale task and then you have um the Microsoft autogen framework which is um you know sort of similar to the assistance framework and um on incate