Lec 21. Language Models

Key Takeaways

Uh today we're going to have a guest lecture from Jacob Andreas and it's going to be on large language models. So um we debated in this course whether to have like a lot of material on large language models or only a little and we decided to have only a little because LMS have actually become such a big and important subject that they deserve their their own dedicated courses. So there are additional courses taught by Jacob and Kim and others which you can take just to learn all about language models. Um today we'll just see some of the basics. Uh but why is this related to deep learning? It's kind of there's two ways of thinking about it. One is that well large language models are all built on top of deep nets. So LLMs are made out of deep nets. That's the way that that it works. But a lot of people have also started to say well what's the next big thing after deep learning? And one one possibility is LLMs really deserve to be considered the next big thing. It's the next paradigm post deep learning. Uh so you can look at it either way. It's a paradigm that's built on top of deep learning, but it is uh it has new kind of fundamental math and characteristics that go beyond um like weight continuous weight updates and NLPs and confinets and so forth. Okay. So, Jacob is a professor here in um EDCS and he is one of the yeah the world experts on large language models especially on kind of the scientific understanding of them not just building them but really understanding what they do. So, let's welcome Jacob. Um, thanks Phillip. So, yeah, just to reiterate, um, first of all, a bunch of the slides that I'm going to be using today are this is a sort of compressed version of a couple of lectures that we teach in the big NLP class. Uh, if you're interested in this stuff, I encourage you to take that, which is 68610 um, as well as UNE's special topics on on language models class and probably more course development in that direction coming in the next couple of years. Um, so okay, uh, this is a a lecture about large language models. we were starting to, you know, plan out a lecture on large language models. So we asked a large language model to plan a lecture on large language models and it gave a reasonable looking outline. Um and so this is more or less the structure that we're going to follow today. But really what this is going to be about uh is you know in fact mostly especially after the first chunk of the lecture abstracting away from some of the details of the uh underlying architecture that we're using to predict language and just talking about what you can do if you have a really good system for uh you know sort of doing language generation doing next token prediction uh and now you sort of can imagine how to build build such a system using some of the stuff that you've been talking about um in this class um but I want start actually by talking not about language models based on neural networks uh but a much much earlier generation of uh tools like this in uh in NLP and machine learning uh which were count-based language models um and so right and have you talked I guess even like most basically about what a language model is or what I mean by that >> you should define it >> okay good so in general the only thing that we are going to be concerned with really for this entire class today um is a model that predicts some uh next word uh in a distribution over or rather that you know takes in I guess a sequence of uh words or tokens or however you want to think of them as input uh and places a distribution over possible next words as output. uh and if you have the ability to do this right if you have the ability to see you know some sequence of input words and get a distribution over uh words that come next via the chain rule you can turn this into a distribution over strings you can sample from it to generate language and you can do all kinds of other things um and this basic question of you know how do we like predict a next word given a bunch of words that came before um is something that people have been thinking about in language processing for a long long time um really dating all the way back to like the end of the second world war uh when people were trying to decode the outputs of the German and English machines and they had these decipherment procedures that would give them a bunch of candidate uh translations and you needed to figure out which one was most plausible. Uh and the best way to guess at that was basically to say well how plausible is this particular decoding that I've proposed to produce as a sequence of letters in German or as a sequence of words in German or something like that. Um and for a long time the ways in which people estimated these kinds of models was just by getting big uh text corpora and counting right. So either just saying well let's ignore the context and figure out uh what the probability you know if I give you a random chunk of English that some word is going to be the totally without any context and if you sample from this you get a sort of naturalistic distribution over individual words that doesn't cohhere into anything meaningful. Um if you look at longer and longer context you say suppose the last two words uh that I ever saw were you know once upon uh and I ask you to guess what word is going to come next. What do you think uh is likely to happen after once upon a time? >> Uh okay, so some people are saying time uh some people are saying uh probably uh both of these things occur with non-trivial probability in a big corpus. Right? If I just go out onto the internet and I count all the number of times that once upon occurs. Uh a lot of those are stories where people are saying once upon uh some of those are typos where people have forgotten the uh and are saying time. There's other stuff that can come next that's licensed by English grammar. And if you just count these things up, right, you say every sequence of four words, what's the distribution over things that could come next, you start to do a reasonably good job of generating text that looks sort of like uh natural language. Um, and this is basically all that the field did from, you know, say the 60s to the uh like early 2000s. And around the early 2000s, people started to say, well, what happens if instead of just estimating this distribution by counting words in a corpus, I actually approximated it with uh a neural network model of the kind that you've seen uh throughout the semester. Um, and it turns out then if you did it, you did only a little bit better uh than just these big count-based models at this next word prediction task. So, what we're looking at in this table is a measure called perplexity, which you can think of as just a measure of the quality of a language model's predictions. you get, you know, a sort of modest improvement in perplexity by moving from classical countbased models to uh neural network models. Um, and as our sort of deep learning toolkit got better and better, we were able to drive this number down a little bit more uh you know uh down to I guess the typical perplexity on a corpus of 107 in in a big corpus. Um, and basically as and hopefully these are models whose names are all familiar to you as a result of being in this course. Um as our basic machine learning technology got better, our ability to do next word prediction uh on a fixed corpus also uh got better. Um and finally right now we live in the age of transformers uh and almost all you know sort of models that we train to do this next prediction task in the modern era um look like this. Now one thing that I'm playing a little fast and loose with here and that we'll come back to in a minute is that I'm just showing you a bunch of numbers or sort of uh model architectures with numbers next to them. Um but in fact you know all of these things and especially once we talk about the difference between GBD2 and GBD3 or some generic transformer language model in GBD2 we're talking about changes in both uh the underlying neural network that we're using to do the prediction and also the scale of the data that we are training on. And so obviously the other big thing that has happened and that you're seeing as these numbers are going down on this chart um is that you know between 2000 and 2020 uh we've developed technologies that allow us to train just much much larger versions of all of these models on larger and larger data sets with more and more parameters. Um and these two things together right better prediction machinery uh and larger data sets are really what are contributing to this improvement uh in prediction. Um that being said, it is possible even in this you know sort of paradigm that we were talking about before to have a large language model. So this I think is actually one of the probably one of the first references to large language models uh in the machine translation literature. Uh it refers to training a model on two trillion tokens uh you know with 300 billion parameters. Uh and you can see that these parameters are actually engram counts. Uh what year do we think this paper is from? Yeah, >> 1995. So not not quite that bad. Uh but early 2000s, right? So really long before these things were in widespread use uh for or in the context of of of deep learning models. Um people were training very very very large uh language models on large data sets with lots of um uh parameters. uh uh and I think the main point that I'm trying to make here is uh that those models were typically not nearly as good as the the neural models that we see today. So I think you know there's a sort of habit of talking that people have especially now uh suggesting that really you know more data and the ability to train larger and larger models is the only thing that uh matters and is the only thing that's kind of driving success in AI today. Um and in fact I think you should take this as some evidence to the contrary that we have been training models with as many parameters uh as many data points as many of the largest language models that we had today. Uh and they weren't as good as the ones that we have today. And there really is something special about the inductive bias that you get from uh neural sequence models that we didn't get in previous generations of language models. And we'll see some concrete examples of that um momentarily. Uh so just to say this very explicitly, right? uh even though we sort of talk about parameter counts as as being funible today, in fact there is a lot of historical evidence that how you use those parameters uh specific architectural choices uh actually are quite consequential con consequential sorry um okay so to go again very briefly through the history of the field right between say 1970 and 2014 a long stretch of time people were not typically interested in these kinds of models as tools in their own right uh that Clearly, it was typical to have some larger pipeline system, an automated speech recognition system or an automatic, you know, sort of machine translation system, uh, that was generating a bunch of candidate outputs, uh, and where what you wanted to do with a language model was just reank those outputs to say of all of the possible transcriptions of this sentence that I just heard, is it more plausible that a person said they wanted to recognize speech or wreck a nice beach? uh or you know in sort of translation do do all the sort of uh genders and numbers and things line up in a way that our machine translations systems which were not that great at the time um weren't all all that good at handling. Um and so for a long time people were interested in language models really just as measures of sort of linguistic well-formedness uh that you could tack on to the end of some other NLP model that was being customtrained to perform some specific task. Um, and what really shifted around uh around 2014 was that people started to realize that you could just take these language models uh and get them to perform the task of interest on their own. So rather than saying, I'm going to train a machine translation system. I'm going to train something that's good at recognizing English sentences and then I'm just going to combine them together. People said, well, what if we just pose machine translation, the problem of taking a French sequence, you know, a French sentence as input, say, and translating it into an English sentence as output. What would happen if we just pose that as a next word prediction problem of the same kind where right maybe these two words are two words in French. Uh this is the first word of the English sentence that you've generated and you're trying to guess the next word of the English sentence as well. Um and so starting around 2014, people started to take this basic language model machinery that we' built up um and use it to do uh sequence generation instead. And in parallel with that, right, our sequence models themselves were getting a lot better. uh we invented things like uh like the transformers. Um and then what's really changed since 2017 has less to do with uh the actual models that we're training and more to do with the sort of procedures by which we are actually training these models. So over here um it was pretty typical to uh say I have my machine translation data set or my speech recognition data set or whatever. I'm going to randomly initialize my model. I'm going to train it just on this data set of interest and what I'll have out at the end is a really good machine translation system uh and and maybe nothing else. Um and what people realized starting around 2017 although again this was an idea that had been floating around in the sort of machine learning literature for a long time um was that you could uh get sort of non-trivial transfer effects between different tasks. So you could for example start by training a model uh just to do next word prediction on a big data set and then fine-tune that model to do some specific task of interest like machine translation or speech recognition or whatever. Um uh and that this was a way to get away with uh much much less data for any specific downstream task of interest. Um, and finally, the sort of era that we live in now, which we might call the era of pre-training and prompting, um, doesn't even go through any sort of downstream task data at all or very little and basically just says, well, if I have a really really good, uh, really general purpose, uh, next word prediction system, what are all the things that I can do with it without having to ever even touch the parameters again or or make only very small changes. Um, and so this is the sort of highle overview of the field uh, or the maybe the history of the field for the last like 10 years or so. Um and we're going to talk in now in a little bit more detail about each of the pieces uh of this. Um really focusing uh on the second two stages here. Um okay. So uh what does it look like especially to operate in this kind of pre-train uh and transfer mode? Um, and the thing that people typically did uh for a long time was actually not quite this language modeling task that we've been looking at here, but something that was called mask language modeling. And I imagine you've seen something like this in different flavors for for vision tasks, if not language tasks. But the basic task is we're going to take some text as input. We're going to hide some words from this text and we're going to ask a model to predict what the missing words were in that text given all of the context that showed up around it. And what's nice about this objective is that you can do it without needing any labeled task specific data, right? You just get all the text that you can get your hands on. Uh you train a model to perform this task and you have a model now that's uh you know maybe you can set some sort of specific classification problems or whatever up as these mass prediction tasks. Um but really for the most part this model is not useful for anything in particular. um what it does know and for reasons we're going to look at in a minute is a lot about the structure of language, a lot about the meaning of language uh in a way that makes its representations very useful for downstream tasks. And so typically what we would do um after uh pre-training uh on as much text data as we could get our hands on with this sort of mass language modeling objective um we would then fine-tune this model just using say one hidden representation from one last layer to do whatever downstream thing it was that we were trying to do whether that's sentiment classification or uh or whatever else. Um uh and this was an incredibly effective paradigm when it first came out. And you know, I think I really want to it's easy with uh the benefit of hindsight to forget about what a sort of big deal uh this basic paradigm was when it happened. And I think even if you had gone to uh you know say NLP conferences for a couple of years before this uh B paper came out which was one of the first papers to really show like large scale transfer effects from uh pre-training on language modeling. people would have said, "Oh, yeah, you know, this next word prediction task, it like seems in principle like it should be the right pre-training task, there was a lot of interest at the time uh in just learning like word embeddings uh at the very inputs to these systems um and and evidence that you could get good transfer effects from those, but nobody had really managed to crack the recipe for uh getting pre-training to work. Um and around the same time this BERT paper and another paper called Elmo which was actually a little bit uh earlier than it and used language or used RNN language models showed that you could train uh at a very large scale on uh this mass language modeling objective and then fine-tune at a very small scale uh on various other kinds of tasks. These are mostly sentence classification tasks uh lexical entailment or I guess logical entailment um sentiment various things like that. um and do very well. Um so why does it work, right? Why is it that we can train a model uh to do this weird nuisance masked uh word prediction task uh and wind up with the ability to do all of these more complicated NLP things. Um and you know I think the main intuition that now maybe is pretty common place but again was I think surprising to a lot of people at the time um is that doing a really really good job of uh even this simple task of predicting words in context um requires both a huge amount of linguistic knowledge right it requires knowing uh that what's going to come after was probably needs to be uh a past tense verb it probably needs to agree with the word MIT um uh and so on and so forth And you need a lot of knowledge about the world, right? Uh if you really want to do a good job of filling in holes in the sentence, you actually need to memorize facts uh contingent facts about history like the fact that MIT was founded in 1861. Um and you know by training on all, you know, again sort of all of the text with all of the facts expressed in it, we would wind up with models that actually embodied a lot of that knowledge and their internal parameters and that could be accessed very uh easily via fine-tuning. Um, okay. One thing, uh, that you do not get out of this masked language modeling paradigm, uh, is a model that looks like this. A model that can actually generate the next word at the end of a sentence given everything that came before. Um, right? And the reason for that is that the way we set up the inputs to these mass language modeling problems, the model already knows exactly how many words are going to be in a sentence, uh, it's been trained in such a way that often the missing words come in the middle of the sentence rather than at the end of the sentence. Um, and so while you get good representations, what you didn't get out of any of these sort of birectional transformer models, uh, was something that you could actually use to do text generation. And so if we wanted to use these representations for doing things like machine translation or summarization or any other task where there was going to be textual output um typically that second stage the thing that was doing the text generation itself uh had then to be trained from scratch. Um so what would it take to go from this kind of pre-training paradigm or I guess this kind of pre-training paradigm this mass language modeling paradigm to something that we could actually use to do text generation uh as well. Um uh and you know I think the main thing that changed uh originally was just that people said okay uh this sort of pre-training thing this mass language or this pre-training thing seems very effective. Um we already have models that can do conditional text generation. We already have models that look like this uh and that have not been pre-trained in a very large way and that predict the next word in a sequence. Uh let's just figure out which one comes next. um uh why not just do both of these things at the same time, right? Rather than training conditional language models, uh to output, uh translations or summaries or whatever, go back really to the paradigm that people were using um all the way back in the sort of pipeline days and just say give me a really really good model of all English sentences or all sentences in all natural languages uh that's not specialized to any particular kind of task. Um and originally people did this in basically the same uh or used these models in basically the same way as they used these BERT models, right? You could get representations out of these things that you could plug into a downstream classifier. Uh you would uh of course also be able to fine-tune these models to then perform targeted generation tasks. Uh again sort of using translation and summarization as the canonical examples. Um and this was really the paradigm uh that we operated in for a long time and people just kept doing this with larger and larger and larger models, more data, more parameters. So, uh yeah, I guess these are these dots are all labeled with uh the uh logos of large industry companies because this was sort of the last year that academic labs uh were really playing this game on their own and and thereafter uh most of what we've seen have been uh various sort of large scale models uh trained to do just this next word prediction task. um questions up to this point. I should say also feel free to interrupt or raise your hands or whatever. We can keep this interactive. Okay, cool. Um and you know, so again for I would say most of the models that we're showing on this slide right now, the main things that people were doing with them were using them as starting points for downstream fine-tuning or for pulling representations out of or whatever. Um, but then something changed and what changed is people started to notice that you could actually get these models to do some fairly surprising things without doing any additional fine-tuning, without really touching their parameters ever again. Um, once these models got to a certain scale of parameters and data, uh, right? So the examples that we're looking at on the slide and this is from a sort of relatively recent paper looking back at this uh but was that large enough models could start to do arithmetic without having been trained in a targeted way to perform arithmetic tasks. They could start to do you know things requiring highly specialized linguistic domain knowledge like producing phonetic transcriptions of words uh giving only their natural language input. um they could start doing things multilingually, right, without necessarily having to go out and collect a targeted data set uh for some task in some language of interest. So the example we're looking at here is you know um doing a question answering task where all the questions and all the answers are in uh Farsc um and various other kinds of complicated things you know answering questions about uh like common misconceptions in the world answering questions about properties of physical objects and physical situations um and so on and so forth. Um and you know this is uh the GPD3 paper which at this point is uh is pretty famous but was one of the first ones to make this observation that you could really do quite well um at a certain scale without needing to do anything after you had pre-trained your model. And I think this is kind of the defining feature of the era that we live in. uh although there are going to be a bunch of qualifications that we come to at the end. Uh that models start to acquire uh models you know sort of train to generate language start to acquire all kinds of surprising capabilities at a certain scale uh without us necessarily needing to do any targeted fine-tuning for specific tasks. Um that being said, you know, part of what happened here is that people figured out uh that these capabilities were present in models. Um, and another part of it is that people figured out actually more sophisticated ways of interacting with these models in order to get these capabilities out. Um, and so the next thing that I really want to talk about here is exactly that. Once you have a really good model of the entire internet, once you have a, you know, sort of thing that's really good at doing this next word prediction task or doing sort of natural string generation in a really general way, how do you then take that model, that distribution over language, and get it to do the specific task that you want it to do? Um and once we start to move away from the uh the fine-tuning way of doing things um really the only sort of mechanism by which we can exert control over a language model right over a text generation system is by changing what we put into it by changing what words we ask the model uh to look at uh or the context in which it's making its next word prediction. Um and one of the most powerful tools that we have uh for changing the context in order to get models to exhibit sort of desired target behaviors nowadays um is what's called uh in context learning. And whether we want to actually you know why why we call it this and and whether this is the right way to think of it or not is something we'll come back to in a minute. But the basic idea here is rather than going to a model and saying uh like we've said before uh what's the probability of uh the word uh given once upon Uh if we're trying to do something like a sentiment classification task, we're going to say ask the model what's the probability of positive given um well so let's naively say suppose the first thing that we wanted to do is just like we're you know assigning star ratings to movie reviews or something. So if I said this movie was terrible, um, one thing that we could imagine doing is just asking a language model, uh, suppose you saw the words this movie was terrible as input. What's the probability that the next word that comes out as output is positive? Um but if you think about this for a minute uh this is actually a very surprising uh next word to come after uh an input like this. Right? If you're randomly browsing the internet uh and you come across a web page and the last words that you see on the web page were this movie was terrible. You would assign a pretty high or pretty low probability uh to any assessment of the movie coming next. Right? This is just not something that you know people don't say this movie was terrible positive. people don't say this movie was terrible negative. And you know, normally when you're in the middle of a review, you're not going to get labels uh for it after the fact. And so you just ask your model again as a kind of distribution over natural looking text uh what's going to come next after this movie was terrible. It's probably going to be putting most of its probability mass on you know uh the word and or you know the start of another movie review or something like that. And so if we want a model uh you know again our sort of language model to be a model of uh or for the specific labeling task that we're trying to perform, we need some way of specifying what kinds of outputs we're expecting and what kind of task we're asking the model to perform. And so the trick that people have settled on which is called uh in context learning or sometimes fshot prompting is just just to sort of create a fictitious document in which the word positive is actually likely to come next. Um and so we'll do that as follows. We'll say something like I loved it. It positive uh whatever worst film ever negative. And now, right, if you imagine coming across this page on the internet somewhere that said, "I loved it. Positive, worst movie ever, negative. This movie was terrible." And I asked you to guess what word was going to come next. Now, with reasonably high probability, you should assume that the next word is either going to be positive or negative. And given that these labels seem to actually match uh the sentiment of the text that comes before them, you might even guess that positive is going to be low probability here and negative is going to be high probability here. And so, the important thing, right, is that uh we've taken the ability to perform generic next token prediction, right? I can show you a weird looking document. You can figure out what the structure of the document is. you as clever human beings in this room and guess what word is going to slot into that structure. Um, and the sort of empirical discovery here is that uh once you've trained a large enough language model on a large enough data set, this is also something that just generic next token predictors acquire the ability to do as well. Um so in context learning um another thing that you can do here uh and something that we'll talk about a little bit more at the end um is what's called not few shot prompting but zerootshot prompting where instead of giving a model uh explicit demonstrations for the task that we're trying to get it to perform. We just give it a natural language description of the instructions for that task. Right? So I guess to be um you know we have this example on the slide instead of uh an example input and an example output and another example input another example output we just give uh a natural language description of the input or natural language description of the task one input and see if an output comes out on its own. Um and this again was a capability that was observed to emerge in models once they got to a sort of large enough scale. Um and you know you can imagine doing all kinds of combinations of these things right you can give both an instruction and a bunch of fshot examples. You can give the examples and then the instruction. You can give a much more detailed instru description of the task that you know talks about all the edge cases and the format in which the output should be returned and whether you want it capitalized or not. Um and for the most part models are quite robust uh to all of these things. Um and uh we've been through all of this before. Um uh and this works remarkably well. Um but as soon as this sort of behavior was observed uh it sparked a lot of debate in the community about what was actually going on under the hood in these models. Um and whether we actually you know uh obviously we can kind of draw an analogy between what we're doing here and training unlabeled data, right? Where I've given you an X and a Y and an X and a Y and an X and I'm asking you to predict a Y. Should we think of what's actually going on there as learning the way we think of gradient descent as learning or is this something more complicated? Um uh and to sort of piece this apart, right, there are a bunch of different things that you can imagine a model learning uh from an input that's being presented like this. One thing that you can imagine it learning is or sort of inferring from this input uh is just the format of the task that it's supposed to be performing. Right? We're now I you know I know from giving you this information uh that we're supposed to output labels that are either the word positive or negative and we haven't seen any other labels. You're supposed to output them immediately after one sentence. uh and that you know you should expect in general that the sentences that you're getting as input uh are going to be sort of shortish sentences and maybe seeming to talk about movies. Um and so one way to think about what might be going on in in context learning is that it's just about learning the format of the output uh that the model uh is supposed to generate and that all the other behavior that you get uh comes from pre-training and not from this context at all. Um oh sorry this is not uh this is just a technical thing that we're going to skip through. Um um sorry I'm a little out of order here. Um okay I guess before well let's let's actually finish this thought uh and then come back to it later. Um sorry my slides are a little out of order here. Um yeah. Well, so so this is maybe a point. So one hypothesis that you might have um about what's going on with this in context learning task is just that you're learn learning these formatting questions um and nothing else. And you know if you think about machine translation as a prototypical example of this um obviously you're not going to learn a new language entirely from scratch from three examples being provided in the input. uh you're certainly not going to learn how to translate from English to French sentences containing uh French words that you've never seen before or English words that you've never seen before. And so to the extent that we observe these kinds of prompting behaviors uh actually making models better at their downstream tasks, uh we probably want to attribute that success uh mostly to uh providing information about the format or inducing the model to better perform this translation task rather than all the other things uh that we wanted it to do. Um, and in fact, something that was observed for a little while, um, was that models didn't really seem to be paying attention to the associations between labels, uh, and input text at all. What we're looking at on the slide right now is an example of a paper from Sewan Min uh, where you actually uh, took your in context examples. You totally shuffled the labels, right? So, you know, maybe I replace this with uh, positive. That's not a working piece of chalk. Uh where did the chalk go? I guess it's this side. You know, change the label there, change the label there, shuffle everything around. Uh these are these red bars. Um and for the most part, they didn't actually hurt performance uh much at all relative to giving uh models actual label data for the task. Um and so for a while, the sort of conventional wisdom was that you don't really want to think of this in context learning phenomenon uh as learning at all. it's really just helping models retrieve uh tasks that they had previously learned how to do. Um there were also all kinds of other weird pathologies that were observed in these models where the particular choice of words that you chose as your labels if you know if you said positive rather than good or the thumbs up emoji or something like that could actually have pretty substantial uh impacts on the quality of the predictions that you got from the model. Um and also that models were extremely sensitive to the order in which these individual examples were presented and you could shuffle them around and again get big uh increases or big drops in performance um uh out of these models. Um however a lot of these issues seem to be going away with scale and I'm not sure I have uh slides for this no uh in this particular talk but even this uh phenomenon that we were looking at here uh where models uh seem to be pretty insensitive to random labels or flipped labels or whatever um in even larger models than the ones that we were looking at here in GBD4 and some of the other things that have come out since then uh models do actually start to pay attention uh to the association between labels and input examples. And nowadays, for example, if you take a state-of-the-art language model and you give it the sort of backwards sentiment task where this is negative and this is positive, um you can for most tasks reasonably expect that models will be smart enough to figure out not only that this is actually a sentiment classification task, but that it's supposed to say negative uh for positive. Um this is also something that our sort of theoretical understanding of has improved a lot in the last couple of years. Um, and we now actually know in particular cases uh that models are capable of implementing real learning algorithms uh internal to their hidden representations. Right? So if I train a model uh to perform something that looks like a classification task or a regression task or whatever, at least on sort of synthetic data in a in a clean room setting, um there's lots of both sort of empirical and theoretical evidence now uh that these models are actually running something that looks like gradient descent or some other learning algorithm uh sort of inside their weights uh where there's a the parameter vector of a smaller model being stored inside the hidden representations. uh and that thing is getting updated across layers as more examples get ingested. Now whether this is what's going on inside big models, nobody knows. Almost certainly the answer is uh it's complicated and that there's both something that looks like real learning and something that looks like task retrieval uh and you know all kinds of other weird complicated heristics that we don't know about. Um the main thing that I want to make is just that this is an area that's moving very quickly. Uh these capabilities are changing very quickly. As recently as 2222, I think most people thought in context learning was not actually learning. And now we think models are actually capable of doing at least in some limited way uh acquisition of new skills in context. Um okay, taking a step back, what does this actually mean for the ways in which we use models? Um right and if we remember back to the sort of highle picture of uh the history of the field that we were giving at the beginning um we've gone from a world where we would go out and we would collect targeted data sets uh for all of the different tasks that we care about and trained some task specific model um to really just having one model for all of the different tasks that we care about and only enough training data uh to actually specify to this big pre-trained model what it is that we're trying to do. And obviously the only way in which we can do this or the main mechanism at least by which we're going to be able to do this is if the data set that we use to train uh that initial big model already contains evidence and information about how to perform most of these specific downstream things that we want to do. uh and you know so in some ways maybe the most sort of surprising uh empirical discovery of the last few years has just been that if you kind of blindly or you know maybe not totally blindly but uh with a limited degree of sophistication scrape the internet you find examples for translation tasks and summarization tasks and entity recognition tasks and sentiment analysis tasks or at least sort of tasks that are close enough to them uh that you learn to acquire all of these capabilities. Um and more than that, right, uh models get the ability to uh you know, not I I should say just do sort of standard tasks that have names for them and standard labeling things that have names for them. Um uh but to do more complicated things that actually require sort of novel step-by-step reasoning for individual problem instances. Um, and this is again something that both uh took a certain amount of scale to discover and a certain amount of actual work by researchers to figure out how to get models to solve some of these more complicated reasoning tasks. So, we're shifting gears a little bit right now when I've been, you know, up till this point when we've been talking about a task, we've been talking about something like this, some standard thing with a name where you can just look at a sentence and basically spit the answer out without having to do too much thinking. Um but obviously there are also lots of things that we want to do with models uh that don't look like this that require more complicated reasoning uh that you know take people a couple of seconds to solve rather than a couple of milliseconds to solve or even longer than that. Um and so a natural question to ask once you've seen these kinds of capabilities is whether models also acquire the ability uh to solve some of these harder reasoning problems. Um, and if you, you know, again, around 2022 and in a totally naive way just asked a model, uh, a question or a state-of-the-art language model a question like this, for the most part, the answer, uh, would be wrong. Um, why is this the case? And you know, I think one piece of intuition to have here, right, is that no matter how deep you're building your transformer, uh, or whatever the other whatever other architecture you're using for your language model, um, you only have a finite amount of sort of computational resources available to you between the moment when you read the last word in the input and the moment at which you're expected to generate the next or the first word in the output. And so if you think about a task like this, right, really the model has to see the word or the the question mark token or the word have or whatever at the end of this question um and do all of the work that it needs to solve this hard arithmetic problem somewhere internal to its intermediate states. Uh and you know not to anthropomorphize these things too much but think about this as trying to solve a problem like this uh in your head with a very very small amount of time. uh and you know I basically I think we should expect or at least not be surprised that models are not able to do this out of the box. Um so what do people do when they solve problems like this uh if it's not just sort of generating the answer right away? Well, it's some amount of step-by-step thinking, right? Either writing out the intermediate steps of an algorithm on some sort of scratch pad uh talking out loud through a solution or so on and so forth. Um, and maybe unsurprisingly, uh, something that has, uh, been observed again once you get to language models at a sufficiently large scale is that this kind of step-by-step reasoning, uh, works for them as well. Um, and so this has come to be called, uh, chain of thought prompting. It's something that I think the like LLM hobbyist community uh, was aware of before it entered the academic discourse. um but uh where uh we now give models uh inputs that force them to reason more sophisticatedly again by just changing the inputs to them. Right? So just like in the sort of fshot prompting case we're interested in improving the capabilities of models and the only mechanism by which we can sort of control their behavior is to change the inputs that we give them. And so now when we're giving demonstrations of a task, rather than just saying here's the question and here's the answer, we're going to say here's the question, here's the bit of the answer that we want the language model to generate and the first piece of that answer should uh include not the final, you know, sort of output to this question, but this reasoning steps that will eventually lead us to the right answer. Um, and so, you know, going back to the example that we had on the previous slide, uh, rather than just saying, here's the question, and 11 comes out, we say, here's the question. Roger starts with five balls, two cans of three tennis balls each is six balls. 5 plus 6 is 11. The kinds of things that you might imagine saying out loud to yourself or writing down um, in the course of doing a problem like this. And when you do this, right, when you prompt models in this way and then you give them new questions, they're pattern matching against the context. And just like we saw in these few shot prompting examples, uh what you get is an answer that first includes a bunch of step-by-step reasoning, maybe following a different pattern from the one that we provided in the input. Um and then the answer at the end. Um and obviously this is something that you can combine with all of the other tricks that we were talking about uh in the context of fshot prompting. Uh meaning you can also provide uh highle description of the task. You can also provide a sort of meta template, the kinds of things that make good step-by-step reasoning instructions, information about the granularity of the reasoning steps and so on. Information about the format in which the final answer should be presented so that you can parse it out um and so on. Um but the really the high level idea here is just rather you know between our sort of x's and our y's we're going to interpose um some reasoning steps and if you do this in the input then you expect to get it in the output as well. Um uh right and so this is just writing this more formally. Right? We're going to sort of uh between our x's and our y's put some z's uh that tell the model show the model how to reason. Um and because uh everything's in the input because we're in this very few shot paradigm that we were looking at over here. Um it's really not that hard for you as an experimentter or a model builder or whatever to write five or 10 or 100 examples of these things to provide as input to the model. Um so this works in a bunch of different uh regimes. You know again I think something that maybe is worth emphasizing is that this is a capability that uh arose you know just sort of from a historical perspective a little bit unpredictably uh and only at a certain scale. So I knew of at least three different groups at MIT uh around 2019 who were trying to do something like this and it wasn't working because the models that we had at the time weren't big enough. uh and then you know GPD3 came out and people tried it again and uh and all these things work. Um you can do this for free response math questions, you can use it for multiple choice questions, you can do it for common sense reasoning. Uh and you know really this is a a a sort of meta trick that you can apply to almost any domain of interest. Um uh so here are some more examples of uh of places you can do this. It's really quite general um and it's really really really quite effective especially on these problems that require a lot of step-by-step reasoning questions. Yes. >> Scale made it possible. >> So why does scale make it possible? I mean again I think even the well so there's a couple of different things right? is recognizing uh that a given input uh is sort of asking for a model to perform some sort of abstract reasoning capability and not just uh to pattern match against a template, right? And so I think a meaningful difference between um like some of these examples and some of the things that we were looking at over here is that what you put into that Z in principle looks very different from problem to problem, right? You're not just filling in holes in a template. uh and that's something that small models you can give them this data they'll do the copying bit and they won't do the generalization bit and it's like the ability to generalize to new reasoning chains that's the thing that you get uh uh I think it is also the case and this is something we'll come back to at the end that in like the very largest models where people apply these things now there is some large amount of deliberate data collection where people are making sure in the training set there are enough examples of these kinds of step-by-step thinking out loud >> uh another question yeah It seems like you can enforce like multi-step reasoning uh through chain of thought prompting and I I'm wondering where does reinforcement learning fall into that. Is that just to make it marginally better? >> Okay, so there's a question about reinforcement learning. We'll talk about reinforcement learning uh at the end. Cool. Um good. And you know, I think maybe the main thing to take away from this is just that uh you can really for some of these hard reasoning problems, especially in larger models, uh dramatically improve the quality of the outputs that you get just by sort of prompting models to to think step by step uh in their outputs by providing examples of the kinds of reasoning chains that you want them to go through. Um, even more infuriatingly, right, as soon as this uh the sort of first round of chain of thought papers came out where people were sort of manually authoring these uh Z's, these reasoning steps, um people started to ask is there just like some magic incantation that you can give models such that I don't have to demonstrate the reasoning steps, you know, a new for every new task that I'm trying to perform, but I can just prompt the model in a totally task generic way that will cause it to do this reasoning and then produce a final answer. Um and uh uh so as you might expect, somebody basically just did a brute force search in the space of uh sort of magic incantations that you could append to the beginning of your answer in a language model. Uh they found let's think step by step is the best string for at least the models that existed in 2022. Um and uh again just maddeningly uh this works super well. So you can take a model uh the way to read this chart uh is you can take a model that gets 18% on an arithmetic task and bring it up to 80% on an arithmetic task just by typing let's think step by step uh before you