Lec 10. Architectures: Memory

Key Takeaways

Um so today we're going to be doing our last of this minieries on machine learning architectures or deep learning architectures. Um so previously we talked about you know CNN's DNN's things like transformers and today we're going to be introducing the concepts of memory and sequence modeling and we have seen different types of ways to model sequences in the past right we talked about like 3D CNN's that have time or transformers that have a sense of time um but today we're going to be introducing a new class of um types of models that handle memory and sequences in a different way um and this will be the first class of architectures that we talk about that's actually um quote unquote touring complete. So this is actually the most general um framework of the architectures that we're going to be introducing. So we'll first motivate sort of again why we think we might want to model sequences, model something like time or order. Um and then we'll talk about how we might use CNN's for sequences. We'll introduce the concept of an RNN or recurrent neural network. Um and then we'll go through um an extension or a specific subset of RNN's called LSTMs. That stands for long short-term memory. Um and these actually seek to specifically avoid one of the common issues with RNN's which is forgetting. Um and then we'll introduce the idea of sequence modeling and and long memory and start talking about some things like auto reggression. So if I look at this, this is a a single frame from a video. Um there's a lot of things that computer vision can understand from this frame alone. Um so where for example and and our human brains kind of do the same thing, right? So if you look at just this frame, um can anyone tell me what this is a picture of? Anything? kids playing. Yeah. Right. Um and maybe they're playing in a classroom, right? So you can understand something about the scene. This looks like maybe a kindergarten classroom. Um we have computer vision models that can understand stuff like objects that are seen in that single frame. Television person chair. Um and we can start doing things like question answering, talking about specific attributes of those objects. What color is the chair? That chair is red. Um but so far none of the things that we've talked about have have really explicitly tried to understand for example relationships between things and how we might predict into the future. So for example what will the girl do next? That's something that from this given frame I don't know probably we we have some ideas about different possibilities but we might not even know um because we also rely on temporal signal. We rely on context and and information. But if we watch this video and we kind of get to the point of that frame, we can sort of guess what's going to happen, right? [laughter] [gasps] So, I don't know if any of you have ever had this experience of a chair being pulled out from underneath you and it's incredibly disorienting when something is not how you expect it might go. Um, but watching this video, there hits a point in the video where we're pretty sure that the kind of likely thing, which is that the girl sits in the chair, is not going to happen, right? Because the guys moved the chair. There's sort of these complex relationships between the actions. Um, and then, you know, things like predicting she's going to fall, that uh she believes that the chair is there, uh, why she believes that the chair is there, whether you guys are going to laugh at it. [laughter] Um, and so all of this is really like very complex and and nuanced understanding about the scenes, the objects, and things like intent, which um we could argue about whether these machine learning models that model sequences are really getting at understanding things like human intent, but they do start to try to get at this sense of what is going to happen next. So sequences can be a lot of things. We talked previously when we introduced um CNN's about um the idea of adding a temporal dimension to um things like images. So now you have like a video. This might be uh pictures from an Italian city uh square over time. You can also think about sequences of language. An evening stroll through a city square is a sequence of words. Um or you could think about, you know, sequences of audio signal. me saying an evening scroll stroll through a city square is a sequence, right? And looking back again at the architecture that we've gotten very familiar with convolutional neural networks, um we can extend those into the space-time dimension and make a cube like a space-time cube. And and again here we're we're showing this in three dimensions to be for simplicity, but actually um it's really four dimensions, right? we're extending this fourth temporal dimension because we have multiple inputs of red, green, and blue for each of those dimensions. Um, and then time just becomes another spatial dimension in that tensor. Right? So you have a 4D tensor instead of a 2D tensor. And so now when we're thinking about how we might actually try to represent time, there's different ways that we can think about capturing time within this cube. And and one interesting way that you can kind of model temporal structure could be something like taking a slice through that space-time cube. So this is what it looks like if you build sort of this new image that takes a single horizontal sort of pixel slice and then runs that through time. So what are we looking at here? What are these sort of stripes likely to be? Yeah, people walking, right? like and then walking at different angles to the camera sort of with different trajectories or different speeds. Um or you could also think about taking sort of a vertical slice through time, right? So now this is showing um you know for a given sort of position horizontally or vertically in the frame like who is going through that position first. Interestingly even from a very long time ago they started modeling images like this for the Olympics um when they were trying to figure out who was the first person to go across the finish line. If you take the finish line and you make that a a single pixel element, you can see exactly what pixel crosses first and that represents the person who won the race. Um, and so there's lots of different ways you can think about representing or modeling time. Um, and then if you actually want to think about how you do convolutions in time, if we take sort of a very simple 1D perspective of this, so now imagine we're just talking about the scalar perspective. Um then what you would do is you would similar to before you would learn some weight matrix over some given fixed time window. So in this case three um sort of elements in that sequence and then you can think of this as a sequencetosequence model. So instead of before where we took an image or something and we would run this across the image in these different patches and then you were sort of constrained by the structure of the image here actually now you can run this in time. So now you're taking input from these sort of three um different positions in the sequence and you're generating an output position. Now you could slide that and you could slide it arbitrarily long. You can take any arbitrary length sequence and you can generate an arbitrary length sequence. So these convolutions in time are not sort they're not sort of constrained. I mean they're only constrained by sort of the length of your input sequence. And you can imagine that if you had something like a static camera that was taking video of this classroom continuously that you could sort of just keep running these convolutions in time and there wouldn't necessarily be a point where you needed to stop like we're constrained by the architecture itself. Now if you consider something like more dimensions right um things like you know these RGB videos um or add even additional dimensions in that RGB channel something like MRI that captures 3D volumes over time or hyperspectral satellite images that can have something like you know 384 different bands. Um you can continue to think about these convolutions but now where time is potentially this infinitely long dimension. Um, and if you're doing this, then what we discover both empirically and also it kind of makes sense intuitively is that as that time sequence becomes arbitrarily long, it's hard for our memories of what happened at the beginning of the sequence to persist. Right? And one of the reasons for this with a convolutional approach is um is the following. So here say we have our temporal convolution and it's taking in some input. you know, we're throwing this showing the scalar vision, but for the sake of uh intuition, um this is a picture of my cat, Frank, who is the cutest cat in the world. And um so say I'm walking around my house taking a video and I start at this point in time and then I go forward and now in the future I have again remember because it's convolution that same weight matrix and now I want to understand what's in here. But the thing is I don't get I don't really have from this um fixed time window any inputs coming from the beginning, right? I' I've only got the window that I can look at. And so based on the information I have, well maybe I look at this and I predict tiger because it's orange and it's outside and you know cats are usually indoors. Um so the idea of something like memory or or trying to extend sort of what we forget move beyond sort of these fixed time windows is to try to capture that that context be able to sort of remember what you saw in the past. So here then maybe you'd have some memory unit and that would actually give you some context give you information. You know since I'm walking around my house it's unlikely I'm going to see a tiger but I did see Frank before so it's maybe still Frank. Um and so this type of you know identification with the context of what you've seen in the past from like you know the same for example sequence is really the the intuition or sort of the motivation behind the de development of what we call recurrent neural networks or these RNNs. Um so transformers, CNN's everything so far mostly had this sort of fixed window in time right there was you know transformers these days you know we can talk about that that context window is getting very very large but it's still fixed. Um but a recurrent neural network makes that infinite right and the way that it does it is you introduce the idea of a hidden unit. Um so now you have for example from this convolution you send you you run your model through the convolution you get some value and then you learn a way to take that value that hidden that hidden unit and predict the output but you also take that hidden unit and you send it forward in time to the next time step. So here now as you're convolving through time you're actually taking in both that new temporal input and your memory from the past. Um and so here if we want to get look at this in a little bit more detail um again just thinking of everything as scalers just because it's it's really easy to visualize this way. Um so a version of like a scalar temporal prediction problem might be something like predicting temperature in Boston tomorrow. Um so here now what you're doing is you're representing time as some sort of discrete sequence though you can apply these to continual sequences and the next hidden state. So now we have you know our inputs running through some model to get some hidden state and then we send this output and we also send that hidden state one time step forward. So here it's as you're kind of running this through what we do is we say your next hidden state is represented by some function f that will depend on the previous hidden state and the current input and then that f that function that's shared over time. So we're not um you know that that function is is again it's fixed over time. the way that we push information forward is fixed over time and that's something that gets learned and it's independent of time what that function is and then similarly we'll have learned some function G that will map from our hidden state to our output prediction for any given time step and that's also going to be shared over time and so here you'll note that this is the first time we're not seeing a DAG Right? This is not a directed asylic graph. We have a loop. We have a cycle because you're going from that hidden state to the hidden state again. Um, and that's recurrent, right? And so you can kind of write this recurrent function for what your hidden state will be. That's some function of your previous hidden state and your current input beta. And then again, your output function is going to be um g mapped from your hidden state. I think it's a little easier to interpret when you think about the computation being unrolled, right? Because then actually for any given time point, it's still a directed graph. It's just that you have this recurrence loop. But so again, this is kind of like the simplest version of this. So now you assume all of those functions are linear layers, which we're very familiar with at this point. So now what you're learning is you're learning U which is mapping your input to the hidden to um to sort of the next um point. You're learning some V that's mapping your previous hidden state in um as well and you're going to combine them and then you're learning V that maps your hidden state to the output. Um and so now here you have some something like this, right? You'll have some nonlinearities in there in the sort of simplest form. So your new hidden state is just nonlinearity applied on top of W * the previous hidden state plus U times your input data at the current time point plus some bias term. And then similarly the output again is some other nonlinearity plus time V plus the current hidden state plus a different bias term. So if I got rid of W here, what would this be called? If I got rid of the the kind of recurrence dimension, if we were just looking at that mapping between the input and the output, I mean since it's scalers, it's a little silly. I mean, it's like a one-dimensional convolution, but imagine if you extend these beyond scalers, it starts looking a little bit like that multi-layer perceptron that we're very familiar with, right? So the only difference here is that that recurrence that's actually sending this information forward over time. And really I think the point here is that the hidden state is intended to capture the most relevant historical information. And what we're learning in W is how do we actually understand what is relevant? Like what do we need to keep? And of course this can be extended to be deep, right? You don't need to only have a single hidden state. you can actually stack a bunch of these on top of each other, have many many sort of different um hidden states, hidden layers. Um and then now these could be any dimensional tensor and there are many many many works um that have looked at different ways to structure these deep recurrent neural networks. But how do we do back prop if we don't have a DAG, right? Oh yeah, >> the recurrent step, how many time steps do you have? So you're like unrolling it, but like how do you know what this >> to infinity and beyond? >> Yeah. Is it a hyperparameter? Is it something that gets into the network to see like what's the optimal number of things? >> So the way this is structured, it doesn't have a sense of stop. I mean it'll just stop when you stop doing it, right? Like when you stop putting in input data, I guess that's when it stops. But there isn't a sense in this model necessarily as it's structured that has an end, right? because you have that weight that's passing the previous hidden unit into the next one. So if you don't give it a next step, I I guess that's the end. But it doesn't know what stopping is >> in practice. Are you repeating it like >> um >> I mean it depends on the problem, right? Like I mean if you're you if you're using these types of um RNNs to do something like model video or model audio clips like they probably have an end, right? So you just stop running them when you get to the end. Um but you could also think of it in like a continual sort of way like maybe um you're continuing to get temperature data every day and every day you predict the next temperature and that's going to go until the world ends. >> Um does that make sense? Yeah. How does this architecture capture the lack of memory in a way like >> um right so the the question was how does this architecture capture memory throughout that very long infinite length because that hidden state only depends on the previous state so it's recurrent so this state only depends on the previous state but the previous state depended on the state before that and the previous state depended on the state before that so you are capturing in every single hidden unit as far back as you have. But we'll talk about how even though that's theoretically true that this construction actually makes it really difficult to learn things across like very very long um time time horizons. Um mostly because you have some limited capacity there and then we'll also talk about actually some of the mathematical limitations of this formulation. But yes, great question. Cool. So how do you do back prop if we have a DG? if we don't have a DAG, right? Um, so the simple practical thing, historically this was something that really tripped people up. Back prop is intended for feed forward networks and now we have loops. Um, but the hack is you just you just decide when you're training what your sort of fixed time window that you're going to propagate back through is. And then now you unroll it through that time window. So you'd say, "Okay, guess what? my time window is going to be this far, right? I'm going to go from V back to XO. And then now you've unrolled it. And of course, you have some shared parameters in that network, but it's actually now again a directed asyclic graph because we're stopping the point where you're going to kind of keep going back forever. And so here you can actually just again if you're going to try to calculate you know the influence um on the output um you know d of x out with respect to d of x in um over you know from zero to time t. Um you can do that by just sending now through this directly as cyclic graph back to that um that zeroth element just by unrolling as far as you want to unroll. And that's a parameter that you would sort of select or tune, right? How far are you going to send your back propagation? And then once you've kind of unrolled it and you've kind of decided on this fixed time window during training, then you can just use the same chain rule. Um, and you can calculate what that gradient is going to be. Um, and the interesting thing here is it does kind of limit maybe how well we might learn things further back in time because of course like the information is still available, right? you're still, you know, at inference time, you're still going to have that previous information getting passed through or getting encoded. But if you only trained the model to send signal back to a a given sort of fixed time window, you could see how intuitively it might teach you to pay, you know, um to kind of only capture some some sort of limited um temporal window of information. Now um in practice there's also other complexities as we increase that time window in terms of what um what that means. Yeah. >> Increase the time window is it then harder to predict if you only use a couple of inputs at first. >> So >> oh interesting. So the question which I think is an interesting one is um if you've trained this model using backrop over um maybe a time window of like 20 now if you only have you know a sequence of three inputs um is that actually limiting your ability to do it shorter? Well so potentially no. um because this is just looking at one component of that um of that gradient. But actually um if we if we think about this um you would you so this gradient is also going to be like this is just the input of that on that and that's like the end of that time limit. But in practice, you would also do everything up to this time point. And we'll go through that in the next couple slides. So you would still be getting that signal from sort of everything within the time window that you're looking at, not just going back to the farthest point, which makes sense because it might be much more helpful to have some of those more frequent um more more recent um gradients than maybe the further ones. Yeah. Uh to go along with that point for like this example would it be like if X in is today's temperature X out is like six days in advance temperature but we are actually inherently predicting like 1 day, 2 day, 3 day, 4 day, 5 day >> to get there. >> Yeah. So what we're showing here is just the fact that if you do unroll this that now you do have this directed graph again. But then like in practice essentially what you're saying is you would only take the gradients up to like t minus whatever x0 was. So this is like one two three four five six. So you'd only be taking those gradients over a sequence of six. And then now for the next point you you'd shift it, right? So now you'd only go to x1 and then for the next point you'd only send gradients from x2. So you have some fixed time window that you're unrolling that you actually accept gradient. You calculate gradients over. >> Yeah. the truncated time window is some approximation of an infinite time window. >> I think that's a reasonable assumption. So the question was um is that fixed time window a reasonable approximation of an infinite time window. Um I would say it's it's maybe a bit of both. So again um I'll walk through it in a sec, but basically with this formulation, the longer time window you have, the harder it is for information to actually make it that far with these weights. Um so at some point actually like extending the time window is probably by construction here not super beneficial. Um yeah. >> Yes. >> I wonder is there ways to optimize what's the time window you need for a specific task or is it just like >> well it's a hyperparameter right so you know when you would look at the literature for this you know people would would ablate that hyperparameter for example um and and I think sometimes it it might be something that you could actually bring some domain knowledge to decide right like what what actually makes sense here. Um so a few conceptual things. If you are going to unroll this um remember that that um that function that maps uh the hidden states forward through time is fixed over time. So those W's are shared. Um so now what does that mean for back prop? So we have this sequence of videos, right? Um, now the loss is going to end up being the sum of the losses from each of those predicted frames, right, over some sequence. And to optimize, you want to learn how you would change W, U, and V so that you know the sum of the kind of loss over that sequence is um is minimized. Right? This is how we do gradient descent. But you want it for the entire sequence sort of within I guess the time window that you're considering. So we want to figure out how we want to change all of these. But of course W is as again like shared over time. So we have to do back prop over shared parameters. So we're going to sum up all of these sort of individual losses. And now we're going to be passing this gradient back. And now um does anyone remember what it looks like when we have these back prop overshared parameters? Right? Because here we have um you know the losses all getting summed up, the gradients all coming down and then there's this kind of waterfall backwards where those gradients are passing through all the W's. So does anyone remember what we do when we're taking gradients over shared parameters? Been a couple weeks. Does this look familiar to anybody? Yeah. So here, you know, there's this idea that if you have kind of shared parameters and they're sort of being shared throughout a network that it's actually the same as branching structure in a directed as cyclic graph. And so when you're doing a gradient, when you're taking the gradient over this branching structure, you're actually just summing the gradients um for the parameter each time that parameter occurs. So for that W, we're going to just be summing those gradients over every single um element that we have that W in. Cool. So here we have W in all these places and now we have kind of this branching structure. And then when we're calculating um the change in that that loss with respect to W, it's the sum over the change in the loss from the gradients from sort of each of those. Cool. And the nice thing is if you're using something like PyTorch, this will happen automatically in autograd as long as you don't do deep copies, right? As long as you're sort of pointing to the actual by reference that actual variable, then autograd handles this kind of um like summing over the gradients within the thing reasonably well. Um so make sure you're not doing deep copies and and it all works out pretty nicely. Um and then I think another point is um you're summing over the past inputs, right? So, so if you're calculating the loss with respect to this one, you're summing over all of those. If you're calculate with respect to this one, it's only over sort of the ones previous to it, etc., etc., because that's only going in one direction. Any other questions about back propagation through an RNN? Cool. Let's move on. So, problem of long range dependencies, right? We talked about well as this time window starts to stretch towards infinity it seems like it would be hard for this thing to remember um you know why not set your time window to infinity you know why not remember everything um so it turns out first memory size um will grow with with that t right so if you need to use some sort of infinitely long sequence as an input that means it takes up a lot more memory and that kind of memory is is non-parametric right um you there's not like a finite set of parameters we can used to model infinite time. Um, RNN's make a what like a marov assumption which is to say that the future hidden state only depends on the immediate preceding hidden state. Right? So you know we say okay it's recurrent but we only depend on just the immediate hidden state before that. And then by putting the right information into that hidden state you can model dependencies that are arbitrarily far apart in theory. Right? because the model learns with W how to keep the relevant information or at least that's the hope. Um so here what you're saying is actually in order to capture those long range dependencies we have to propagate that information through a very long chain of dependencies and this ends up being uh a bit difficult. Um so let's walk through this this very big chalk. Um so for simplicity just to walk through the what the math looks like we're going to assume we don't have any nonlinearities and all the bias terms are set to zero. Um so say we have you know h sub one. Um, and that's going to be equal to, like we said before, w of h sub0 um, plus u of x sub1. So, h sub one, the hidden state at that first step is going to be equal to the weights times the hidden state of the previous step times plus these weights times the new input. Now if we write out what this recurrence looks like. Now if we say h sub 2 now that's going to be equal to w of w of h sub0 plus um u of x sub1 plus u of x sub 2. Right? So what do you notice here? Now we have a w^2 term essentially. And so now if you do it for another time step, you're going to have w cubed etc all the way up to w to the n. Right? So if you have this quadratic relationship with your weights matrix, what happens for values in that weight matrix that are small? They'll go to zero, right? And anything in that weight matrix that's big will be infinite. So what happens is in order for this to be stable at all during training, all the values kind of need to be close to one um which means it's kind of hard to capture really variable information in there. And then additionally kind of no matter what, even if you're just a little bit below one, when you go to infinity, it will go to zero, right? because you have this um this sort of exponential relationship. And so what this means is often that old observations in a recurrent neural network are forgotten, right? Um and that the stochcastic gradients become really high variance and you can either have these vanishing gradients or exploding gradients. And so this is why um we sort of started to look towards LSTMs. Yeah. >> So I'm wondering that fix this problem and if not why not? >> Um so the question was um we previously talked about the spectral norm um and using that as a as a way to try to mitigate things like these vanishing or exploding gradients. Um, so you could try to build something in here that would try to normalize these weights so that they would be again like you know close to one essentially I mean or or not not go to sort of infinity or not. this function is um like this part here like the fact that because of the recurrence relationship you're taking an exponential in the weights matrix. Um you could take a spectral norm over that exponential, but you're still going to take the exponential first. So that might help you with the exploding gradients problem, but it's not going to change the fact that the small things are going to go to zero. Does that make sense? Yeah. Cool. Um, and actually if you I I thought this was kind of a fun blog post um that specifically takes uh it's from a control theory professor, but they're looking at stability analysis in recursion. Um, and so if anyone's interested in sort of the different ways that people think about what makes things stable or not stable um in a system if you have recursion um I thought this was kind of a nice uh a nice blog post. Yes. So, you said gradients can explode or vanish. Do they both correspond to old memories being forgotten? And also, as a followup, like isn't it not a terrible thing that old memories are forgotten? Because that's sort of what humans do? >> Well, that's a So, so the question is, um, if you have exploding and vanishing, do they both correspond to memories being forgotten? And no. Uh, the exploding ones correspond to unstable training, which we also don't really want. Um, and the vanishing ones would be old memories being forgotten. Um, and then, uh, do we actually want to remember things over long time horizons? And that's kind of the universal question, like do we need this? And I would say that there are definitely cases where we don't, and then there maybe are some cases where we where we do, right? So, so an example might be um, think about temperature. So, we have kind of these daily cycles of temperature. So we'd like to remember that the temperature got warmer in the middle of the day than at night yesterday. So if we're looking at predicting temperature on some sort of fine grain horizon, you'd want to be able to remember at least periodically a day back. But then also you have these yearly cycles of temperature, right? Where it's hotter in the summer and cooler in the winter. So maybe if you're trying to model temperature, you also want to have this, you know, yearly dependency. And then you have things like these weather cycles like El Nino that happen every seven years. And so so depending on what you're trying to model, there may actually be a lot of value in the longer and longer term memory if there's signal in more and more time scale. Does that make sense? Yeah. Um, cool. So this is why we introduced LSTMs. Um, which is to try to deal with forgetting to try to deal with this vanishing gradient problem. And the reason the vanishing gradient problem is more of a problem than the unstable um exploding gradients is exactly like um what what this guy over here mentioned. there are things we can do to try to force you know to take normalization over the gradients to to keep them from exploding different ways that we can kind of constrain it or regularize it or clip it but we can't we can't do that when things go to zero um so LSTMs are essentially um designed so that the default behavior is to not forget and then you have to learn to forget um and so the default is the identity, your bias to like not forget anything and then you learn what to forget, kind of like garbage collection, what to throw away so that you can add new stuff. Um, and so um, if you look at this, so this is kind of like the standard recurs recurrent neural network, right? We have some nonlinearity. Um, this is a this is kind of a simplified version where it looks like we're sending in the inputs. There's a function here that's mapping this in and then we're sending out some hidden state, but then there would be some function mapping that to the output. We're just only looking at kind of the hidden layer component of this model. And so what an LSTM does is it builds a little memory controller that decides what to save and what to delete. And it turns out this is still touring complete. And it's actually very close to a touring machine. It's kind of like that idea of a hidden tape that you can read to or write from. Um, and so now every single one of these sort of functions that maps us from our inputs and then to what the sort of next hidden state should be has this little controller built in. And so let's go through the different sort of components of this thing. So first we have this cell state C. This is a new thing that we're introducing for an LSTM. And this is kind of the tape in the touring machine. And this cell state is what gets modulated by the t part that decides what to forget and what to add. Um so then we'll have some function f of t um and this function uh will you'll operate over you know applying your sort of previous weights to the previous hidden state and your current state and then your biases and that um that is going to be nonlinearized by something like a sigmoid. So if it's large it goes to one and if it's small it goes to zero. And so now if you're taking this thing now if you multiply by zero so if it's if this this um value ends up being small um coming out of the function then it's telling you forget this thing um and then so what you want to do is give high values to things you want to remember and low values to things you want to forget. Um so that's like this what this component of the LSM is is in charge of deciding what to remember and what to forget. And then each element of that cell state essentially because of the um the way that we're using a sigmoid and not something like a relu which would be sort of unbounded is it's essentially the identity function with some masking of what we want to forget. So then there's this next component and this one is determining which indices we want to write to and what to write. So now we have um essentially this I that is used um to determine basically which indices we're going to write to and then the C of T um is going to be what how that information kind of gets added to the cell state. Um yeah and then finally we need to do the actual forgetting and so this is where this kind of comes in that is what what's quite different is for the first time now we have these multiplicative relationships right so you're multiplying you're taking the previous cell state and you're multiplying it by this f subt which is going to be kind of either ones or zeros or or very close to and then so now that's kind of updating the the cell state in the first place and then you're going to add this new information. So you now have this I subt multiplied by this um the the new cell state C subt. So in this case it's removing information and adding information. Okay. So now after you've actually updated that information in the cell state now we're going to decide what gets output to the next hidden state. um it's not mathematically similar but it we just learned about transformers right where we have sort of the keys val um keys um queries and values. So in that in that case when we talk about this like sort of value projection these value matrix um that's kind of telling you how you should output that next state or how you should output sort of um the sort of output of that thing. And this feels to me like on vibes kind of like kind of like values, right? You're you've already decided what this you've updated the cell states information and now you want to say okay now what how do we take that and update it to the next hidden state. Um and then note so we have now we have both these hidden states which are getting passed through but we also have that ticker tape that cell state that's also getting passed through. So there's kind of two things going on. Cool. So really this important point like the important thing that this has beyond that vanilla RNN is the fact that you have almost this gating functionality that's provided by multiplicative updates to the cell state where now we explicitly learn what to remember and what to forget and then you multiply by one. So you have this identity instead of multiplying by yourself which removes the vanishing and exploding gradients. We no longer have that exponential. Um, instead you're only ever going to do something like the identity or remove it and add something new. Um, and it's sort of you because the default is the identity. It's kind of like a skipped connection or a residual connection in a resident, right? You have this component where you could just learn throughout this entire thing to only ever send the identity, right? Like, okay, just just send the same information forward all the time. Awesome. Any other questions about LSTMs? Now, this part's like a little fiddly. Um, so, you know, definitely feel free to go back through or um or visit uh this really nice blog post uh that we pulled some of these visualizations from from Corsola. >> Yeah, >> it's element wise for the the hidden state. Is that correct? So the multiplication is basically like you're learning this gating function f that's like telling you what to forget and then you're learning this i that's telling you like um what new information to add back in. And so those two things are are going to be multiplied in and they're going to be multiplied in like elementwise in your C depending on what the sort of um size of C is. >> Yeah. be like or something, >> huh? >> Sigma be here. >> Oh, sigma is um the sigas are sigmoid functions. >> Literally literally sigmoid functions. Um yeah. And and the reason that people have to use you could use another nonlinearity, but the reason you use a sigmoid function here is because it's bounded between zero and one. So you kind of want large values to map to one and small values to map to zero. >> Yes. Yeah. Yeah. >> The question was if the weights are on the diagonal and the answer is no. Cool. Okay. Let's talk about sequence models. So, um so we just talked about a bunch of different ways that you can think about trying to model sequences. Um but uh there's this sort of new kid in town of um what we call an auto reggressive model. Um, so previously we said, okay, we're going to have some fixed time window. We're going to be sending this memory forward. Things could be infinitely long. But the idea of an auto reggressive model is is quite simple and we've introduced them before. So the idea is you have some beginning of a sequence and then you ask to it to predict what comes next and then this is the auto reggressive part. You add that in and now again you ask it predicts what comes next. Right? And this idea of sort of continually, you know, predicting what's next, predicting what's next. This is how we think about doing, you know, modern generative AI. Um, a lot of these things like chatbt, these vision or these language models will explicitly be um trained to be auto reggressive. Though, of course, there's no reason why you can't take the same framework and instead um, you know, tell it to fill in the blanks or something. So when you're actually training one of these auto reagive models, [snorts] um you might have a bunch of different examples of sentences, right? And then you know the beginnings of sentences and then the next word. And so then the the learner has to learn from the beginning of a sentence what is the next word. And then when you're actually sampling, um, you would do something like send in some new beginning of a sentence and then use your predictor sort of until you want to stop or until the predictor decides to stop to predict the next thing. Yeah. >> You do you mean you like if I predict something at time t, I take that prediction past that. So now, >> so now you predict something at time t and then you take that prediction and you add it to now what's your input. So now your input is everything up to t minus one and your prediction for t and then you ask it to predict t plus one and then you add it in. >> This is what you sampling is used there. >> Yeah. Yeah. Yeah. So then actually the sampling part like this is that's exactly what would happen, right? You would you would send in as your input color this green ideas sleep ask your predictor what comes next says furiously and then you would take that furiously and you could edit in two and you could say okay well what if I said colorless green ideas sleep furiously then what comes next and then you know this is this idea of auto reggression that you're sort of using your predictions adding them to the input and then predicting again and again and again >> is that one way you could evalate just these like samples and see what it produces. >> Yeah. So, I mean, you could think of using an RNN to kind of predict predict the next word. So, like your output would be what we think the next word might be. But what an RNN doesn't have um is like this explicit like loop back where now that output from like up to t plus one becomes the part of the input at the next time step. Does that make sense? So it's not like innately something you would do in an RNN, but you could totally like loop it around that way and and kind of construct it. >> Yeah. >> So with this model, the predictor outputs sort of a distribution and then you sample one element from this distribution, right? I guess but what if you put [laughter] >> um anyway continue. What what if you instead of sampling one and addit adding it to the input, what if you put the distribution in >> um Okay. Yeah, you're you're just begging the question. We're about to get into the the probability models. Um but the question was basically like uh what we said what I just said was you take you know the prediction and maybe that would be a distribution and you're picking the maximum and then you put that maximum in like what if instead you put the distribution of possible um next options in and we'll talk about different ways that people think about doing that. Um yeah. So essentially what we're talking about is um or one way that you can think about doing this is trying to learn some probability of the next word, right? And you can it's you can take that and factoriize the probability of that joint distribution. Um basically you're just reformulating the conditional distribution, right? The probability of the the next word given everything we've seen before. is true for any probability distribution. You you factoriize them out and you get something that's um multiplicative um and uh and is somewhat you know still auto reggressive right because you're taking the probability of every previous thing up until that point. So you can think of this the sequence models in um in like transformers when we talked about it in the transformer lecture last week as modeling for example the probability of a sentence and then that could be generative right so here what this looks like in practice so say you're trying to look at the probability of once upon a time that's the same as the probability of once the probability of starting with the letter the first word once times the probability of getting a pawn pawn on given that your first word was once times the probability that you'd get a given that you already had once upon and then you know etc etc. Um so this is kind of how you would ma model the probability of this sort of given sentence. Um the likelihood of a sentence, right? So what's a function that we could use that would map the probability of the next word given the previous word? maybe some of the things that we've talked about in class specifically the probability. >> Yeah. Go ahead. >> Logistic regression or some steep thing going into logistic regression. >> Um so you said logistic regression. >> Softmax. >> A soft max. Oh yeah. Yeah. So a softmax takes an input and then gives us something that might not be a probability distribution but at least sums to one, right? Um so what is one of the ways we frequently use a softmax when we're doing classification? Yes, exactly. Um so we could just treat this as a next word classifier, right? And we I think we talked in the hacker lecture about how a reasonable starting point for a lot of things is just being like all right, let's let's see if we can formulate it as a multi-class classifier. Um so here then the input is once upon a and then you learn a next word classifier that would probably by taking a softmax um give us some distribution over the likelihood of the next words right um and so now here a standard multiclass classifier will use some neural net that softmax will squish the outputs into a probability mass function right non- negative vector sums to one and then you learn that by penalizing it via cross entropy. Um, so how would you map words to classes? How many classes would there be? Um, lots, right? So if you use words and use every possible word in the English language, which there are many more words in some other languages on Earth, and you represent all those words as one hot vectors of size K, K would be about a 100,000. And we know that multiclass classification starts to be a lot less sort of easy to learn when the number of classes starts to scale quite large. Though it is possible for example like we run some 100,000 way classifiers for species in I naturalist right there's about 450,000 species currently in I naturalist we train classifiers for all of them but it need you need a lot of data and it is it is it can be quite unstable. So instead, you know, another thing that you could do is you could represent every character as a class, right? So now in the English language, the alphabet is 26. So now we only have K equals 26. Um, and that's a really reasonable set of classes. Um, but the thing is now your classification is fewer way, but the sequence prediction is a lot harder, right? You have to take a lot more time steps. There's a lot more predictions that you have to take. And um and it's pretty easy for these things to kind of devolve um over time, right? Like imagine, you know, it it starts spelling something weird and then like where are you going to go? Um so ideally you'd want something somewhat in between. And it turns out in practice that what's what's kind of a sweet spot is this idea of um bite pairs, which is two character pairs. Um [clears throat] and so now uh you represent words about you know about a thousand 26 times 26 maybe plus a few more of these bite pairs and that's actually the these tokens of kind of these two character pairs is is quite frequent even in sort of the largest scale language of models that we have. Cool. So let's talk about then what this might look like. So you have this kind of probability this classifier for the next step basically and then you send your class your output prediction forward. Um so so one example of this um this is an example that's taking in some caffeine molecule and then you're you have a model that's trying to output a description of that molecule. So maybe that would look something like this. A mild stimulant that enhances cognitive ability. So you use this caffeine molecule. You have some representation that you use to sort of start your your word generator somewhere and then you would describe this. Maybe this we would call like a molecule to text. So one way you could do this right would be something like an LSTM. So now you have um you have your molecule, you run it through some GNN that gives you some representation of that molecule that you can use to sort of condition on your starting point. Then once you have the starting point, the LSTM is just predicting that next word at any given point, right? And you're taking um the sort of previous things into account. So if you're actually looking to train this now, um the training data is something like a mild stimulant that enhances cognitive ability, end character, like stop predicting. So now for the first time, we've introduced the idea of stopping um when you're done. And then your outputs would maybe be some probability distribution or or sort of likelihood over those next words given the previous words. And then you could train it using something like a max likelihood objective where you want to maximize the probability the model assigns to each target word. Right? You want it to sort of uh maximize the probability of the the word that was actually in the sentence. Um and then uh so now you have your targets. you one hot encode them and then you can do something like maximizing cross entropy because now you have this multi-class classification you can train it the same way we've trained things before. Now, one thing you might actually want to do in training, you know, currently we're sa