LLM Context Window Paradox

Key Takeaways

context Windows have come up a lot today in the news and you've probably heard a little bit about it but if you're not super familiar with llms it's super helpful to understand what does like a context mean and like why is it so important and why are these models like touting the size of their context Windows let's jump into it I'll explain it to you so a context window you can kind of think about it is almost like the short-term memory of a large language model and to put it into kind of like a real world scenario imagine you're reading a book but at any point in time you can only reference and only understand what is said on a given page which means that you kind of forget every all the pages before it and you don't have any access to all the pages that are in front of it this is your context window and context windows are super important because that's what allows these large langers models to create like coherent and contextualized responses so if you've ever been chatting with chat GPT and you see it reference things back that you had said before it are mentioned and it kind of seems super human it's because what it's doing is pulling out of its context window things that were said and again you can almost think of like a context window as RAM in a computer right like it can only hold so much in this temporary memory storage and context windows are important because as you have lengthy discussions or cram a bunch of information into a large language model that context window shifts forward and backwards right so you could look at this timeline here imagine you've had like a really lengthy conversation with a large language model like chat GPT or CLA or some other one uh and that context window will shift and so if you do it long enough you'll see it'll start to forget things that came at the beginning of the conversation because as it adds in new information it has to get rid of something like it is a finite amount of memory and a finite amount of data these models can reference at any point in time and that means that the problems that they can solve and the amount of text that they can coherently generate is limited as well and you might be thinking well like why don't we just we've got tons of compute and a really large amounts of RAM and computers out there I've got a solution let's just make the context window infinite like why do we have to limit it to a certain amount of size and this is where a really interesting problem comes in and it's called the context window Paradox and so the context window Paradox is basically this really interesting byproduct that happens when as you start to grow these context window sizes it almost starts to degrade the performance of the llm meaning that it kind of is no longer able to understand what is important or relevant right so we've kind of hit these not theoretical maximums but we've Hit The Sweet Spot in these context Windows where the text that the llm will generate seems magical seems really coherent seems to able to pull from what was being said before and referen back in ways that is very very um intuitive and very very uh positive what we've seen is that as we just start to wildly expand these context Windows it starts to kind of give out gibberish and all of a sudden it starts to kind of lose its train of thought almost and a lot of the models out there you've probably seen this taunting you know kind of touting what their context window sizes are and this at the time of writing right is kind of some of the context sizes with the number of words from the developers they come from I think this is actually wrong here I think Gemini 1.5 Pro is actually 1.5 million tokens but for like a given frame of reference to GPT 4 uh as of today is around 8,000 tokens or about 6,000 Words which means at any point in time pb4 can only reference you know that local kind of segmented remember against the timeline only about 6,000 words so if you have a 10,000w conversation those first th 4,000 words it's not going to remember it's only going to remember that back 6,000 words and for us this is a really really interesting problem to solve because crowdbotics here we're interested in using large language models to understand and write code and code is a lot of tokens right so you know like on average let's say that an Enterprise codebase is about a million lines of code with about 50 characters per line that's 50 million characters or about 12.5 million tokens so you can see that if you we've got 12.5 million that we need it to understand all of those but we can only handle about 8,000 at a time or in some of the best cases maybe 1.5 million we're still significantly orders of magnitude off and this has been a really really fun problem to work on and something that we have a pretty unique solution to do doing this if you want to learn more about that I'm not going to go into it today please reach out we would love to chat on how we have done that and probably reveal some of those secrets in a future video but that and and of itself is the context window Paradox and there's tons of good research in white papers and things you can go look to learn more but I hope that if you watch this to this point you now kind of understand why context windows are so important and why a lot of these model providers are actually kind of usiing in their marketing material all over the place but until next time check out some of these videos and we'll see you on the next one