Why GPT Hits a Memory Wall

Key Takeaways

Imagine trying to read the entire library while holding every page you've ever read in your hands [music] at the same time. That's what attention asks a transformer to do. And for a long time, that limitation defined how much language a model could understand. So how did modern LLM suddenly jump from a few thousand tokens to [music] tens even hundreds of thousands? The answer is not one specific trick. It's a shift in how we think about the attention itself. At its core, self attention compares every token to every other token. That comparison grows quadratically. Double the sequence length and attention cost grows four times. Not really efficient. KV cache helped us generate tokens efficiently, but it didn't solve the fundamental problem. The model still needs to store and attend to every previous key and value. Eventually, memory runs out. This is where the traditional transformer hits the wall. For years, that wall seemed unavoidable. attention was powerful but inherently expensive. Long documents, books, code bases, entire conversations, they simply did not fit. Then you should ask a different question. What if attention didn't need to see everything at once? What if memory could be external or compressed or reused over the time? That question led to infinite attention. Infinite attention is not a single algorithm. It's a design philosophy. The idea is simple to state but smart in the implication. A model should be able to attend over an efficiently unbounded sequence without storing all tokens explicitly. So you may ask what does that mean? Instead of treating attention as a giant matrix, infinite attention treats it as a stream. Tokens arrive over the time. Some information is kept in high resolution. Some information is compressed. Some information is summarized and returned to memory just like humans do. To understand how this works, we need to break attention into two roles. It decides where to look and it decides what to remember. Traditional transformers mix both into one operation. Infinite attention separates them. The core idea is this. Recent tokens are handled with a standard attention. All tokens however are written into memory. There's basically a compressed representation that still influences the present like what you write in your footnote or every notes that you want to summarize and a specific page that you don't want to forget. The model never forgets everything. It forgets details but keeps meaning like what you do as tokens stream in the model periodically summarizes parts of the past into memory vectors. Those vectors represent topics, entities, events, long range dependencies and so on so forth. These memory vectors are not static. They are trainable. They're updated continuously and most importantly they can be attended to just like normal tokens. This changes the complexity. Instead of attention scaling with entire sequence length, it scales with fixed window of the recent tokens plus a fixed size memory that makes the cost linear over the time. Infinite attention also works naturally with KV cache. Recent tokens still use cached keys and values. Older tokens are folded into the memory. The model never recomputes attention over the full pass. It reuses the structure. The key inside here is the memory replaces raw history. There's another important detail. Memory is not just a summary. It is attention error. The model learns what to store based on what future tokens tend to need. If something is rarely referenced again, it fades. If something matters later, it is preserved. This is learned end to end. This is why infinite attention works so well. The model is not handgineered to remember specific things. It learns memory management. The same way it learns language through optimization. Our op plays a critical role here. If you want to understand the rope algorithm specifically, you can watch this video. Because rope encodes relative position, memory vectors can interact meaningfully with new tokens even when absolute position is lost. Distance is still matters. Order still matters, but position ids are no longer required. Infinite attention also explains why modern language models can read books, analyze long code bases, and trace conversations across ours. They are not storing everything. They're storing what matters. This approach connects transformers to a much older idea in machine learning recurrent models but with a crucial difference. RNN compress everything into a single hidden state. Infinite attention uses a structure addressible memory. It keeps multiple memory slots. Each one can be attended independently that makes memory expressive not bottleneck. This is also why infinite attention generalizes because memory size is fixed. The model can in principle process infinite sequences. A context window becomes a soft constraint not a hard limit. You may ask here that's a perfect technique but it should have a cost right? Of course this comes with a trade-offs. Memory is approximate. Fine grain details can be lost. It's like you are memorizing something as abstract view not detail view. Compression decisions matter but for reasoning narrative coherence and long-term dependency tracking. This trade-off is worth it. Modern lung context systems combine multiple techniques. A sliding window attention for local precision. KV cache for efficiency, rope or rope for positional generalization and infinite attention for a scalable memory. Together they break the original transformer limits. So when you see a model advertised with a 100,000 or 1 million token context window, it's not brute force. It's architecture. It is a rethinking of what attention is meant to do. Attention is no longer just comparison. It's memory management. That shift is what enables the next generation of models. And with that I would like to say to you good morning, good afternoon, good evening. Bye. [music] >> [music] [music]