The Odd Geometry Behind GPT’s Ability to Remember

Large language models were never designed to handle very long sequences. Classic Transformers rely on absolute positional embeddings, which break once you go beyond the context lengths seen during training. And yet modern models like LLaMA can reason over tens, even hundreds of thousands of tokens. So what changed? In this video, we dive deep into Rotary Positional Embeddings (RoPE), the geometric trick that allows modern LLMs to generalize to long context windows without exploding parameters or breaking attention. You’ll learn: - Why absolute positional embeddings fail for long sequen…