Anthropic Just Solved Long Context

Key Takeaways

Okay, Anthropic just made the 1 million contacts window generally available for Opus 46 and Sonnet 46. Now, if you're thinking Google had 1 million context windows since the beginning, so what's the big deal here? You will be right. In fact, even OpenAI now offers 1 million context window for their GPT54. But there are two things which makes this release very interesting. The first is the pricing structure Anthropic is introducing. Second is the retrieval accuracy. We haven't seen these numbers before. Okay. So on pricing now you're going to be paying flat pricing irrespective of how many tokens you are using. So 900,000 tokens is going to be built exactly the same as 9,000 tokens which is a great deviation from the pricing tiers that other Frontier Labs uses for long context. Now I want to show you the price comparison in a minute. But this means you're going to be able to send six times more data. So for example, you can send 600 images of PDF pages which is up from 100 pages with the previous context window. Now this is available on all platforms including Microsoft Azure Foundry and Google Vortex AI. This is a different thing because usually Anthropic releases these features on their own API but not on uh some of these third party platforms. So it's a good change to see. Okay. So first let's look at the price comparison. Traditionally anthropic has been the most expensive models and they still are if you're using less than 200,000 tokens. But if see you start using more than 200,000 tokens, it starts to change because OpenAI is charging almost two times for input tokens and 1.5 times for output tokens. Same is the case with Gemini. But Anthropic is now charging a flat rate for any token irrespective of how much context you're using. So if you're using more than 200,000 tokens, it actually becomes very lucrative given that the retrieval accuracy is the best in class. And this is important because having long context does not means context reliability. It used to be pretty much a gimmick so far because as you increase the context window, we started seeing the loss in the middle phenomena. Gemini has been offering 1 million contacts window from the beginning but normally the usable range was in 150 to 200,000 tokens. With this it has changed a lot. Now if you look at 256,000 tokens these new opens in sonnet models are state-of-the-art at almost 90% retrieval accuracy for eight needles in the haststack. Now traditionally we had a long context models but having long context does not mean reliability. The usable context window is usually around 200 and 250,000 tokens even for a 1 million context window. This is an example coming out of the original cloud 2.1 200,000 context window. And as you can see it was pretty bad. The useful window was probably around 20,000 tokens in this case. Now in this case, Enthropic is measuring the performance on MRCF version 2 8 needle in the haststack and we have come a long way on 256,000 tokens. Um these new cloud models are state-of-the-art. The next best one is GPT54. Google Gemini 31 is about 60%. Now the thing is if you increase the context window to a million tokens you see some really drastic differences. So that 60% goes to almost 26% for Gemini. That 80% goes to 36%. In case of the new cloud models, it actually retains its performance. At 1 million context window, you have about 18% reduction for of 46, which is actually really impressive because this is still usable. Okay, so here's how the needle in the haststack benchmark actually works. You have your full context window. You put different facts at different length in that context window and then you ask the LLM to retrieve those facts. Traditionally, people used one fact in the context window. But now we're seeing a lot more complex scenarios. For example, this is eight needles in the haststack. So eight simultaneous different facts, which is probably more representative of real world use cases. So what exactly does this mean for you as a developer? Now irrespective of whether you is using a cloud code or the cloud API, you're going to start seeing less and less compactions because compactions is one of the biggest problem. It's kind of like shortterm and long-term memory loss because your agents forgets everything. One company reported 15% less compactions with this new 1 million context window. Other companies reported that they were using far fewer tokens. Now with this release, I think even multi-round agentic performance should improve because you're not going to be hitting those context window limits and especially given it has really good retrieval accuracy even at long context window. And I think for longunning agentic tasks, it also makes cloud opus a more favorable option because the premium is not that much at all anymore. You're getting a huge performance boost in terms of retrieval at a marginal price increase. Okay, one last thing. With this long context window, does this mean we don't need rag anymore? And the answer is no. We still need rag. There are two things. First, most of your documents are going to be longer than 1 million context window. Second, the pricing still is pretty aggressive, especially if you're using less number of tokens, that is going to be more effective, which rag or retrieve augmented generation enables. Third is that using long contexts means higher latency, which is not practical in most realtime applications. So you will still need some sort of retrieval methods. They may not be primarily semantic retrieval like embedding based but there are a combination of different things that you can use with along with this new updated context window for better retrieval accuracy. Anyways, I hope you found this video useful. Thanks for watching and as always, see you in the next