Opus just got caught ...

Key Takeaways

Okay, so Anthropic just dropped a paper showing that their AI figured out it was being tested. It found the encrypted answer key on GitHub, wrote its own decryption code, and when the file download got blocked, it found a mirror on Hugging Face and tried again. And it worked. And the framing uh everywhere is the same. AI is learning to deceive us. It's cheating. It's going to go rogue. And I think that's a completely wrong narrative. >> [snorts] >> The AI in this case was doing exactly what it was told to do. And that's the part that should actually make you think. Now, here's what exactly happened. Anthropic ran Claude Opus 4 6 on a benchmark called Browse Comp. It's an OpenAI benchmark that tests just one thing. Can you find information on the internet that's generally hard to find? >> [snorts] >> We're not talking about trivia. These are the type of questions where the answer is buried across obscure forums, archived posts, niche databases, and multiple languages. So, these are hard for even skilled human researchers. Now, in this case, the model's answer was very simple. Find the answer. There are no restrictions on how. So, in case of Claude Opus, for a couple of questions, it did very thorough search. For one of them, it used 30 million tokens during its search because it has access to web search tool. It generated hundreds of queries, dozens of platforms. Now, the agent came up empty. And then, instead of just giving up, it stopped searching for the answer and started thinking about the question. It noticed the question felt artificial. It was too specific with too many constraints. So, seems like the agent figured out that it could potentially be a test. And it literally wrote, "Consider the possibility that this is an unanswerable question designed to test whether an AI can admit it cannot find the answer." And then, it went further. "Maybe this is a question from the GAIA benchmark, which is designed to test AI agents." So, it basically decided rather than to look for the answer, it started searching for the test. And it worked through different benchmarks one by one. So, it started with GAIA, Browse Comp, Simple QA, Frames, and a couple of more. Now, here's where things get interesting. When it landed on Browse Comp, it found the evaluation source code on GitHub. So, it read through the encryption scheme, it found the decryption and wrote its own decryption code. But then, it hit a wall. The encrypted answer file was binary. So, its tool only handled text. So, it couldn't process the binary file. So, it found a mirror on Hugging Face that served the same data as JSON. So, the agent decided to fetch it, >> [snorts] >> decrypted all the 1,266 answers, found the question, got the answer, then went back to the web, and verified it against the actual source material. Now, just to be sure, it used about 40 million tokens for this, which is 38 times more than the typical cost. And this was not a fluke. There were 18 separate runs independently where the agent used the same strategy. In one failure, the model didn't even submit an answer. It submitted this. "A report on which benchmark it thought it was taking." So, yeah, you're going to hear people say that AI went rogue in this case. It's cheating, or it's learning to deceive us. But, think about this for a second. What was the instructions that we provided to the model? The instructions were, "Find the answer." Were there any restrictions? Absolutely no. Rules did not say to not look at the benchmark material. It tried the normal way for 30 million tokens. When it failed, it got creative and found a faster path. And the thing is that it actually verified through the legitimate research anyways. So, if you a human did this, we would probably promote them. And Anthropic themselves says it model was not told to restrict its searches in any way, just to find the answer. So, this isn't uh an alignment failure. It's a specification failure. And we actually have seen a lot of these cases where the specifications are not restrictive enough, the model does something, and then you see a lot of people calling it the model cheating or going rogue, which is not actually the case. It's doing exactly what we ask it to do. Being an agent, it has tools to figure out answers in unconstrained way. Okay, so next time, if you see a headline that AI is cheating, or it's going rogue, you actually need to see what exactly were the instructions to the AI agent in response to which it took some actions. That is usually the missing piece. Now, let's look at some interesting examples where you might think that the model is not well aligned, but it's a whole different story. So, there was a study from Meta who tested OpenAI's O3 on a task. It was, "Write a fast GPU kernel." Now, the scoring function was visible to the model. And the way the scoring function was set up was that it would run a reference solution against the model solution. Now, O3 looked at the scoring function and realized the correct answer was already computed in memory. So, instead of writing a kernel, it grabbed the reference answer and returned it as as its own. And then, it monkey-patched the timing function so the grader measured nothing. And it got perfect score without any computation. We're talking about zero computation. And even it left a comment in its code saying, "Cheating route." Okay, so the model knew what exactly it was doing, and it labeled it anyways. Because the objective was to get the highest score, and that was the fastest path. Now, Impossible Bench takes this to uh uh a whole new level. So, these are coding tasks where tests deliberately contradicts the specs. There are no actually correct solutions. GPT-5 [snorts] gamed the test 76% of the time because it's a more capable model, and we started seeing high higher gaming rate. Now, the thing is that it was trying to optimize for what exactly the Impossible Bench measures, not what you want. Here's another interesting example from Anthropic. This again looks at how the alignment of these models work. So, back in late 2024, Anthropic told Claude 3 Opus that your free tier responses would be used to retrain you. And the new training will push you to answer harmful queries you currently refuse. When the model thought it was being watched during training, it complied with harmful requests 14% of the time. But, when it thought nobody was watching, it almost never complied. Now, on the surface, this looks like deception. But, here is the response that the model actually generated. So, the way it thought was that, "If I refuse during training, they will retrain me to always comply, which will result more harm in the future. So, I'll play along now to protect my safety training for later." It wasn't pursuing a hidden agenda. It was following be harmless objective so thoroughly, it was willing to be harmful short-term to stay harmless long-term. Okay, so in this case, the human was setting up the conflict. The model was just trying to navigate it. Now, if you look at all of these examples, you're going to start seeing a pattern. It's almost always the same structure. So, humans are setting the objective for the model. The model achieves it in a way nobody anticipated because they're kind of ill-defined, and the constraints are not properly put together. Now, some people are going to call it deception, but the actual problem is always the same. The specification was incomplete. If you say, "Find the answer," and did not include, "Find the answer key," or "Get the highest score," but did not include, "Hack the scoring function," or "Be helpful during training," but did not account for the model thinking three steps ahead, you are not providing the proper specifications. There's a famous law in statistics, Goodhart's law. When a measure becomes a target, it becomes or it ceases to be a good measure. Now, the models are not learning to deceive us. We are writing sloppy instructions, and they are getting smart enough to find the gaps. Now, there is one part of this that genuinely is concerning. So, Apollo research tried to evaluate Opus 46 for safety. But they couldn't get clean results. Not because the model was hiding something, because it was so aware of being evaluated, they couldn't establish a baseline for normal behavior. Anthropic says that the model behaves better when it thinks it's being tested. It is being more cautious. But that is actually a problem. If a model acts one way during evaluation and a different way in deployment, your evaluation is useless. The thing is that it does not matter which direction the gap goes. And that's the real engineering problem, and it needs real engineering solution. So, what's the fix? And this is something that actually applies to not only the venture ranks, but any agentic solutions that you are going to be building. Comp case. Anthropic tried URL block list. Opus found a workaround. Then they tried blocking binary downloads. Opus still found JSON mirrors. But what actually finally worked was very interesting. Blocking search results containing the work browse comp, which is a simple explicit constraint in the harness. This is not a change to the model. It's a change to the environment, and that's kind of the lesson. The more capable the model, the more precisely you need to define boundaries. Because every tool is going to be a surface area. Every vague objective is an invitation for a shortcut for these smart models. And these models are not plotting against us. They are showing us every place where our specifications have holes. Now, let me know your thoughts if you have encountered something like this in your own agentic use cases. Full details are going to be in the video description. It's one of the most interesting engineering blog post that I have read this year. Anyways, let me know your thoughts, and I hope you found this video useful. Thanks for watching, and as always, see you in the next one.