The truth about Claude Code's evals

Key Takeaways

Unfortunately, X or Twitter is like a medium where you just get all these strong opinions of like don't do evals. It's bad. We tried it. It doesn't work. We're claud code and we don't do evals. And there's just so much nuance behind all of it cuz a lot of these applications are standing on the shoulders of evals. The fine-tuned cloud models have been evaluated on many coding benchmarks. They are actually probably very systematic about the error analysis to some extent. I bet you that they are monitoring who is using Claude, how many people are using Claude, how many chats are being created, how long these chats are. They're also probably monitoring in their internal team. They're dog fooding. Anytime something is off, they maybe have a queue or they send it to the person developing cloud code and this person is implicitly doing some form of error analysis. All of this is ebels.