[State of Code Evals] After SWE-bench, Code Clash & SOTA Coding Benchmarks recap — John Yang

From creating *SWE-bench* in a Princeton basement to shipping *CodeClash,* *SWE-bench Multimodal,* and *SWE-bench Multilingual,* *John Yang* has spent the last year and a half watching his benchmark become the de facto standard for evaluating AI coding agents—trusted by Cognition (Devin), OpenAI, Anthropic, and every major lab racing to solve software engineering at scale. We caught up with John live at *NeurIPS 2025* to dig into the state of code evals heading into 2026: why *SWE-bench went from ignored (October 2023) to the industry standard* after Devin's launch (and how Walden emailed him two weeks before the big reveal), how the benchmark evolved from Django-heavy to *nine languages across 40 repos* (JavaScript, Rust, Java, C, Ruby), why *unit tests as verification are limiting* and long-running agent tournaments might be the future (CodeClash: agents maintain codebases, compete in arenas, and iterate over multiple rounds), the *proliferation of SWE-bench variants* (SWE-bench Pro, SWE-bench Live, SWE-Efficiency, AlgoTune, SciCode) and how benchmark authors are now justifying their splits with curation techniques instead of just "more repos," why *Tau-bench's "impossible tasks" controversy* is actually a feature not a bug (intentionally including impossible tasks flags cheating), the tension between *long autonomy (5-hour runs) vs. interactivity* (Cognition's emphasis on fast back-and-forth), how *Terminal-bench unlocked creativity* by letting PhD students and non-coders design environments beyond GitHub issues and PRs, the *academic data problem* (companies like Cognition and Cursor have rich user interaction data, academics need user simulators or compelling products like LMArena to get similar signal), and his vision for *CodeClash as a testbed for human-AI collaboration*—freeze model capability, vary the collaboration setup (solo agent, multi-agent, human+agent), and measure how interaction patterns change as models climb the ladder from code completion to f