Why we no longer evaluate SWE-bench Verified

1. 1Recognize the limitations of SWE-bench Verified
2. 2Understand the issues of flawed tests and training leakage
3. 3Consider using SWE-bench Pro as an alternative
4. 4Evaluate the impact on coding progress evaluation

Software engineers and developers on a team benefit from understanding the limitations of SWE-bench Verified and the recommendation to use SWE-bench Pro instead, as it impacts their coding progress evaluation

💡 SWE-bench Verified is no longer a reliable measure of coding progress due to contamination and flawed tests

🚫 OpenAI drops SWE-bench Verified due to contamination & flawed tests! 💻