Introducing SWE-bench Verified
📰 OpenAI News
OpenAI releases SWE-bench Verified, a human-validated subset of SWE-bench for evaluating AI models' ability to solve real-world software issues
Action Steps
- Download SWE-bench Verified dataset
- Use SWE-bench Verified to evaluate AI models' performance in solving real-world software issues
- Analyze annotation results to identify areas for improvement
- Fine-tune AI models using SWE-bench Verified to improve their autonomous software engineering capabilities
Who Needs to Know This
Software engineers and AI researchers can benefit from SWE-bench Verified to evaluate and improve the performance of large language models in autonomous software engineering tasks
Key Insight
💡 SWE-bench Verified provides a more accurate evaluation of AI models' autonomous software engineering capabilities
Share This
🚀 OpenAI releases SWE-bench Verified to evaluate AI models' ability to solve real-world software issues! 💻
Key Takeaways
OpenAI releases SWE-bench Verified, a human-validated subset of SWE-bench for evaluating AI models' ability to solve real-world software issues
Full Article
# Introducing SWE-bench Verified | OpenAI
[Skip to main content](https://openai.com/index/introducing-swe-bench-verified#main)
[](https://openai.com/)
* [Research](https://openai.com/research/index/)
* Products
* [Business](https://openai.com/business/)
* [Developers](https://openai.com/api/)
* [Company](https://openai.com/about/)
* [Foundation(opens in a new window)](https://openaifoundation.org/)
[Try ChatGPT(opens in a new window)](https://chatgpt.com/?openaicom-did=8588cb24-3329-443a-bfe9-5d6a0f18edca&openaicom_referred=true)
* Research
* Products
* Business
* Developers
* Company
* [Foundation(opens in a new window)](https://openaifoundation.org/)
Introducing SWE-bench Verified | OpenAI
Table of contents
* [Background on SWE-bench](https://openai.com/index/introducing-swe-bench-verified#background-on-swe-bench)
* [Adapting SWE-bench as a Preparedness Evaluation](https://openai.com/index/introducing-swe-bench-verified#adapting-swe-bench-as-a-preparedness-evaluation)
* [SWE-bench Verified](https://openai.com/index/introducing-swe-bench-verified#swe-bench-verified)
* [Our Approach](https://openai.com/index/introducing-swe-bench-verified#our-approach)
* [Annotation Results](https://openai.com/index/introducing-swe-bench-verified#annotation-results)
* [Performance on SWE-bench Verified](https://openai.com/index/introducing-swe-bench-verified#performance-on-swe-bench-verified)
* [Discussion & Limitations](https://openai.com/index/introducing-swe-bench-verified#discussion-and-limitations)
* [Data downloads](https://openai.com/index/introducing-swe-bench-verified#data-downloads)
August 13, 2024
[Milestone](https://openai.com/research/index/milestone/)
# Introducing SWE-bench Verified
We’re releasing a human-validated subset of SWE-bench that more reliably evaluates AI models’ ability to solve real-world software issues.
[Download SWE-bench Verified(opens in a new window)](https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified)
Share
_Updated February 24, 2025_
As part of our [Preparedness Framework](https://openai.com/preparedness/), OpenAI develops a range of metrics to track, evaluate, and forecast models’ abilities to act autonomously. The ability to autonomously complete software engineering tasks is a key component of our Medium risk level in the Model Autonomy risk category. Evaluating these capabilities is challenging due to the complexity of software engineering tasks, the difficulty of accurately assessing generated code, and the challenge of simulating real-world development scenarios. Therefore, our approach to Preparedness must also involve careful examination of evaluations themselves, to reduce the potential for underestimating or overestimating performance in important risk categories.
One of the most popular evaluation suites for software engineering is [SWE-bench(opens in a new window)](https://www.swebench.com/)[1](https://openai.com/index/introducing-swe-bench-verified#citation-bottom-1)—a benchmark for evaluating large language models’ (LLMs’) abilities to solve real-world software issues sourced from GitHub. The benchmark involves giving agents a code repository and issue description, and challenging them to generate a patch that resolves the problem described by the issue. Coding agents have made impressive progress on SWE-bench, with top scoring agents scoring 20% on SWE-bench and 43% on SWE-bench Lite according to the [SWE-bench leaderboard(opens in a new window)](https://www.swebench.com/) as of August 5, 2024.
Our testing identified some SWE-bench tasks which may be hard or impossible to solve, leading to SWE-bench systematically underestimating models’ autonomous software engineering capabilities. We’ve collaborated with the authors of SWE-bench to address those issues in a new release of the benchmark that should provide more accurate evaluations.
## Background on SWE-bench
Each sample in the SWE-bench test set is created from a r
[Skip to main content](https://openai.com/index/introducing-swe-bench-verified#main)
[](https://openai.com/)
* [Research](https://openai.com/research/index/)
* Products
* [Business](https://openai.com/business/)
* [Developers](https://openai.com/api/)
* [Company](https://openai.com/about/)
* [Foundation(opens in a new window)](https://openaifoundation.org/)
[Try ChatGPT(opens in a new window)](https://chatgpt.com/?openaicom-did=8588cb24-3329-443a-bfe9-5d6a0f18edca&openaicom_referred=true)
* Research
* Products
* Business
* Developers
* Company
* [Foundation(opens in a new window)](https://openaifoundation.org/)
Introducing SWE-bench Verified | OpenAI
Table of contents
* [Background on SWE-bench](https://openai.com/index/introducing-swe-bench-verified#background-on-swe-bench)
* [Adapting SWE-bench as a Preparedness Evaluation](https://openai.com/index/introducing-swe-bench-verified#adapting-swe-bench-as-a-preparedness-evaluation)
* [SWE-bench Verified](https://openai.com/index/introducing-swe-bench-verified#swe-bench-verified)
* [Our Approach](https://openai.com/index/introducing-swe-bench-verified#our-approach)
* [Annotation Results](https://openai.com/index/introducing-swe-bench-verified#annotation-results)
* [Performance on SWE-bench Verified](https://openai.com/index/introducing-swe-bench-verified#performance-on-swe-bench-verified)
* [Discussion & Limitations](https://openai.com/index/introducing-swe-bench-verified#discussion-and-limitations)
* [Data downloads](https://openai.com/index/introducing-swe-bench-verified#data-downloads)
August 13, 2024
[Milestone](https://openai.com/research/index/milestone/)
# Introducing SWE-bench Verified
We’re releasing a human-validated subset of SWE-bench that more reliably evaluates AI models’ ability to solve real-world software issues.
[Download SWE-bench Verified(opens in a new window)](https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified)
Share
_Updated February 24, 2025_
As part of our [Preparedness Framework](https://openai.com/preparedness/), OpenAI develops a range of metrics to track, evaluate, and forecast models’ abilities to act autonomously. The ability to autonomously complete software engineering tasks is a key component of our Medium risk level in the Model Autonomy risk category. Evaluating these capabilities is challenging due to the complexity of software engineering tasks, the difficulty of accurately assessing generated code, and the challenge of simulating real-world development scenarios. Therefore, our approach to Preparedness must also involve careful examination of evaluations themselves, to reduce the potential for underestimating or overestimating performance in important risk categories.
One of the most popular evaluation suites for software engineering is [SWE-bench(opens in a new window)](https://www.swebench.com/)[1](https://openai.com/index/introducing-swe-bench-verified#citation-bottom-1)—a benchmark for evaluating large language models’ (LLMs’) abilities to solve real-world software issues sourced from GitHub. The benchmark involves giving agents a code repository and issue description, and challenging them to generate a patch that resolves the problem described by the issue. Coding agents have made impressive progress on SWE-bench, with top scoring agents scoring 20% on SWE-bench and 43% on SWE-bench Lite according to the [SWE-bench leaderboard(opens in a new window)](https://www.swebench.com/) as of August 5, 2024.
Our testing identified some SWE-bench tasks which may be hard or impossible to solve, leading to SWE-bench systematically underestimating models’ autonomous software engineering capabilities. We’ve collaborated with the authors of SWE-bench to address those issues in a new release of the benchmark that should provide more accurate evaluations.
## Background on SWE-bench
Each sample in the SWE-bench test set is created from a r
DeepCamp AI