Introducing HealthBench
📰 OpenAI News
OpenAI introduces HealthBench, a benchmark for evaluating AI systems in health settings, with 5,000 realistic health conversations and custom physician-created rubrics
Action Steps
- Explore the HealthBench dataset and evaluation framework
- Use HealthBench to evaluate the performance of AI models in health settings
- Develop and improve AI models based on the results and feedback from HealthBench
Who Needs to Know This
Data scientists, AI engineers, and healthcare professionals can benefit from HealthBench to develop and evaluate more effective AI models for health applications
Key Insight
💡 HealthBench provides a meaningful, trustworthy, and unsaturated evaluation framework for AI systems in health, supporting progress and improvement in the field
Share This
🚀 Introducing HealthBench: a new benchmark for AI in health! 🏥💻
Key Takeaways
OpenAI introduces HealthBench, a benchmark for evaluating AI systems in health settings, with 5,000 realistic health conversations and custom physician-created rubrics
Full Article
# Introducing HealthBench | OpenAI
[Skip to main content](https://openai.com/index/healthbench#main)
[](https://openai.com/)
* [Research](https://openai.com/research/index/)
* Products
* [Business](https://openai.com/business/)
* [Developers](https://openai.com/api/)
* [Company](https://openai.com/about/)
* [Foundation(opens in a new window)](https://openaifoundation.org/)
Log in[Try ChatGPT(opens in a new window)](https://chatgpt.com/?openaicom-did=ba7fa89a-5f61-4f9e-b5eb-cfb916b17e46&openaicom_referred=true)
* Research
* Products
* Business
* Developers
* Company
* [Foundation(opens in a new window)](https://openaifoundation.org/)
Introducing HealthBench | OpenAI
Table of contents
* [Dataset description](https://openai.com/index/healthbench#dataset-description)
* [Performance of models](https://openai.com/index/healthbench#performance-of-models)
* [Comparison against physician baselines](https://openai.com/index/healthbench#comparison-against-physician-baselines)
* [Trustworthiness of HealthBench](https://openai.com/index/healthbench#trustworthiness-of-healthbench)
* [Where we go from here](https://openai.com/index/healthbench#where-we-go-from-here)
May 12, 2025
[Publication](https://openai.com/research/index/publication/)
# Introducing HealthBench
An evaluation for AI systems and human health.
[Read paper(opens in a new window)](https://cdn.openai.com/pdf/bd7a39d5-9e9f-47b3-903c-8b847ca650c7/healthbench_paper.pdf)[View code(opens in a new window)](https://github.com/openai/simple-evals)
Listen to article
Share
Improving human health will be one of the defining impacts of AGI. If developed and deployed effectively, large language models have the potential to expand access to health information, support clinicians in delivering high-quality care, and help people advocate for their health and that of their communities.
To get there, we need to ensure models are useful and safe. Evaluations are essential to understanding how models perform in health settings. Significant efforts have already been made across academia and industry, yet many existing evaluations do not reflect realistic scenarios, lack rigorous validation against expert medical opinion, or leave no room for state-of-the-art models to improve.
Today, we’re introducing HealthBench: a new benchmark designed to better measure capabilities of AI systems for health. Built in partnership with **262** physicians who have practiced in **60** countries, HealthBench includes **5,000** realistic health conversations, each with a custom physician-created rubric to grade model responses.
HealthBench is grounded in our belief that evaluations for AI systems in health should be:
* **Meaningful: Scores reflect real-world impact.** This should go beyond exam questions to capture complex, real-life scenarios and workflows that mirror the ways individuals and clinicians interact with models.
* **Trustworthy: Scores are faithful indicators of physician judgment.** Evaluations should reflect the standards and priorities of healthcare professionals, providing a rigorous foundation for improving AI systems.
* **Unsaturated: Benchmarks support progress.**Current models should show substantial room for improvement, offering model developers incentives to continuously improve performance.
Alongside the HealthBench benchmark, we're also sharing how several of our models perform, setting a new baseline to improve upon.
## Dataset description
_HealthBench tests how well AI models perform in realistic health scenarios, based on what physician experts say matters most._
The 5,000 conversations in HealthBench simulate interactions between AI m
[Skip to main content](https://openai.com/index/healthbench#main)
[](https://openai.com/)
* [Research](https://openai.com/research/index/)
* Products
* [Business](https://openai.com/business/)
* [Developers](https://openai.com/api/)
* [Company](https://openai.com/about/)
* [Foundation(opens in a new window)](https://openaifoundation.org/)
Log in[Try ChatGPT(opens in a new window)](https://chatgpt.com/?openaicom-did=ba7fa89a-5f61-4f9e-b5eb-cfb916b17e46&openaicom_referred=true)
* Research
* Products
* Business
* Developers
* Company
* [Foundation(opens in a new window)](https://openaifoundation.org/)
Introducing HealthBench | OpenAI
Table of contents
* [Dataset description](https://openai.com/index/healthbench#dataset-description)
* [Performance of models](https://openai.com/index/healthbench#performance-of-models)
* [Comparison against physician baselines](https://openai.com/index/healthbench#comparison-against-physician-baselines)
* [Trustworthiness of HealthBench](https://openai.com/index/healthbench#trustworthiness-of-healthbench)
* [Where we go from here](https://openai.com/index/healthbench#where-we-go-from-here)
May 12, 2025
[Publication](https://openai.com/research/index/publication/)
# Introducing HealthBench
An evaluation for AI systems and human health.
[Read paper(opens in a new window)](https://cdn.openai.com/pdf/bd7a39d5-9e9f-47b3-903c-8b847ca650c7/healthbench_paper.pdf)[View code(opens in a new window)](https://github.com/openai/simple-evals)
Listen to article
Share
Improving human health will be one of the defining impacts of AGI. If developed and deployed effectively, large language models have the potential to expand access to health information, support clinicians in delivering high-quality care, and help people advocate for their health and that of their communities.
To get there, we need to ensure models are useful and safe. Evaluations are essential to understanding how models perform in health settings. Significant efforts have already been made across academia and industry, yet many existing evaluations do not reflect realistic scenarios, lack rigorous validation against expert medical opinion, or leave no room for state-of-the-art models to improve.
Today, we’re introducing HealthBench: a new benchmark designed to better measure capabilities of AI systems for health. Built in partnership with **262** physicians who have practiced in **60** countries, HealthBench includes **5,000** realistic health conversations, each with a custom physician-created rubric to grade model responses.
HealthBench is grounded in our belief that evaluations for AI systems in health should be:
* **Meaningful: Scores reflect real-world impact.** This should go beyond exam questions to capture complex, real-life scenarios and workflows that mirror the ways individuals and clinicians interact with models.
* **Trustworthy: Scores are faithful indicators of physician judgment.** Evaluations should reflect the standards and priorities of healthcare professionals, providing a rigorous foundation for improving AI systems.
* **Unsaturated: Benchmarks support progress.**Current models should show substantial room for improvement, offering model developers incentives to continuously improve performance.
Alongside the HealthBench benchmark, we're also sharing how several of our models perform, setting a new baseline to improve upon.
## Dataset description
_HealthBench tests how well AI models perform in realistic health scenarios, based on what physician experts say matters most._
The 5,000 conversations in HealthBench simulate interactions between AI m
DeepCamp AI