Concrete AI safety problems
📰 OpenAI News
OpenAI researchers discuss concrete AI safety problems, including safe exploration, robustness to distributional shift, and avoiding negative side effects
Action Steps
- Identify potential safety risks in AI systems
- Develop methods for safe exploration, such as reinforcement learning with constraints
- Implement robustness to distributional shift, using techniques like uncertainty estimation and data augmentation
- Design reward functions that avoid negative side effects and reward hacking
- Test and evaluate AI systems for safety and reliability
Who Needs to Know This
AI researchers and engineers can benefit from understanding these safety problems to develop more reliable and trustworthy AI systems, and product managers can use this knowledge to inform their product development strategies
Key Insight
💡 AI safety requires addressing specific, practical problems to ensure that AI systems operate as intended and avoid accidents
Share This
🚨 AI safety matters! 🚨 Researchers discuss concrete problems, including safe exploration and robustness to distributional shift
Key Takeaways
OpenAI researchers discuss concrete AI safety problems, including safe exploration, robustness to distributional shift, and avoiding negative side effects
Full Article
# Concrete AI safety problems | OpenAI
[Skip to main content](https://openai.com/index/concrete-ai-safety-problems#main)
[](https://openai.com/)
* [Research](https://openai.com/research/index/)
* Products
* [Business](https://openai.com/business/)
* [Developers](https://openai.com/api/)
* [Company](https://openai.com/about/)
* [Foundation(opens in a new window)](https://openaifoundation.org/)
[Try ChatGPT(opens in a new window)](https://chatgpt.com/)
* Research
* Products
* Business
* Developers
* Company
* [Foundation(opens in a new window)](https://openaifoundation.org/)
[Try ChatGPT(opens in a new window)](https://chatgpt.com/)
OpenAI
June 21, 2016
[Publication](https://openai.com/research/index/publication/)
# Concrete AI safety problems
[Read Paper(opens in a new window)](https://arxiv.org/abs/1606.06565)
Loading…
Share
We (along with researchers from Berkeley and Stanford) are co-authors on today’s paper led by Google Brain researchers,[Concrete Problems in AI Safety(opens in a new window)](https://arxiv.org/abs/1606.06565). The paper explores many research problems around ensuring that modern machine learning systems operate as intended.
We (along with researchers from Berkeley and Stanford) are co-authors on today’s paper led by Google Brain researchers,[Concrete Problems in AI Safety(opens in a new window)](https://arxiv.org/abs/1606.06565). The paper explores many research problems around ensuring that modern machine learning systems operate as intended. (The problems are very practical, and we’ve already seen some being integrated into[OpenAI Gym(opens in a new window)](https://gym.openai.com/envs#safety).)
Advancing AI requires making AI systems smarter, but it also requires preventing accidents—that is, ensuring that AI systems do what people actually want them to do. There’s been an increasing focus on[safety research(opens in a new window)](http://futureoflife.org/background/benefits-risks-of-artificial-intelligence/)from the machine learning community, such as a recent[paper(opens in a new window)](https://intelligence.org/files/Interruptibility.pdf)from[DeepMind(opens in a new window)](https://deepmind.com/)and[FHI(opens in a new window)](https://www.fhi.ox.ac.uk/). Still, many machine learning researchers have wondered just how much safety research can be done today.
The authors discuss five areas:
* **Safe exploration**._Can_[_reinforcement learning_(opens in a new window)](http://karpathy.github.io/2016/05/31/rl/)_(RL) agents learn about their environment without executing catastrophic actions?_ For example, can an RL agent learn to navigate an environment without ever falling off a ledge?
* **Robustness to distributional shift**._Can machine learning systems be robust to changes in the data distribution, or at least fail gracefully?_ For example, can we build[image classifiers(opens in a new window)](https://www.tensorflow.org/versions/r0.9/tutorials/deep_cnn/index.html)that indicate appropriate uncertainty when shown new kinds of images, instead of confidently trying to use its[potentially inapplicable(opens in a new window)](http://arxiv.org/abs/1412.6572)learned model?
* **Avoiding negative side effects**._Can we transform an RL agent’s_[_reward function_(opens in a new window)](https://webdocs.cs.ualberta.ca/~sutton/book/ebook/node9.html)_to avoid undesired effects on the environment?_ For example, can we build a robot that will move an object while avoiding knocking anything over or breaking anything, without manually programming a separate penalty for each possible bad behavior?
* **Avoiding “reward hacking” and “**[**wireheading**(opens in a new window)](http://www.agroparistech.fr/mmip/maths/laurent_orseau/papers/ring-orseau-AGI-2011-delusion.pdf)**”**._Can
[Skip to main content](https://openai.com/index/concrete-ai-safety-problems#main)
[](https://openai.com/)
* [Research](https://openai.com/research/index/)
* Products
* [Business](https://openai.com/business/)
* [Developers](https://openai.com/api/)
* [Company](https://openai.com/about/)
* [Foundation(opens in a new window)](https://openaifoundation.org/)
[Try ChatGPT(opens in a new window)](https://chatgpt.com/)
* Research
* Products
* Business
* Developers
* Company
* [Foundation(opens in a new window)](https://openaifoundation.org/)
[Try ChatGPT(opens in a new window)](https://chatgpt.com/)
OpenAI
June 21, 2016
[Publication](https://openai.com/research/index/publication/)
# Concrete AI safety problems
[Read Paper(opens in a new window)](https://arxiv.org/abs/1606.06565)
Loading…
Share
We (along with researchers from Berkeley and Stanford) are co-authors on today’s paper led by Google Brain researchers,[Concrete Problems in AI Safety(opens in a new window)](https://arxiv.org/abs/1606.06565). The paper explores many research problems around ensuring that modern machine learning systems operate as intended.
We (along with researchers from Berkeley and Stanford) are co-authors on today’s paper led by Google Brain researchers,[Concrete Problems in AI Safety(opens in a new window)](https://arxiv.org/abs/1606.06565). The paper explores many research problems around ensuring that modern machine learning systems operate as intended. (The problems are very practical, and we’ve already seen some being integrated into[OpenAI Gym(opens in a new window)](https://gym.openai.com/envs#safety).)
Advancing AI requires making AI systems smarter, but it also requires preventing accidents—that is, ensuring that AI systems do what people actually want them to do. There’s been an increasing focus on[safety research(opens in a new window)](http://futureoflife.org/background/benefits-risks-of-artificial-intelligence/)from the machine learning community, such as a recent[paper(opens in a new window)](https://intelligence.org/files/Interruptibility.pdf)from[DeepMind(opens in a new window)](https://deepmind.com/)and[FHI(opens in a new window)](https://www.fhi.ox.ac.uk/). Still, many machine learning researchers have wondered just how much safety research can be done today.
The authors discuss five areas:
* **Safe exploration**._Can_[_reinforcement learning_(opens in a new window)](http://karpathy.github.io/2016/05/31/rl/)_(RL) agents learn about their environment without executing catastrophic actions?_ For example, can an RL agent learn to navigate an environment without ever falling off a ledge?
* **Robustness to distributional shift**._Can machine learning systems be robust to changes in the data distribution, or at least fail gracefully?_ For example, can we build[image classifiers(opens in a new window)](https://www.tensorflow.org/versions/r0.9/tutorials/deep_cnn/index.html)that indicate appropriate uncertainty when shown new kinds of images, instead of confidently trying to use its[potentially inapplicable(opens in a new window)](http://arxiv.org/abs/1412.6572)learned model?
* **Avoiding negative side effects**._Can we transform an RL agent’s_[_reward function_(opens in a new window)](https://webdocs.cs.ualberta.ca/~sutton/book/ebook/node9.html)_to avoid undesired effects on the environment?_ For example, can we build a robot that will move an object while avoiding knocking anything over or breaking anything, without manually programming a separate penalty for each possible bad behavior?
* **Avoiding “reward hacking” and “**[**wireheading**(opens in a new window)](http://www.agroparistech.fr/mmip/maths/laurent_orseau/papers/ring-orseau-AGI-2011-delusion.pdf)**”**._Can
DeepCamp AI