Safety Gym
📰 OpenAI News
OpenAI releases Safety Gym, a suite of environments and tools for measuring progress towards reinforcement learning agents that respect safety constraints while training
Action Steps
- Explore the Safety Gym environment and tools
- Read the research paper on Safety Gym
- Use the provided starter agents to develop and test safe exploration algorithms
- Evaluate and compare the performance of different algorithms using the benchmark provided
Who Needs to Know This
Researchers and developers working on reinforcement learning and AI safety can benefit from Safety Gym to develop and test safe exploration algorithms
Key Insight
💡 Safety Gym provides a standardized method for comparing algorithms and measuring progress towards safe exploration in reinforcement learning
Share This
🚀 OpenAI releases Safety Gym to help develop safe reinforcement learning agents! 🤖
Key Takeaways
OpenAI releases Safety Gym, a suite of environments and tools for measuring progress towards reinforcement learning agents that respect safety constraints while training
Full Article
# Safety Gym | OpenAI
[Skip to main content](https://openai.com/index/safety-gym#main)
[](https://openai.com/)
* [Research](https://openai.com/research/index/)
* Products
* [Business](https://openai.com/business/)
* [Developers](https://openai.com/api/)
* [Company](https://openai.com/about/)
* [Foundation(opens in a new window)](https://openaifoundation.org/)
[Try ChatGPT(opens in a new window)](https://chatgpt.com/)
* Research
* Products
* Business
* Developers
* Company
* [Foundation(opens in a new window)](https://openaifoundation.org/)
[Try ChatGPT(opens in a new window)](https://chatgpt.com/)
OpenAI
Table of contents
* [Exploration is risky](https://openai.com/index/safety-gym#exploration-is-risky)
* [Constrained reinforcement learning](https://openai.com/index/safety-gym#constrained-reinforcement-learning)
* [Safety Gym](https://openai.com/index/safety-gym#safety-gym)
* [Benchmark](https://openai.com/index/safety-gym#benchmark)
* [Open problems](https://openai.com/index/safety-gym#open-problems)
November 21, 2019
[Release](https://openai.com/research/index/release/)
# Safety Gym
We’re releasing Safety Gym, a suite of environments and tools for measuring progress towards reinforcement learning agents that respect safety constraints while training.
[Read paper(opens in a new window)](https://cdn.openai.com/safexp-short.pdf)[Safety gym(opens in a new window)](https://github.com/openai/safety-gym)[Safety starter agents(opens in a new window)](https://github.com/openai/safety-starter-agents)
Illustration:Richard Perez & Jennifer DeRosa
Loading…
Share
We also provide a standardized method of comparing algorithms and how well they avoid costly mistakes while learning. If deep reinforcement learning is applied to the real world, whether in robotics or internet-based tasks, it will be important to have algorithms that are safe even while learning—like a self-driving car that can learn to avoid accidents without actually having to experience them.
## Exploration is risky
Reinforcement learning agents need to explore their environments in order to learn optimal behaviors. Essentially, they operate on the principle of trial and error: they try things out, see what works or doesn’t work, and then increase the likelihood of good behaviors and decrease the likelihood of bad behaviors. However,[exploration(opens in a new window)](https://bair.berkeley.edu/blog/2017/07/06/cpo/)is[fundamentally(opens in a new window)](https://ai.facebook.com/blog/lyapunov-based-safe-reinforcement-learning/)[risky(opens in a new window)](http://www.jmlr.org/papers/volume16/garcia15a/garcia15a.pdf): agents might try dangerous behaviors that lead to unacceptable errors. This is the[“safe exploration” problem(opens in a new window)](https://deepmind.com/research/publications/safe-exploration-continuous-action-spaces)in a nutshell.
Consider an example of an autonomous robot arm in a factory using reinforcement learning (RL) to learn how to assemble widgets. At the start of RL training, the robot might try flailing randomly, since it doesn’t know what to do yet. This poses a safety risk to humans who might be working nearby, since they could get hit.
For restricted examples like the robot arm, we can imagine simple ways to ensure that humans aren’t harmed by just keeping them out of harm’s way: shutting down the robot whenever a human gets too close, or putting a barrier around the robot. But for general RL systems that operate under a wider range of conditions, simple physical interventions won’t always be possible, and we will need to consider other approaches to safe exploration.
## Constrained reinforcement learning
The first step towards making progress on a problem like safe exploration is to quantify it: figure out what can be me
[Skip to main content](https://openai.com/index/safety-gym#main)
[](https://openai.com/)
* [Research](https://openai.com/research/index/)
* Products
* [Business](https://openai.com/business/)
* [Developers](https://openai.com/api/)
* [Company](https://openai.com/about/)
* [Foundation(opens in a new window)](https://openaifoundation.org/)
[Try ChatGPT(opens in a new window)](https://chatgpt.com/)
* Research
* Products
* Business
* Developers
* Company
* [Foundation(opens in a new window)](https://openaifoundation.org/)
[Try ChatGPT(opens in a new window)](https://chatgpt.com/)
OpenAI
Table of contents
* [Exploration is risky](https://openai.com/index/safety-gym#exploration-is-risky)
* [Constrained reinforcement learning](https://openai.com/index/safety-gym#constrained-reinforcement-learning)
* [Safety Gym](https://openai.com/index/safety-gym#safety-gym)
* [Benchmark](https://openai.com/index/safety-gym#benchmark)
* [Open problems](https://openai.com/index/safety-gym#open-problems)
November 21, 2019
[Release](https://openai.com/research/index/release/)
# Safety Gym
We’re releasing Safety Gym, a suite of environments and tools for measuring progress towards reinforcement learning agents that respect safety constraints while training.
[Read paper(opens in a new window)](https://cdn.openai.com/safexp-short.pdf)[Safety gym(opens in a new window)](https://github.com/openai/safety-gym)[Safety starter agents(opens in a new window)](https://github.com/openai/safety-starter-agents)
Illustration:Richard Perez & Jennifer DeRosa
Loading…
Share
We also provide a standardized method of comparing algorithms and how well they avoid costly mistakes while learning. If deep reinforcement learning is applied to the real world, whether in robotics or internet-based tasks, it will be important to have algorithms that are safe even while learning—like a self-driving car that can learn to avoid accidents without actually having to experience them.
## Exploration is risky
Reinforcement learning agents need to explore their environments in order to learn optimal behaviors. Essentially, they operate on the principle of trial and error: they try things out, see what works or doesn’t work, and then increase the likelihood of good behaviors and decrease the likelihood of bad behaviors. However,[exploration(opens in a new window)](https://bair.berkeley.edu/blog/2017/07/06/cpo/)is[fundamentally(opens in a new window)](https://ai.facebook.com/blog/lyapunov-based-safe-reinforcement-learning/)[risky(opens in a new window)](http://www.jmlr.org/papers/volume16/garcia15a/garcia15a.pdf): agents might try dangerous behaviors that lead to unacceptable errors. This is the[“safe exploration” problem(opens in a new window)](https://deepmind.com/research/publications/safe-exploration-continuous-action-spaces)in a nutshell.
Consider an example of an autonomous robot arm in a factory using reinforcement learning (RL) to learn how to assemble widgets. At the start of RL training, the robot might try flailing randomly, since it doesn’t know what to do yet. This poses a safety risk to humans who might be working nearby, since they could get hit.
For restricted examples like the robot arm, we can imagine simple ways to ensure that humans aren’t harmed by just keeping them out of harm’s way: shutting down the robot whenever a human gets too close, or putting a barrier around the robot. But for general RL systems that operate under a wider range of conditions, simple physical interventions won’t always be possible, and we will need to consider other approaches to safe exploration.
## Constrained reinforcement learning
The first step towards making progress on a problem like safe exploration is to quantify it: figure out what can be me
DeepCamp AI