---
title: "REVERSAL-BENCH: A Reversibility Axis and Reset Oracle for Measuring the Reset-Free RL Cliff"
description: "REVERSAL-BENCH addresses the challenge of evaluating reset-free reinforcement learning when environments contain irreversible transitions. It controls reversibility with continuous parameter ρ ∈ [0, 1] and supplies a reset oracle, a ground-truth mechanism for verifying state recoverability across eight manipulation settings and five physics engines. Evaluations spanning standard actor-critic, safe reinforcement learning, and specialized reset-free methods find a sharp reversibility cliff: reset-free agents become absorbed into irrecoverable states as ρ increases, while episodic agents maintain steady learning. Geometrically identical reversible counterparts indicate the breakdown is caused by irreversibility rather than obstacle complexity, and the effect persists under learned policies in full physics simulations. The released suite includes a multi-simulator dataset with recoverability labels and a reset oracle; a safety shield predicts recoverability accurately, but active recovery mainly works when agents can steer away from traps."
---

# REVERSAL-BENCH: A Reversibility Axis and Reset Oracle for Measuring the Reset-Free RL Cliff

[Apple](https://yomu.fyi/company/apple) · Sep 17, 2026

**Type:** Benchmark

## Summary

REVERSAL-BENCH addresses the challenge of evaluating reset-free reinforcement learning when environments contain irreversible transitions. It controls reversibility with continuous parameter ρ ∈ \[0, 1\] and supplies a reset oracle, a ground-truth mechanism for verifying state recoverability across eight manipulation settings and five physics engines. Evaluations spanning standard actor-critic, safe reinforcement learning, and specialized reset-free methods find a sharp reversibility cliff: reset-free agents become absorbed into irrecoverable states as ρ increases, while episodic agents maintain steady learning. Geometrically identical reversible counterparts indicate the breakdown is caused by irreversibility rather than obstacle complexity, and the effect persists under learned policies in full physics simulations. The released suite includes a multi-simulator dataset with recoverability labels and a reset oracle; a safety shield predicts recoverability accurately, but active recovery mainly works when agents can steer away from traps.

## Context

Reset-free reinforcement learning removes external episodic resets, so entering an irrecoverable state can permanently trap an agent and halt further learning. The benchmark targets this failure in manipulation settings where actions such as pushing objects off tables or spilling granular substances cannot be undone.

## Approach / What changed

REVERSAL-BENCH varies environmental reversibility through ρ ∈ \[0, 1\], evaluates policies across eight manipulation settings and five physics engines, and uses a reset oracle to verify whether states are recoverable. It compares reset-free, episodic, safe RL, and constrained methods, including geometrically identical reversible environments and a safety shield that predicts or intervenes before irreversible failures.

## Takeaways

- Increasing irreversibility produces a sharp failure boundary for reset-free agents: they are absorbed into irrecoverable states, while episodic agents continue learning steadily.
- Comparisons with geometrically identical reversible environments attribute the breakdown to irreversibility rather than obstacle complexity, and the effect also appears in full physics simulations with learned manipulation policies.
- The safety shield predicts recoverability accurately, but active recovery usually succeeds only when the agent can physically steer away from the trap.

## FAQ

### Why do reset-free agents stop learning after some failures?

Without external resets, entering an irrecoverable state permanently absorbs the agent. Once trapped, further learning halts because the agent cannot return to a recoverable state.

### How does REVERSAL-BENCH distinguish irreversibility from difficult obstacles?

It evaluates geometrically identical reversible counterparts. The observed breakdown changes with reversibility, supporting irreversibility rather than obstacle complexity as the causal factor.

### Can a safety shield prevent irreversible failures?

The evaluated shield predicts recoverability accurately and can intervene before irreversible failures. Active recovery primarily succeeds when the agent can physically steer clear of the trap.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Reinforcement Learning](https://yomu.fyi/topic/reinforcement-learning), [Robotics](https://yomu.fyi/topic/robotics), [Safety](https://yomu.fyi/topic/safety)

- Source: [Apple](https://machinelearning.apple.com/research/reversal-bench-rl-cliff)
- Source URL: https://machinelearning.apple.com/research/reversal-bench-rl-cliff
- Ingested by Yomu: 2026-09-17T15:23:07.058Z

[Read original post](https://machinelearning.apple.com/research/reversal-bench-rl-cliff)