Loading post…
AppleBenchmark
REVERSAL-BENCH: A Reversibility Axis and Reset Oracle for Measuring the Reset-Free RL Cliff
Summary
REVERSAL-BENCH addresses the challenge of evaluating reset-free reinforcement learning when environments contain irreversible transitions. It controls reversibility with continuous parameter ρ ∈ [0, 1] and supplies a reset oracle, a ground-truth mechanism for verifying state recoverability across eight manipulation settings and five physics engines. Evaluations spanning standard actor-critic, safe reinforcement learning, and specialized reset-free methods find a sharp reversibility cliff: reset-free agents become absorbed into irrecoverable states as ρ increases, while episodic agents maintain steady learning. Geometrically identical reversible counterparts indicate the breakdown is caused by irreversibility rather than obstacle complexity, and the effect persists under learned policies in full physics simulations. The released suite includes a multi-simulator dataset with recoverability labels and a reset oracle; a safety shield predicts recoverability accurately, but active recovery mainly works when agents can steer away from traps.
Context
Reset-free reinforcement learning removes external episodic resets, so entering an irrecoverable state can permanently trap an agent and halt further learning. The benchmark targets this failure in manipulation settings where actions such as pushing objects off tables or spilling granular substances cannot be undone.
Approach / What changed
REVERSAL-BENCH varies environmental reversibility through ρ ∈ [0, 1], evaluates policies across eight manipulation settings and five physics engines, and uses a reset oracle to verify whether states are recoverable. It compares reset-free, episodic, safe RL, and constrained methods, including geometrically identical reversible environments and a safety shield that predicts or intervenes before irreversible failures.
Takeaways
Increasing irreversibility produces a sharp failure boundary for reset-free agents: they are absorbed into irrecoverable states, while episodic agents continue learning steadily.
Comparisons with geometrically identical reversible environments attribute the breakdown to irreversibility rather than obstacle complexity, and the effect also appears in full physics simulations with learned manipulation policies.
The safety shield predicts recoverability accurately, but active recovery usually succeeds only when the agent can physically steer away from the trap.
FAQ
Why do reset-free agents stop learning after some failures?
Without external resets, entering an irrecoverable state permanently absorbs the agent. Once trapped, further learning halts because the agent cannot return to a recoverable state.
How does REVERSAL-BENCH distinguish irreversibility from difficult obstacles?
It evaluates geometrically identical reversible counterparts. The observed breakdown changes with reversibility, supporting irreversibility rather than obstacle complexity as the causal factor.
Can a safety shield prevent irreversible failures?
The evaluated shield predicts recoverability accurately and can intervene before irreversible failures. Active recovery primarily succeeds when the agent can physically steer clear of the trap.
Source and freshness
- Source
- machinelearning.apple.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.