ResSafe: Learning Safety Filtering with Residual Reinforcement Learning for Humanoids

Figure 1: ResSafe validated on Unitree G1 balance tasks, both with and without payloads.

Background: The Cost of Coupling Performance and Safety

Reinforcement learning for humanoid control is inherently multi-objective: nominal performance (agility, motion-tracking accuracy) must be balanced against safety and robustness, and these objectives often conflict, forming a Pareto trade-off. The bad news in practice: RL algorithms rely on gradient-based optimization and typically converge to locally optimal solutions, yielding a suboptimal Pareto frontier. Consequently, improving robustness or safety today — via heavier domain randomization or stronger failure penalties — almost inevitably degrades performance. Classical safe RL based on constrained MDPs suffers the same issue, with safety and performance coupled through the Lagrangian.

The starting point is an observation about human motor intelligence: when learning a new sport, humans do not relearn safety — maintaining balance, avoiding collisions — from scratch, but rely on reflexive motor skills built over years of growth and physical interaction. The goal therefore becomes a generalizable safety-filtering policy that transfers across tasks without retraining safety from scratch. As a first step, the paper explicitly decouples the learning of performance and safety during training.

Figure 2: (a) The suboptimal Pareto frontier when performance and safety are coupled in a single policy; (b) ResSafe's residual decoupling; (c) Training guidance strategy for humanoids.

Method

From Safety Filtering to Action-Constrained Optimal Control

Consider a safety-constrained optimal control problem with system trajectory $\dot{x}(t) = f(x(t), u(t))$ and control input bounded by a compact set $u(t) \in \mathcal{U}$. Safety is defined by a constraint set requiring $\xi(t) \in \mathcal{C}$ throughout, representable as the zero-superlevel set of a continuous function: $\mathcal{C} = \{x : h(x) \ge 0\}$. The control objective is:

$$\min_{u(\cdot)} \int_0^T \gamma^t \, \ell(x(t), u(t)) \, dt \quad \text{s.t.} \quad h(\xi(t)) \ge 0,\ \forall t$$

where the first term is the task objective (running cost $\ell$ and discount $\gamma$) and the second the safety constraint. The paper decouples this into a safety problem and an action-constrained optimal control problem. From Hamilton-Jacobi (HJ) reachability, the worst-case safety value function is:

$$V(x) = \inf_{u(\cdot)} \sup_{t \in [0,T]} h(\xi(t))$$

The maximal control-invariant set within $\mathcal{C}$ is its zero-superlevel set; once a trajectory exits it, the constraint is inevitably violated. The maximal safe action set $\mathcal{U}_{\text{safe}}(x)$ collects all inputs that prevent violation. The original problem can then be written in action-constrained form and, in classical safe control, conservatively approximated by CBFs or predictive control as a safety filtering architecture: a reference policy optimizes the task objective first, then a filter maps potentially unsafe reference actions into safe ones.

The filter uses a min-norm objective — modifying the reference action only when safety requires it:

$$u_{\text{filter}}(x) = \arg\min_{u \in \mathcal{U}_{\text{safe}}(x)} \lVert u - u_{\text{ref}}(x) \rVert^2$$

The filtered output is equivalently the unfiltered reference action plus a filter correction:

$$u_{\text{filtered}} = u_{\text{ref}} + \underbrace{(u_{\text{filter}} - u_{\text{ref}})}_{\Delta u}$$

In the CBF literature $\Delta u$ is called the CBF augmentation input. ResSafe's core idea: do not learn the safe action set or the value function — learn this correction $\Delta u$ directly with residual RL.

Discriminating Hyperplane and Its Limits

For control-affine dynamics (which covers rigid-body robot dynamics), the safe action set can be represented at each step by a halfspace constraint:

$$\mathcal{U}_{\text{safe}}(x) = \{u : a(x)^\top u \le b(x)\}$$

where $(a(x), b(x))$ defines a discriminating hyperplane (DH) in control-input space, separating certified safe inputs from potentially unsafe ones. This representation unifies classical CBF and HJ-reachability filters, and its key advantage is that hyperplane parameters can be learned directly from data — no explicit dynamics model and no explicit computation of the safety value function above.

The cart-pole running example illustrates it: stabilize the cart at a target position while keeping the pole upright, with a safety constraint on cart position deliberately conflicting with the task. A PPO-trained MLP reference policy solves the task; a PPO-learned discriminating hyperplane then safety-filters it. It works — but the authors flag two problems: DH learns the constraint representation rather than the filtered safe action, so it generalizes poorly to higher-dimensional systems, and it imposes halfspace structure on the safe action set, which may be impractical for more complex systems whose safe sets are nonconvex or disconnected. This motivates the residual RL route.

Figure 3: Cart-pole example. The residual RL safety filter (blue in (c)) closely approximates the min-norm filtered input from the discriminating hyperplane (green background shows the learned safe action set), with slight conservativeness.

Residual RL Safety Filtering

To handle humanoids — high-dimensional, uncertain, and stochastic — the paper adopts an MDP-based probabilistic notion of safety: maximize the probability of constraint satisfaction (safety rate). Compared with existing approaches that solve the reachability value function with RL, the key difference is that this problem reformulates as a sum-over-reward problem:

$$\max_{\pi} \mathbb{E}\left[\sum_{t=0}^{T-1} \mathbb{1}\{h(x_t) \ge 0\}\right]$$

whereas existing methods involve a min-over-time objective, leading to a different class of Bellman equations for which off-the-shelf algorithms like PPO are not directly applicable without specialized modifications. Residual RL does not learn the safe action set but learns the correction directly, parameterizing $\Delta u$ as a residual policy taking state $x$ and nominal action $u_{\text{ref}}$ as input, and incorporating the min-norm objective:

$$\max_{\pi_{\text{res}}} \mathbb{E}\left[\sum_t \mathbb{1}\{h(x_t) \ge 0\} - \lambda \lVert \pi_{\text{res}}(x_t, u_{\text{ref}}(x_t)) \rVert^2\right]$$

where $\lambda$ weights the min-norm penalty. The constant term is the "alive bonus" — already a hand-designed reward term in many practical RL-based robot learning frameworks. If the state is not directly accessible, observations can be substituted. On cart-pole, the residual RL policy closely reproduces the DH min-norm filtered input, only slightly more conservative; the authors argue this extra conservativeness acts as a robustness margin and is precisely why the safety rate improves on humanoid balance tasks.

Training Design for Humanoid Balance

Unitree G1 has very high state and input dimensionality, so carefully guiding residual RL training is essential for sufficient exploration, robustness, and generalization:

Reference policy. Trained with an established balance motion-tracking pipeline, but instead of tracking a single reference motion, the pipeline generates 10,000 balance motions covering a wide range of feasible humanoid configurations and behaviors with varying difficulty. Training on this diverse dataset encourages the safety policy to learn a general balance prior rather than overfit one trajectory.

Reward design. Compact: beyond the two terms above, two additional safety rewards are added — a base velocity stabilization term (discouraging excessive horizontal drift) and an action smoothness penalty (reducing abrupt control changes).

Training strategy. PPO with an asymmetric actor-critic: the actor is the deployable residual policy receiving only deployment-time observations (proprioceptive states plus the reference policy action); the critic, used only in training, accesses privileged simulation information such as ground-truth body states. Two additional techniques matter: introducing varying payloads at various body locations and random pushes to expose the policy to failure or near-failure states — without such excitation the policy learns an impractically conservative filter; and training the residual policy across multiple reference policy checkpoints so it generalizes to unseen unfiltered actions.

Experiments

Simulation: Performance–Robustness and Performance–Safety Trade-offs

Evaluated on Unitree G1 in IsaacGym against three baselines: the reference motion-tracking policy trained without payload domain randomization (Baseline), the same trained directly with payload randomization (Baseline*), and a PPO-trained discriminating hyperplane safety filter (DH).

The reading is clear: the baseline tracks reasonably but its fall rate climbs under random payloads — accurate motion tracking alone does not guarantee robustness when dynamics change. Training directly with payload randomization improves robustness but pays with higher tracking error (24.95 → 44.40 mm without payload, nearly double — classic over-conservatism). ResSafe achieves the lowest fall rate while keeping tracking error close to nominal: a better robustness–performance balance.

This ablation best demonstrates decoupling's value. Adding a moderate safety cost in a single policy does cut falls (1.17% → 0%), but larger safety weights severely degrade tracking without consistently improving safety (at weight 100, error explodes to 353.55 mm and the fall rate returns to 1.63%). ResSafe improves safety without severe tracking degradation — the direct payoff of separating nominal tracking from residual safety correction.

Figure 4: Fall rates across reference policy checkpoints. A residual policy trained on a single checkpoint degrades markedly on others; multi-checkpoint training stays robust.

The generalization study shows that a residual policy trained on a single reference checkpoint degrades significantly on other checkpoints with a much higher fall rate, whereas one trained across multiple checkpoints stays robust when deployed with unseen reference policies. The learned residual correction thus captures general safety-relevant behavior rather than overfitting to a specific controller. The DH filter is inferior on both counts.

Hardware Validation

On the physical G1, ResSafe outperforms both the baselines and the discriminating hyperplane in fall rate and tracking error across multiple motions and payload conditions. One honest detail: Baseline* tracks better under payload (it is explicitly trained with a tracking reward), but its error across motions without payload is worse due to over-conservatism from domain randomization. Overall, the residual safety filtering policy transfers to real hardware and improves safety and robustness on challenging motion tasks.

Significance and Limitations

ResSafe carves out a practical route between classical safety filtering (model-based, poorly scalable to high dimensions) and single-policy safe RL (coupled objectives, suboptimal Pareto frontier): learn the safety correction directly with residual RL, preserving min-norm filtering behavior (intervening only when necessary) without online optimization or an explicit dynamics model, and naturally supporting reuse across tasks and reference policies. The paper also connects it back to control-theoretic safety filtering — the learned residual behaves equivalently to a min-norm safety filter, an implicit safety-filtering mechanism.

The authors state five limitations: the filter is learned from data and therefore provides no hard guarantees in the classical sense; the residual policy is consistently more conservative than the DH filter, and a theoretical account of this remains open; the approach relies on standard sim-to-real techniques (system identification, domain randomization) without explicitly addressing sim-to-real transfer, which contributes to the simulation–hardware gap; generalization is bounded by the diversity of the training motion dataset, though fine-tuning can adapt to new motions; and the current framework is a "blind" safety filter without exteroceptive perception — whereas human balance and safety behavior relies heavily on visual feedback and environmental awareness, making perception-enabled safety an important direction for future work.