PRIOR: Perceptive Learning for Humanoid Locomotion with Reference Gait Priors

PRIOR: Perceptive Learning for Humanoid Locomotion with Reference Gait Priors

Authors: Chenxi Han, Shilu He, Yi Cheng, Linqi Ye, Houde Liu

Affiliations: Tsinghua University · ZERITH Robotics · Shanghai University

Paper: https://arxiv.org/abs/2603.18979

---

One-sentence Summary

PRIOR is a single-stage end-to-end RL framework on Isaac Lab that combines a parametric gait generator, a GRU state estimator, and terrain-adaptive footstep rewards to let the ZERITH Z1 humanoid achieve 100% traversal success on stairs, boxes, and gaps while maintaining natural human-like gaits.

Background and Motivation

Deploying humanoids in human-centric environments requires locomotion controllers that negotiate diverse terrain geometries — stairs, boxes, gaps — while preserving the natural bipedal gaits essential for safe coexistence. RL is the dominant paradigm, but recent work achieves either terrain-robust locomotion via perceptive policies or natural gait synthesis via motion priors — rarely both simultaneously without substantial system complexity: multi-stage teacher-student distillation, adversarial discriminators for style, or extensive sim-to-real calibration. These hinder reproducibility and raise the barrier to entry.

The framework is guided by three observations. First, LiDAR elevation mapping, while geometrically precise, relies on odometric integration that accumulates drift; egocentric depth images provide a self-contained terrain signal inherently free of such drift. Second, adversarial motion priors suffer from mode collapse, reward ambiguity, and hyperparameter sensitivity, exacerbated on challenging terrains where the policy must deviate from reference motions; a parametric gait generator offers comparable stylistic guidance via deterministic supervision, sidestepping these instabilities. Third, the community transition from Isaac Gym to Isaac Lab calls for pipelines natively built on the newer stack; PRIOR is developed entirely within Isaac Lab with systematic optimizations yielding a 3× speedup over the vanilla baseline.

Core Contributions

1. Parametric gait generator: Produces phase-conditioned joint trajectories by dynamically blending motion-capture primitives, supplying velocity-adaptive motion targets that replace adversarial style losses. Needs only a minimal set of high-fidelity motion templates.
2. GRU state estimator: Fuses proprioceptive history with egocentric depth, trained via self-supervised auxiliary objectives (heightmap reconstruction and linear-velocity prediction) to distill local terrain geometry into a compact latent — no external localization or manual annotation needed.
3. Terrain-adaptive footstep rewards: Bias swing-leg placement toward geometrically favorable contact regions for reliable footholds on discontinuous surfaces.

Method Details

Perception-Driven Locomotion Framework

PRIOR adopts an asymmetric actor-critic architecture trained single-stage end-to-end, letting perception and control co-evolve and avoiding the error amplification of two-stage distillation. The policy is optimized with PPO.

The actor takes (i) a 45-dim proprioceptive observation $\mathbf{o}_t = [\boldsymbol{\omega}_t, \mathbf{g}_t, \mathbf{c}_t, \boldsymbol{\theta}_t, \dot{\boldsymbol{\theta}}_t, \mathbf{a}_{t-1}]^\top$, (ii) a 163-dim state estimator output $\mathbf{e}_t$, and (iii) a 1-dim gait phase $\phi_t$. It outputs a 12-dim action $\mathbf{a}_t$ modulating the default standing configuration: $\boldsymbol{\theta}_t^{\text{target}} = \boldsymbol{\theta}^{\text{default}} + \mathbf{a}_t$, tracked by a PD controller:

$$\boldsymbol{\tau}_t = K_p(\boldsymbol{\theta}_t^{\text{target}} - \boldsymbol{\theta}_t) - K_d \dot{\boldsymbol{\theta}}_t$$

with $K_p=60.0$, $K_d=2.0$ matching ZERITH Z1 hardware. The critic receives noise-free base linear velocity $\mathbf{v}_t$, proprioception $\mathbf{o}_t$, height-map scan $\mathbf{m}_t$ from RayCaster, and reference gait phase $\phi_t$: $\mathbf{s}_t = [\mathbf{v}_t, \mathbf{o}_t, \mathbf{m}_t]^\top$.

State Estimator

The estimator fuses proprioception (stacking horizon $H_1=10$) and depth images (stacking horizon $H_2=2$, each frame cropped to [36,64]). Proprioception is encoded by an MLP to 128 dims; depth by a CNN to 128 dims; concatenated and fed to a single-layer GRU. The 163-dim output $\mathbf{e}_t = [\hat{\mathbf{v}}_t, \mathbf{z}_t, \mathbf{h}_t]^\top$ comprises a 3-dim base-velocity estimate, a 32-dim latent, and a 128-dim height-map latent. $\mathbf{h}_t$ is decoded to estimated terrain $\hat{\mathbf{m}}_t$; $\mathbf{e}_t$ predicts next-step proprioception $\hat{\mathbf{o}}_{t+1}$. Self-supervised MSE loss uses privileged critic information:

$$\mathcal{L} = \text{MSE}(\hat{\mathbf{v}}_t, \mathbf{v}_t) + \text{MSE}(\hat{\mathbf{o}}_{t+1}, \mathbf{o}_{t+1}) + \text{MSE}(\hat{\mathbf{m}}_t, \mathbf{m}_t)$$

Reference Gait Priors

Human motion-capture data from static postures to various forward velocities is retargeted to ZERITH Z1. Foot-contact information segments the data into stable gait cycles; the second or third cycle is selected to avoid non-stationary acceleration/deceleration dynamics, and a 1D Gaussian filter removes high-frequency noise. Given commanded velocity $\mathbf{v}=[v_x, v_y, \omega]$ and gait phase $\phi \in [0,1)$, reference trajectories are synthesized via weighted interpolation. The interpolation factor $\alpha = \text{clip}(\frac{|u_x| - u_l}{u_u - u_l + \varepsilon}, 0, 1)$, gait period $T_u = (1-\alpha)T_l + \alpha T_u$, phase update $\phi_{t+1} = (\phi_t + \frac{\Delta t}{T_u}) \bmod 1$, and blended trajectory $\theta_d(\phi) = (1-\alpha)\theta_l(\phi) + \alpha \theta_u(\phi)$. Near-zero velocity switches to a standing pose $\theta_{\text{stand}}$.

Gait-aware rewards use exponential tracking terms $r_i = \exp(-\lambda_i e_i)$ summed as $r_{\text{gait}} = \sum_i w_i r_i$, constraining pose consistency ($r_{\text{pos}}$, 0.10), velocity matching ($r_{\text{vel}}$, 0.05), motion-trend tracking ($r_\Delta$, 0.05), and key support-joint stabilization ($r_{\text{ankle}}$, 0.05).

High-Throughput Training Infrastructure

A heterogeneous observation-buffer management scheme uses VRAM only as a transient rendering buffer, asynchronously transferring observation tensors to CPU memory, decoupling physics from data caching. This supports 1024 parallel environments on a single RTX 4090 (24 GB), achieving a 3× speedup over vanilla Isaac Lab. Depth images update at 30 Hz; control at 50 Hz; policy converges in ~12000 iterations.

Method Overview

flowchart TD
    A[Depth images d_t H2=2] --> B[CNN encoder 128-dim]
    C[Proprioception o_t H1=10] --> D[MLP encoder 128-dim]
    B --> E[GRU memory module]
    D --> E
    E --> F["State estimate e_t: v_hat, z, h 163-dim"]
    F --> G[Actor network]
    H[Gait phase phi_t] --> G
    I[Parametric gait generator] --> H
    I --> J[Reference trajectory theta_d]
    J --> K[Gait-aware reward r_gait]
    G --> L[Action a_t 12-dim]
    L --> M[PD controller Kp=60 Kd=2]
    M --> N[ZERITH Z1 12DoF legs]
    K --> G

Experimental Results

Setup

Training on a single RTX 4090, 1024 parallel environments, depth at 30 Hz, control at 50 Hz. The policy is exported via ONNX and deployed on ZERITH Z1's onboard compute (23 DoF: 6 per leg, 3 waist, 4 per arm).

Figure 1: PRIOR framework demonstrated on ZERITH Z1. (A)–(D) show traversal of four representative terrain types.

Ablation Study

Method	Mean Level	Pyramid Stairs	Inverted Stairs	Boxes	Plane	Mean Reward
PRIOR (full)	5.7533	1.0	1.0000	1.0000	1.0	26.3462
w/o reference gait	5.7735	1.0	1.0000	1.0000	1.0	23.7233
w/o $\hat{\mathbf{m}}_t$	5.7672	1.0	0.7734	1.0000	1.0	13.1775
w/o $\mathbf{d}_t^{H2}$	5.4627	1.0	0.3750	0.9687	1.0	10.1463
H1=6	5.7417	1.0	1.0000	1.0000	1.0	19.3234
w/o landing reward	5.7403	1.0	1.0000	1.0000	1.0	22.6262

Although "w/o reference gait" achieves a slightly higher mean curriculum level (5.7735), its reward (23.7233) is ~10% lower than the full framework, showing traversal capability alone does not guarantee motion quality. Without gait prior, policies exploit unstable or high-frequency oscillatory motions; with the humanoid gait constraint the robot maintains near-perfect success (all 1.0) with smoother, more energy-efficient locomotion. Removing explicit terrain estimation drops reward to 13.1775 and inverted-stairs success to 0.7734. Removing temporal depth yields the worst performance (reward 10.1463, inverted stairs 0.375), losing the ability to anticipate terrain variations. Reducing proprioceptive history to H1=6 drops reward to 19.3234; a longer window implicitly estimates ground friction and CoM deviation. Removing landing rewards drops reward to 22.6262 with less stable foot placement.

Figure 2: PRIOR framework overview — asymmetric actor-critic, state estimator (yellow), and reference gait generator (blue).

Landing State Reward Components

Reward	Description	Weight
$r_{\text{air}}$	Promotes gait rhythm	1.25
$r_{\text{slide}}$	Minimizes ground slipping	-0.10
$r_{\text{dbl-air}}$	Penalizes one-leg walking	-1.00
$r_{\text{swing}}$	Ensures leg lift height	-20.0
$r_{\text{stumble}}$	Prevents foot-obstacle tripping	-30.0
$r_{\text{edge}}^{L/R}$	Encourages safe foot placement	-2.00

$r_{\text{stumble}}$ has the largest magnitude (-30.0), reflecting that foot tripping is among the most dangerous failure modes in humanoid stair climbing. $r_{\text{swing}}$ (-20.0) ensures sufficient swing-leg clearance over step edges. $r_{\text{air}}$ (1.25) encourages regular gait rhythm, preventing shuffling strategies. $r_{\text{edge}}$ guides footholds toward step-tread centers.

Figure 5: Training curves. Top: mean reward; bottom: mean terrain level. The full PRIOR framework achieves the highest reward while maintaining high terrain level.

Figure 6: Foot-placement behavior with and without landing state reward. Without it, foot-placement stability degrades.

观测向量

$$ \mathbf{o}_{t}=\left[\boldsymbol{\omega}_{t},\;\mathbf{g}_{t},\;\mathbf{c}_{t},\;\boldsymbol{\theta}_{t},\;\dot{\boldsymbol{\theta}}_{t},\;\mathbf{a}_{t-1}\right]^{\top} $$

PD控制力矩

$$ \boldsymbol{\tau}_{t}=K_{p}(\boldsymbol{\theta}_{t}^{\mathrm{target}}-\boldsymbol{\theta}_{t})-K_{d}\dot{\boldsymbol{\theta}}_{t} $$

内部模型损失

$$ \mathcal{L}=\mathrm{MSE}\!\left(\hat{\mathbf{v}}_{t},\mathbf{v}_{t}\right)+\mathrm{MSE}\!\left(\hat{\mathbf{o}}_{t+1},\mathbf{o}_{t+1}\right)+\mathrm{MSE}\!\left(\hat{\mathbf{m}}_{t},\mathbf{m}_{t}\right) $$

Limitations

1. Limited motion templates: The gait generator relies on a small set of mocap templates (static to various forward speeds). Lateral movement, turning, and other complex modes may be under-modeled. The hard standing threshold $v_{\text{th}}$ may cause discontinuities in the transition region.
2. Sim-only success rates: The 100% traversal success is achieved in the simulation curriculum; the paper provides no quantitative real-world traversal data. Although the policy is deployed via ONNX on ZERITH Z1, the sim-to-real gap (sensor noise, unmodeled dynamics) may affect real performance.
3. Compute dependency: Training requires a single RTX 4090 24 GB for 1024 parallel environments. The 30 Hz depth update may be insufficient for high-speed scenarios; perception latency may cause foothold-planning lag during dynamic terrain changes.

Conclusion and Outlook

PRIOR is an efficient, reproducible single-stage RL framework on Isaac Lab for perception-aware humanoid locomotion. It combines a GRU-based explicit terrain-reconstruction state estimator with a parametric gait generator in a unified pipeline, enabling ZERITH Z1 to traverse diverse challenging terrains with high precision and robustness. Experiments show high average rewards and 100% traversal success across complex terrains, with the learned policy handling relatively high difficulty, demonstrating effectiveness and generalization.

The core insight is that natural gaits and terrain robustness need not be achieved simultaneously through complex adversarial training or multi-stage distillation. The parametric gait generator replaces adversarial style losses with deterministic supervision; the self-supervised state estimator distills terrain geometry from depth without external localization. Both co-evolve in single-stage end-to-end training, mutually reinforcing. The ablation proves each component contributes complementary, indispensable gains — temporal depth is especially critical, with inverted-stairs success dropping from 100% to 37.5% when removed.

Golden Quote

Nature need not come from adversaries — when gait priors are injected as deterministic trajectories rather than a discriminator, the robot learns "how to walk like a human" instead of "how to fool the judge."