CReF: Cross-modal and Recurrent Fusion for Depth-conditioned Humanoid Locomotion

Authors: Yuan Hao, Ruiqi Yu, Shixin Luo, Guoteng Zhang, Jun Wu, Qiuguo Zhu (Institute of Cyber-Systems and Control, Zhejiang University; School of Control Science and Engineering, Shandong University)
Link: arXiv:2603.29452 [cs.RO] · Project: github.com/cometlogic/cref.github.io (project page repository listed in the paper; not publicly reachable at the time of writing) · Code: not released as of 2026-09-03

One-line summary

CReF maps onboard proprioception and a forward-facing depth image directly to humanoid joint-position targets, with no height map, no auxiliary geometric supervision, and no privileged teacher. Its three architectural ingredients — proprioception-queried cross-modal attention, a gated residual fusion block, and a GRU recurrent branch regulated by a highway output gate — are complemented by a terrain-aware foothold placement reward derived from foot-end point clouds. Trained in Isaac Gym with 4096 environments and deployed zero-shot on an AGIBOT X2 Ultra, it traverses stairs, gaps, platforms, and real-world scenes including handrails and strong reflective interference.

Background and Motivation

Perceptive humanoid locomotion has so far been dominated by two families of designs, both of which insert an explicit geometric abstraction between sensing and control. The first family mediates perception through robot-centric 2.5D elevation maps: early systems used them for foothold selection and predictive control, later learning-based controllers conditioned on terrain maps, and recent attention-based map encoders let the policy emphasize future steppable regions. The interface is compact and interpretable, but the policy's performance is bounded by the fidelity of the elevation representation, which by construction cannot express overhangs, vertical clutter, or perforated obstacles.

The second family consumes depth directly but shapes the depth branch with auxiliary geometry-related objectives: terrain reconstruction, decoder-constrained latent estimation, or distillation from privileged teachers that are themselves grounded in geometric abstractions. These targets supply structured supervision and often improve optimization and transfer, yet they anchor the learned representation to the inductive bias of the auxiliary target. The paper frames this precisely: the blind spots of the intermediate representation are not removed but inherited through supervision. A forward-facing camera cannot see the underfoot region, so auxiliary targets were also used historically to inject near-foot geometry — but the cost is a representational ceiling.

A further practical concern with the transfer-oriented designs in this literature is that they often rely on synthetic depth corruption, realistic stereo-artifact simulation, or calibration uncertainty injection to narrow the sim-to-real gap. These techniques can bias robustness toward the assumed artifact distribution and increase training complexity. CReF asks whether a carefully designed single-stage policy — raw depth plus proprioception in, joint targets out — can reach the front of terrain difficulty without any of this machinery.

The motivating scenarios are the ones where geometric anticipation genuinely matters: stair descent, where foothold errors cause edge contacts and slipping; gaps and abrupt height transitions; and cluttered human environments where the terrain is not a clean height field. The paper's contribution list is correspondingly threefold: (i) the single-stage depth-conditioned framework CReF, achieving stable performance at the leading terrain-difficulty levels across all trained categories; (ii) a terrain-aware foothold placement reward that extracts supportable candidates from foot-end point-cloud samples and shapes touchdown toward them; and (iii) real-world zero-shot deployment across multiple terrain-transition tasks and broader unseen environments.

Preliminaries: Platform and Training Setup

The deployment platform is the AGIBOT X2 Ultra humanoid: approximately 1.31 m tall, roughly 39 kg, with an official peak joint torque of 120 Nm. Onboard sensing is a forward-facing Intel RealSense D435i pitched 25° downward relative to the horizontal plane, and inference runs on an NVIDIA Jetson AGX Orin. The policy is transferred zero-shot from simulation with no task-specific fine-tuning; depth-frame selection uses hardware timestamps fitted by linear regression to map camera time to system time, giving an effective latency of 20 ms.

Training runs in Isaac Gym with 4096 parallel environments at a 50 Hz control frequency. Depth is rendered at $64\times 48$ resolution and updated at 20 Hz. Depth rendering is implemented with NVIDIA Warp: robot links are approximated as capsules and self-occlusion is computed on the GPU via first-hit ray-capsule intersection. A single RTX 4090 trains for about 30 hours (20,000 iterations). Notably, no synthetic depth corruption is injected during training — the sim-to-real gap is bridged by the policy structure and training distribution alone. Optimization is standard PPO with an asymmetric critic: besides the proprioceptive observation $\mathbf{o}^{p}_{t}$, the value network receives the ground-truth base linear velocity $\mathbf{v}_{t}$ and a robot-centric terrain height observation $\mathbf{m}_{t}$ (local elevation samples), forming $\mathbf{s}_{t}=\big[\mathbf{o}^{p}_{t},\;\mathbf{v}_{t},\;\mathbf{m}_{t}\big]$. For comparison, the Humanoid Parkour Learning (HPL) baseline — originally demonstrated on the full-size Unitree H1 (≈1.8 m, ≈47 kg, ≈360 Nm knee torque) — is re-implemented on the X2 Ultra so the comparison is controlled; the original HPL results serve only as a terrain-difficulty reference.

Method

Observation and action space. At each control step the actor consumes the proprioceptive observation and the raw depth image $\mathbf{D}_{t}\in\mathbb{R}^{H\times W}$:

$$\mathbf{o}^{p}_{t}=\Big[\boldsymbol{\omega}_{t},\;\mathbf{r}^{\mathrm{grav}}_{t},\;\mathbf{u}^{\mathrm{cmd}}_{t},\;\mathbf{q}_{t}-\mathbf{q}_{0},\;\dot{\mathbf{q}}_{t},\;\mathbf{a}_{t-1}\Big],\qquad \mathbf{o}_{t}=\big(\mathbf{o}^{p}_{t},\;\mathbf{D}_{t}\big).$$

The action is a residual around the nominal standing pose, tracked by a low-level PD controller: $\mathbf{q}^{\mathrm{target}}_{t}=\mathbf{q}_{0}+\mathbf{a}_{t}$. The raw depth is normalized as $\mathbf{D}_{t}=\mathbf{D}^{\mathrm{raw}}_{t}/d_{\max}-0.5$ with $d_{\max}$ the preset maximum range — a small but important detail, since centering the normalized depth keeps the input distribution stable across near and far returns.

Depth tokenizer and velocity estimator. A lightweight CNN tokenizer converts the normalized depth image into $N$ local tokens of dimension $d$: $\mathbf{Z}_{t}=\mathcal{T}_{\theta}(\mathbf{D}_{t})\in\mathbb{R}^{N\times d}$. Jointly, an auxiliary base-velocity estimator $\hat{\mathbf{v}}_{t}=\mathrm{LSTM}\!\left([\mathbf{o}^{p}_{t};\mathcal{C}_{\psi}(\mathbf{D}_{t})]\right)$ — with $\mathcal{C}_{\psi}$ a compact depth compression module — is supervised by simulator ground-truth base linear velocity with an $\ell_{2}$ loss. This matters because the X2 Ultra's state estimator cannot be trusted for accurate base linear velocity from onboard sensing alone during aggressive terrain interaction, and the depth stream carries implicit ego-motion information (optical-flow-like structure) that an LSTM can exploit.

Cross-modal attention. The proprioceptive tokenizer $\mathcal{P}_{\phi}$ ingests proprioception concatenated with the estimated base velocity, $\mathbf{e}^{p}_{t}=\mathcal{P}_{\phi}\!\left([\mathbf{o}^{p}_{t};\hat{\mathbf{v}}_{t}]\right)$, while depth tokens are layer-normalized, $\mathbf{E}^{d}_{t}=\mathrm{LN}(\mathbf{Z}_{t})$. The key design decision is that the proprioceptive token is the query:

$$\mathbf{Q}_{t}=\mathrm{LN}(\mathbf{e}^{p}_{t})\mathbf{W}_{q},\qquad\mathbf{K}_{t}=\mathbf{E}^{d}_{t}\mathbf{W}_{k},\qquad\mathbf{V}_{t}=\mathbf{E}^{d}_{t}\mathbf{W}_{v},\qquad \bar{\mathbf{e}}^{d}_{t}=\mathrm{MHA}\!\big(\mathbf{Q}_{t},\mathbf{K}_{t},\mathbf{V}_{t}\big),$$

with $\mathbf{Q}_{t}\in\mathbb{R}^{1\times d}$ and $\mathbf{K}_{t},\mathbf{V}_{t}\in\mathbb{R}^{N\times d}$. Because the query encodes the robot's current dynamic state, attention retrieves terrain information that is relevant to the current pose and velocity — e.g., the step the swing foot is about to take rather than uniformly-weighted geometry. The output is concatenated with the proprioceptive token: $\mathbf{x}_{t}=[\mathbf{e}^{p}_{t};\bar{\mathbf{e}}^{d}_{t}]$.

Gated residual fusion (GRF). The fusion input passes through a shared latent projection $\tilde{\mathbf{x}}_{t}=\phi\!\big(\mathbf{W}_{1}\,\mathrm{LN}(\mathbf{x}_{t})+\mathbf{b}_{1}\big)$ with ELU activation, then a second linear layer splits it into a content branch and a gate branch, $\begin{bmatrix}\mathbf{c}_{t}\\\mathbf{g}_{t}\end{bmatrix}=\mathbf{W}_{2}\,\tilde{\mathbf{x}}_{t}+\mathbf{b}_{2}$ with $\mathbf{c}_{t},\mathbf{g}_{t}\in\mathbb{R}^{2d}$, and the fused feature is

$$\mathbf{f}_{t}=\mathbf{x}_{t}+\mathbf{c}_{t}\odot\sigma(\mathbf{g}_{t}).$$

The residual path keeps optimization stable while the sigmoid gate controls, channel by channel, how much of the multimodal correction is admitted. This is the block whose removal hurts most in the mid-to-hard regime after cross-attention (Table II: overall 90.45% → 83.78%).

Recurrent fusion with a highway gate. Temporal integration addresses the fact that a single forward-facing depth frame cannot disambiguate terrain structure or contact timing — the camera looks ahead, not underfoot. A GRU accumulates short-horizon context, $\mathbf{h}_{t}=\mathrm{GRU}(\mathbf{f}_{t},\mathbf{h}_{t-1})$, $\mathbf{z}^{\mathrm{rec}}_{t}=\mathbf{W}_{h}\mathbf{h}_{t}$, and a highway-style output gate blends recurrent and feedforward features:

$$\beta_{t}=\sigma\!\big(\mathbf{W}_{\beta}[\mathbf{z}^{\mathrm{rec}}_{t};\mathbf{f}_{t}]+\mathbf{b}_{\beta}\big),\qquad\mathbf{y}_{t}=\beta_{t}\odot\mathbf{z}^{\mathrm{rec}}_{t}+(1-\beta_{t})\odot\mathbf{f}_{t}.$$

The gate is state-dependent by construction: it can lean on memory during ambiguous or risky phases (flight, steep posture, terrain transitions) and fall back to a strong feedforward response when the current observation already suffices. Section III-B3 of the paper verifies exactly this behavior quantitatively.

Terrain-aware foothold placement reward. The second contribution is a reward that shapes where the foot lands. Each foot maintains a short buffer of point-cloud samples in its local frame. After command-conditioned forward gating — filtering near-foot points with a minimum forward distance equal to the distance covered in 0.5 s under the current forward command — the remaining points are partitioned into overlapping candidate windows of size $24\,cm\times 10\,cm$ with a 4 cm stride. For the $k$-th window of foot $f$ at time $t$, $\mathcal{P}^{f}_{t,k}=\{\mathbf{p}^{f}_{t,k,i}\}_{i=1}^{n^{f}_{t,k}}$, the mean and covariance are computed and eigen-decomposed,

$$\boldsymbol{\mu}^{f}_{t,k}=\frac{1}{n^{f}_{t,k}}\sum_{i=1}^{n^{f}_{t,k}}\mathbf{p}^{f}_{t,k,i},\qquad \Sigma_{t,k}^{f}v_{t,k,j}^{f}=\lambda_{t,k,j}^{f}v_{t,k,j}^{f},\qquad\lambda_{t,k,1}^{f}\leq\lambda_{t,k,2}^{f}\leq\lambda_{t,k,3}^{f},$$

and roughness is defined as $\rho^{f}_{t,k}=\sqrt{\max(\lambda^{f}_{t,k,1},0)}$ — the square root of the smallest eigenvalue, i.e., the thickness of the point patch off its best-fit plane. A window is accepted as a supportable foothold candidate if it is sufficiently planar, approximately horizontal, and not recessed:

$$\rho_{t,k}^{f}<r_{\mathrm{th}},\qquad|v_{t,k,1,z}^{f}|>\eta_{\mathrm{th}},\qquad\mu_{t,k,z}^{f}>h_{\min},$$

where the eigenvector $v_{t,k,1}^{f}$ corresponds to the surface normal and the $z$-component test enforces near-horizontality. Candidates refresh only at liftoff and stay fixed until the next touchdown to avoid chattering. At touchdown, the distance in the $x$-$z$ plane between the realized contact position and the nearest candidate is

$$d^{f}_{xz}=\min_{k}\left\|\begin{bmatrix}p^{f}_{x,t}\\p^{f}_{z,t}\end{bmatrix}-\begin{bmatrix}p^{f,\star}_{x,t,k}\\p^{f,\star}_{z,t,k}\end{bmatrix}\right\|_{2},\qquad r_{\mathrm{fh}}=\sum_{f\in\mathcal{F}}I^{f}_{\mathrm{td}}\exp\!\left(-d^{f}_{xz}/s_{xz}\right).$$

The crucial distinction from prior work: BeamDojo-style foot contact quality rewards are prohibitive — they punish bad contacts after they happen. This reward is directional and anticipatory — it pulls the swing foot toward locally supportable regions before contact occurs. That difference is visible in the stair-touchdown distributions below.

The full reward table (Table I of the paper) contains 21 terms: low-speed-aware linear velocity tracking $\exp\!\left(-\frac{\mathbb{I}_{s}\|\mathbf{e}_{v}\|_{1}+(1-\mathbb{I}_{s})\|\mathbf{e}_{v}\|_{2}^{2}}{\sigma_{v}}\right)$ (switching to an $\ell_1$ form below a speed threshold $v_s$), angular velocity tracking, orientation, a contact-shaping term $\mathbb{I}_{\mathrm{stand}}+(1-\mathbb{I}_{\mathrm{stand}})\,\mathbb{I}^{0.2\mathrm{s}}_{\mathrm{single}}$ that rewards standing while encouraging short-horizon single-foot contact, feet distance, the foothold placement term above, base height relative to the support-foot average $\bar{z}^{\mathrm{supp}}$, and a battery of regularization and safety penalties (action rate, action smoothness, foot slip, foot impact acceleration/velocity, joint limits, hip and ankle position limits, stumble). The average support-foot height is itself a small construction: $\bar{z}^{\mathrm{supp}}=\frac{\sum_{f}\mathbb{I}(F^{f}_{z}>1)\,z^{f}}{\sum_{f}\mathbb{I}(F^{f}_{z}>1)}$.

graph TD
  DT["Forward depth D_t 64x48"] --> NM["Normalize: D_raw/d_max - 0.5"]
  NM --> TK["CNN depth tokenizer Z_t"]
  TK --> LN1["LayerNorm to E_d"]
  OB["Proprioception o_p + velocity est."] --> PT["Proprio tokenizer e_p"]
  PT --> QQ["Q = LN(e_p) W_q"]
  LN1 --> KK["K = E_d W_k"]
  LN1 --> VV["V = E_d W_v"]
  QQ --> MHA["Cross-modal MHA"]
  KK --> MHA
  VV --> MHA
  MHA --> XX["x_t = [e_p ; e_d_bar]"]
  PT --> XX
  XX --> GRF["Gated residual fusion
f_t = x_t + c_t * sigmoid(g_t)"]
  GRF --> GRU["GRU temporal fusion
z_rec_t = W_h h_t"]
  GRF --> GT["Highway gate beta_t"]
  GRU --> GT
  GT --> YY["y_t = beta*z_rec + (1-beta)*f"]
  YY --> HD["Action MLP head"]
  HD --> AC["Joint targets q_0 + a_t"]
  RD["Foot point-cloud reward branch
Eq. 17-23 (training only)"] -.-> GRF

The diagram traces the actor's forward pass; the reward branch is dashed because it exists only during training. No module of the policy consumes the candidate footholds explicitly — the reward shapes the policy, and at deployment the policy's own features decide where to step.

Experiments

Evaluation covers three questions: (i) does the full framework beat its ablations and a perceptive baseline on terrain traversal; (ii) does the foothold reward improve stair contact placement; and (iii) does the highway gate recruit recurrent memory in the intended situations. Each policy is evaluated in 2000 parallel environments with rollouts up to 40 s; stair foot-placement analysis uses 2048 parallel stair rollouts; gate statistics come from 4096 parallel environments under a fixed forward command. MuJoCo serves only as an additional out-of-distribution cross-simulator test.

Architectural comparison. Full CReF leads the overall success rate across stairs (riser/tread 10–20 cm trained, 23 cm OOD), gaps (30–75 cm trained, 80 cm OOD), and platforms (20–40 cm trained, 43 cm OOD). The advantage widens exactly where it should — at the hardest and out-of-distribution settings:

Method	Stairs Hard (20/30)	Stairs OOD (23/30)	Gap Hard (75 cm)	Gap OOD (80 cm)	Platform OOD (43 cm)	MuJoCo OOD	Overall
Full CReF	97.85	72.75	92.15	44.70	84.35	100% (20/20)	90.45
w/o Cross-Attn	78.70	34.00	83.00	8.60	64.80	95% (19/20)	78.56
w/o GRF	83.60	49.10	90.05	29.95	68.55	95% (19/20)	83.78
w/o Highway Gate	93.65	50.20	86.35	17.20	73.10	100% (20/20)	83.29
HPL (re-impl.)	71.05	27.40	78.25	20.85	55.15	5% (1/20)	74.57

Removing cross-modal attention causes the largest degradation — on OOD gaps the success rate collapses from 44.70% to 8.60% — confirming that state-conditioned depth feature extraction is the load-bearing component. Removing the highway gate costs 22.75 points on OOD stairs while leaving easy stairs nearly intact (98.40 vs 99.75), precisely the asymmetric signature of a module that matters only under ambiguity. The re-implemented HPL baseline trails across the board and manages only 1 of 20 MuJoCo OOD terrains, though the paper is careful to note that HPL's published results were obtained on the much larger Unitree H1 and are not a like-for-like benchmark.

Stair failures split by direction. Table III of the paper separates ascending and descending failures out of 2000 stair environments per difficulty. Across every method, descending failures grow much faster than ascending ones as difficulty rises — descent is the hard regime. Full CReF records the fewest descending failures at every difficulty, e.g., 49 versus 338 (w/o Cross-Attn), 289 (w/o GRF), 88 (w/o Highway Gate), and 338 (HPL) on hard stairs, and 454 versus at least 703 for every variant on OOD stairs. This is consistent with the foothold reward being anticipatory: descending requires committing the swing foot toward a lower support region before visual evidence is complete.

Foothold distribution analysis. Comparing against a foot contact quality reward (FCQR, following BeamDojo) under otherwise identical training, the proposed reward tightens the touchdown distributions on hard stairs: the median absolute deviation within the tread drops from 3.0 cm to 1.5 cm in ascent and from 2.8 cm to 1.4 cm in descent, and the logged ankle-riser collisions in ascending rollouts disappear entirely. The gain is not a shift of the touchdown mean — the whole distribution concentrates, which is the signature of improved placement precision and repeatability rather than a re-aimed average.

Highway gate behavior. Gate values (averaged over channels, aggregated by state group) are higher on step-like terrains than on flat ground, higher during flight than during stable support, and higher in risky states — defined as $|\mathrm{roll}|>0.20$ rad or $|\mathrm{pitch}|>0.20$ rad — than in stable states. Each of these three trends matches the gate's intended function: emphasize recurrent memory exactly when instantaneous observations are least sufficient.

Real-world zero-shot transfer. On the physical X2 Ultra, each indoor task is assessed over 20 trials; one stair trial is a complete ascent-descent traversal of indoor stairs with 15 cm rise and 30 cm tread:

Task	Setting	Success / Trials
Stairs	Indoor, 15 cm rise / 30 cm tread	20/20
Platform	Indoor, 40 cm	20/20
Gap	Indoor, 80 cm	18/20
OOD terrain	Indoor OOD scene	19/20

Beyond the standardized tasks, the policy stably climbs real stairs with side railings at 20 cm rise and 26 cm tread under changing illumination, and remains effective in scenes with severe depth degradation — large invalid depth holes from strong reflection, dense vegetation, asymmetric boundaries, and other non-traversable clutter that produces substantial depth out-of-distribution effects. Because no synthetic depth corruption was used in training, these transfers are evidence that the architecture itself, rather than domain-randomization machinery, is carrying the robustness. The paper also reports more than 20 consecutive stair traversals in a single deployment session (Fig. 1).

Fig. 1 (from the paper): CReF enables robust real-world humanoid locomotion, including more than 20 consecutive stair traversals, a 40 cm platform, an 80 cm gap, and real-world stairs with 20 cm rise and 26 cm tread.

Fig. 2 (from the paper): overview of the single-stage depth-conditioned policy combining cross-modal attention, gated residual fusion, recurrent fusion, and the terrain-aware foothold placement reward.

Fig. 3 (from the paper): representative real-world rollouts across stair traversal with side railings, entrance-step and platform-like transitions, and outdoor pathways; highlighted regions show where environmental factors introduce out-of-distribution depth observations.

Fig. 4 (from the paper): highway gate values under different traversal conditions; higher values indicate stronger reliance on recurrent features.

Limitations

The authors state the main limitation directly: the system relies on active depth measurements from the RealSense sensor, which are sensitive to illumination and reflective surfaces, and depth alone discards appearance and texture cues that could improve perceptual robustness. Their proposed direction is binocular RGB-based sensing to jointly recover depth structure and texture.

Several further limitations are visible from the evidence. First, the strongest real-world numbers come from a single robot platform and a modest trial count (20 per task); there is no cross-platform validation, and the comparison with HPL is explicitly not like-for-like because of the hardware gap between the X2 Ultra and the Unitree H1. Second, the foothold reward operates on simulator point-cloud samples during training; its planarity, horizontality, and non-recession thresholds ($r_{\mathrm{th}}$, $\eta_{\mathrm{th}}$, $h_{\min}$) are fixed by hand, and their sensitivity is not characterized. Third, depth is rendered at only $64\times 48$ and 20 Hz, so the approach's behavior with higher-fidelity or noisier real sensors at higher rates remains untested, and the absence of synthetic depth corruption means robustness is demonstrated empirically rather than guaranteed across artifact distributions. Finally, code is not publicly released as of this writing (the cited project-page repository is not reachable), which limits reproducibility.

Conclusion and Outlook

CReF makes a clean case that perceptive humanoid locomotion does not need geometric intermediates: a single-stage policy with the right fusion structure — proprioception-queried attention, gated residual fusion, and a state-gated recurrent branch — plus a directional foothold reward reaches the front of terrain difficulty in every trained category and transfers zero-shot to handrails, reflective interference, and outdoor clutter. The ablations are unusually informative: each removed component degrades performance most in the regime it was designed for (cross-attention on OOD gaps, the highway gate on OOD stairs), which is strong evidence the architecture is doing what it claims. The honest limitations — active depth fragility, single-platform real-world evaluation, hand-tuned foothold thresholds — point to the natural next steps: binocular RGB sensing, cross-platform transfer, and release of the training code.

Golden quote

"The blind spots of the intermediate are not removed, but inherited through supervision." — the paper's critique of auxiliary geometric targets, and the founding argument for single-stage end-to-end depth-conditioned locomotion.