GigaBrain-WBC-0.5: A Behavior World Model for Robust Whole-Body Control with Environment Interaction

Paper Information

Title: GigaBrain-WBC-0.5: A Behavior World Model for Robust Whole-Body Control with Environment Interaction

Authors: Ziyang Cheng, Tianshu Tang, Jinxin Lan, Xinze Chen, Yuhan Gong, Zhichao Liu, Changzhong Wu, Yahao Mao, Zongyan Deng, Mingxuan Ma, Huasen Xi, Yilong Liu, Yutong Wu, Xiaofeng Wang, Yang Wang, Yun Ye, Guan Huang, Xiaojie Jin, Zheng Zhu, Jiwen Lu

Link: arXiv:2608.18234

Code: No public code repository available. The paper states "All code and experimental configurations will be released as open-source," but nothing has been released at the time of ingestion.

Background and Motivation

Humanoid robots are valuable because they can use hands and feet together in environments built for people. Whole-body motion tracking policies turn a humanoid into a robust control interface: the teleoperator—or an upstream model—only supplies a coarse movement intent, while the low-level policy keeps the robot balanced and physically feasible. Such a tracker serves both as a data-collection interface and as the execution layer beneath learned high-level policies.

Existing trackers, however, live on flat ground. Almost all current controllers (SONIC, HoloMotion-1, Humanoid-GPT) are trained and evaluated in an empty scene with a flat floor. The robot never steps onto anything, sits on anything, or carries a load. The policy is never asked to notice that its own dynamics have changed—that ground reaction force now arrives 18 cm higher, or that a payload has displaced its center of mass—and never learns to exploit such changes.

The first obstacle is data. An environment-interactive tracker needs terrain and object geometry paired with consistent reference motions at scale. SceneBot attempted to address this by reconstructing the scene from motion in hindsight, but its terrain is a 2.5D elevation map that cannot express the spatial geometry whole-body interaction requires—chair seats with clearance underneath, table edges, handrails—and its corpus is only 7.5 hours of specialized interaction data, driven by per-link contact labels that no teleoperator can produce online.

A second, less obvious difficulty is that the robustness recipe for flat-ground trackers does not transfer. On flat ground, the field's answer to command robustness has been to enlarge the reference distribution until the policy stays balanced under almost any command. Once the environment varies, that answer fails: a wide lunge feasible on flat ground may be infeasible on a narrow step, and leaning back into a chair is nonsense on flat ground. The feasible set becomes conditional on the environment, and RL policies behave unpredictably once a command departs from training distribution—a risk terrain sharply amplifies.

A further requirement: when a command is infeasible, an online mechanism must recognize it and respond with a best-effort motion rather than an emergency stop or fallback to standing—both abort the operator's task and, on a staircase, are more dangerous than continuing. Similarly, when the robot has fallen, handing control to a specialist get-up controller interrupts the task just as the fall did. Remaining driveable when the command is impossible and when the body is down are complementary halves of robust whole-body control.

Core Contributions

Contribution 1: Behavior World Model. The policy predicts not only its next action but also its next state and the distribution over its next latent behavior command. Training the controller this way changes what it must represent: predicting its own next state requires internalizing the contact dynamics currently acting on it, and predicting the next command distribution requires modeling which behaviors the environment admits—precisely what flat-ground trackers never need.

Contribution 2: Automatic Spatial Terrain Annotation. Recovers genuine 3D contact geometry—chairs, tables, boxes, stair treads—directly from retargeted motion. Unlike elevation maps, the output is real spatial geometry, allowing terrain-paired motion corpora at the scale of existing motion datasets rather than scene-capture sessions.

Contribution 3: Online OOD Command Filter. Driven by the world model's own predicted distribution, identifies out-of-distribution commands and projects them onto the nearest behavior the policy can execute, rather than rejecting them. The filter is stateless and closed-form, exposing a single runtime-tunable safety radius.

Method

Problem Setting

The system controls a Unitree G1 humanoid with 29 actuated DOFs at 50 Hz. At each step $t$, the policy receives proprioceptive observation $\bm{s}_t \in \mathbb{R}^{67}$ (projected gravity, base angular velocity, torso linear acceleration, joint positions and velocities), the previous action $\bm{a}_{t-1} \in \mathbb{R}^{29}$, and a reference window $\bm{c}_t$ over the next 10 frames; it outputs PD joint targets $\bm{a}_t \in \mathbb{R}^{29}$.

Figure 1: GigaBrain-WBC-0.5 overview. Automatic motion–terrain annotation coupled with a causal Behavior World Model.

Behavior World Model

The reference window $\bm{c}_t \in \mathbb{R}^{440}$ comprises 10 future frames, each containing: (i) 29 reference joint positions, (ii) 6D rotation of the reference root relative to the current robot root, (iii) reference root translation relative to the previous frame, (iv) reference root translation relative to the current robot root, and (v) the world down direction in the reference root frame.

An MLP encoder $E$ produces a continuous latent behavior command $\bm{z}^{\mathrm{raw}}_t = E(\bm{c}_t + \bm{\epsilon}_t) \in \mathbb{R}^{64}$, represented as two 32-dimensional tokens. Latent space shaping uses a finite scalar quantizer and a small MLP decoder attached during training only—the decoder maps the quantized latent back to the clean 440-d window, while the policy always receives the unquantized latent. This makes quantization a pure regularizer: with 32 levels per dimension, a latent that folds too much information into too few dimensions becomes indistinguishable after rounding and fails to reconstruct.

The per-frame input $\bm{e}_t = [\bm{s}_t, \bm{a}_{t-1}, \bm{z}_t] \in \mathbb{R}^{160}$ is projected to 256 dimensions and processed by a 6-layer causal Transformer with 4 heads and rotary position embeddings. Three linear heads emit:

$$\bm{a}_t \in \mathbb{R}^{29}, \quad \hat{\bm{s}}_{t+1} \in \mathbb{R}^{67}, \quad \mathcal{G}_t = \{\pi_k, \bm{\mu}_k, \log \bm{\sigma}_k\}_{k=1}^{K} \in \mathbb{R}^{516}$$

with $K=4$ diagonal Gaussian components over the next latent command $\bm{z}_{t+1} \in \mathbb{R}^{64}$. The first head is the policy; the other two make the network a world model. Predicting $\bm{s}_{t+1}$ forces representation of current contact dynamics—exactly what changes when stepping onto a box or picking up a load. Predicting $p(\bm{z}_{t+1} \mid \text{history}_t)$ forces modeling which behaviors the situation admits. The mixture is deliberately multi-modal: distinct behavior modes occupy separate latent regions, and collapsing them into a single Gaussian would smear them together.

The full objective combines PPO loss with four auxiliary terms:

$$\mathcal{L} = \mathcal{L}_{\mathrm{PPO}} + 0.01\,\mathcal{L}_{\mathrm{recon}} + 1.0\,\mathcal{L}_{\mathrm{cycle}} + 0.01\,\lVert \hat{\bm{s}}_{t+1} - \bm{s}_{t+1} \rVert^2 - 0.005\,\log \mathcal{G}_t(\bm{z}_{t+1})$$

Automatic Spatial Terrain Annotation

Figure 2: From motion to spatial terrain. (a) Kinematic replay. (b) Contact detection. (c) 3D point cloud. (d) Penetration-filtered clusters. (e) Fitted 3D terrain primitives.

The pipeline samples points on the collision geometry of contact-relevant links (ankles by default, optionally hands, knees, elbows, pelvis, torso back), and replays the retargeted trajectory kinematically in MuJoCo. After Savitzky–Golay smoothing, a per-sample state machine marks contact when normal speed falls below 0.2 m/s and tangential speed below 0.3 m/s, with at least 2 of the preceding 10 frames showing normal deceleration above 1 m/s². This deceleration signature distinguishes support events from momentarily slow limbs.

Detected contacts become geometry through: discarding points below 10 cm or with downward normals; deduplication on a 1 mm voxel grid; a whole-robot penetration filter that keeps a point only if a support offset 5 cm outward never intersects any body at any frame; planar grouping via normal agreement (40°) and offset (10 cm); DBSCAN clustering; and fitting each cluster with a 1 cm-thick oriented box.

Filtering Out-of-Distribution Commands

At deployment, the previous step's mixture $\mathcal{G}_{t-1}$ predicts which commands are consistent with training experience. The in-distribution region per Gaussian component is defined via the squared Mahalanobis radius:

$$M^2_k(\bm{z}) = \sum_{i=1}^{D} \left(\frac{z_i - \mu_{k,i}}{\sigma_{k,i}}\right)^2, \quad D=64$$

where $\sigma_{k,i} = \exp(\log \sigma_{k,i})$ with $\log \sigma_{k,i}$ clamped to $[-3, 10]$. The most plausible mode is selected by a MAP-style rule:

$$k^{\star} = \arg\max_k \left[\log(\pi_k + 10^{-10}) - \frac{1}{2D} \sum_{i=1}^{D} \left[\left(\frac{z_i - \mu_{k,i}}{\sigma_{k,i}}\right)^2 + 2\log \sigma_{k,i} + \log 2\pi\right]\right]$$

When a command is out-of-distribution, it is retracted radially toward the mode center onto the safety ellipsoid boundary:

$$\bm{z}^{\star}_t = \bm{\mu}_{k^{\star}} + \sqrt{R^2_{\mathrm{safe}} / M^2_{k^{\star}}(\bm{z}^{\mathrm{raw}}_t)} \; (\bm{z}^{\mathrm{raw}}_t - \bm{\mu}_{k^{\star}})$$

This is a closed-form rescaling, not the Euclidean-nearest point on the ellipsoid (which would require solving a Lagrange multiplier every frame). The retracted command still points from the trusted mode toward what the operator asked for, so the robot keeps attempting the task. The filter is memoryless: O(1) per step, well under 1 ms against a 20 ms control period.

graph TD
    A["Reference window c_t
440-dim"] -->|MLP encoder E| B["Latent command z_raw
64-dim"]
    B -->|Previous mixture G_{t-1}| C{"Mahalanobis
M² > R²?"}
    C -->|Yes: OOD| D["Radial retraction
to safety ellipsoid"]
    C -->|No: ID| E["Pass through z_t = z_raw"]
    D --> E
    E --> F["Causal Transformer
6 layers, 4 heads"]
    F --> G["Action a_t
29-dim"]
    F --> H["State prediction s_{t+1}
67-dim"]
    F --> I["Next mixture G_t
4 GMM components, 516-dim"]
    I -.->|Next step| C

Training

Three large retargeted human-motion corpora are used: Bones-Seed (288 h), MotionMillion (900 h), and MotionDecode (1000 h). Within them, 12.50 h, 22.22 h, and 37.85 h respectively are identified as containing spatial terrain interaction. Terrain and flat-ground motions mix at a controlled ratio of 0.2. PPO optimization in Isaac Lab uses sequence-level updates with per-layer KV prefix caching. Rollouts use 4096 parallel environments on flat ground, reduced to 512 once terrain geometry and fallen-state initialization are enabled.

Two robustness ingredients: a fraction of flat-ground episodes start from synthetic fallen poses under a trunk inclination curriculum, with tracking rewards gated by a smooth recovery gate; persistent random external forces at wrists and torso emulate payloads and contact loads.

Experimental Results

Table 1: MuJoCo sim-to-sim comparison. MPKPE in mm, Jerk in rad/s³, SR in %.

On Standard flat-ground tracking, the behavior world model costs nothing in free space: 76.6 mm MPKPE and 96.3% SR is the most accurate and reliable of the four, ahead of SONIC (82.3 mm, 94.1%). The one metric not led is root linear velocity (211.1 vs HoloMotion-1's 121.3 mm/s), attributed to intermittent root-translation masking—a policy running without absolute root feedback two-thirds of the time re-anchors in small corrections.

The Terrain split is where environment-aware training separates. All three baselines collapse to 14–19% SR with MPKPE between 283–331 mm—having never experienced non-floor contact, they treat a step or seat as a disturbance to reject. GigaBrain-WBC-0.5 holds 81.3% SR at 93.3 mm, a 4.3× improvement in survival and 3.0× reduction in tracking error. Internally: moving from flat ground to terrain raises our MPKPE by 22%, while it raises the baselines' by 3–4×.

Figure 3: Hardware environment interaction. (a)-(d) Matched comparisons with SONIC under the same live commands. (e) Carrying a large case while walking onto a platform and down to the floor.

On OOD, where every reference is physically impossible, GigaBrain-WBC-0.5 survives 83.1% of clips against 50.0–70.6% for baselines, and its executed motion stays closest to the intent (158.0 mm vs 208.0 mm). On Fall, the gap is categorical: baselines recover from 0.7–5.9%, ours from 99.3%.

Figure 4: Robustness versus precision as a function of the safety radius $R_{\mathrm{safe}}$.

Table 2: Human audit of automatic terrain annotation.

Cross-Embodiment Transfer

Transferring the G1 checkpoint to the Maker L01 humanoid reuses the same recipe: retargeting the corpus to L01 and fine-tuning the G1 checkpoint on a single 8-GPU node recovers whole-body tracking quickly, while training the same architecture on L01 from scratch converges slowly. The behavior world model appears to carry structure about how a humanoid interacts with its environment that is not specific to one set of link lengths.

Figure 5: Whole-body motion capability—agile motion, household tasks, and industrial tasks, all from one tracking policy.

Limitations

Limitation 1 (author-stated): The filter is validated in simulation and its radius must be re-calibrated per checkpoint and platform before being relied upon on hardware. It flags commands unlike the policy's training experience rather than reasoning about physical risk directly, so interception of infeasible commands is high-probability rather than certain.

Limitation 2 (author-stated): The terrain annotation is derived from contact evidence and therefore recovers only the geometry a motion actually touches—surfaces present but unused are not reconstructed, and scenes are supports rather than complete environments.

Limitation 3 (analytical): Root linear velocity tracking on Standard flat ground does not lead (211.1 vs 121.3 mm/s), a direct cost of intermittent root-translation masking. While this does not affect keypoint accuracy, it may limit applications requiring precise velocity matching, such as dynamic tasks synchronized with external forces.

Conclusion and Outlook

GigaBrain-WBC-0.5 demonstrates that the two capabilities missing from current whole-body trackers—interacting with the environment and remaining well-behaved when asked to do something the environment does not permit—are the same capability seen from two sides, both following from making the controller predict its own future rather than only its next action. Around this premise, the authors built an automatic pipeline recovering spatial contact geometry from ordinary retargeted motion, and a stateless closed-form projection turning the model's own predicted distribution into an online OOD filter governed by one scalar radius. Achieving the highest success rate across all four regimes among three large-scale baselines—81.3% terrain interaction (4.3× the strongest baseline) and 99.3% fall recovery (16.8×)—the G1 checkpoint transfers to the Maker L01 robot through fine-tuning.

As the first behavior world model for humanoid whole-body control, this work opens a direction toward controllers combining environment interaction with robust execution. Connecting the world model's state prediction to a more direct notion of physical risk, extending terrain reconstruction to non-supporting geometry, and larger-scale real-robot validation are natural next steps.