WholeBodyWAM: Generalizing Pre-trained World-Action Priors to Humanoid Loco-Manipulation via WBC-Grounded Coordination

Paper Metadata

One-Sentence Summary

Instead of relearning whole-body behavior, WholeBodyWAM keeps the manipulation priors of a pre-trained 14B world-action model (WAM), grounds heterogeneous whole-body controller (WBC) commands through a unified semantic interface (UWBC), and gates manipulation-to-WBC attention with task-directional arm manipulability (CASA), turning 15K teleoperated demonstrations into a generalizable humanoid loco-manipulation policy.

Figure 0: Overview (paper Fig. 1). Unlike expanding action interfaces or learning humanoid world-action mappings from scratch, WholeBodyWAM preserves and generalizes pre-trained priors through WBC-grounded coordination.

Background and Motivation

World Action Models jointly model robot actions and future visual dynamics inside a single generative model, providing reusable priors for manipulation. DreamZero, Cosmos Policy and Fast-WAM validated this recipe on tabletop and arm-centric settings, where the action interface is a function of local hand-object interaction. Extending it to bipedal humanoids changes the problem qualitatively.

The paper identifies three coupled obstacles. Whole-body coordination complexity: success requires not only plausible hand-object interaction but also knowing when and how the base, torso and waist should move to support the arms. Heterogeneous WBC semantics: controllers such as SONIC, AMO, GEAR and HugWBC expose command ontologies with different physical meanings, so the same body motion demands entirely different combinations of task-space commands and joint targets, blocking reusable action semantics. Humanoid data scarcity: large-scale robot data can pre-train a WAM, but paired humanoid loco-manipulation demonstrations remain scarce, leaving little direct supervision for the coupling between manipulation intent and whole-body adjustment.

Prior work probes two routes. Direct action-space expansion resizes the action head of a pre-trained tabletop WAM and fine-tunes on whole-body demonstrations, as DiT4DiT did for humanoid bimanual manipulation; it preserves transferable priors but leaves WBC semantics and coordination to implicit learning from limited supervision, often yielding well-formed manipulation intent with poorly grounded controller commands and temporally inconsistent body motion. Monolithic whole-body modeling, as in MotionWAM and omega-0, learns humanoid-native WAMs in controller-compatible latents; it produces coherent whole-body motion but entangles policy prediction with controller-specific conventions and depends on costly large-scale whole-body supervision.

Both routes miss the same question: what does the pre-trained prior already contain, and what is actually missing for humanoids? The paper poses it directly: how can pre-trained world-action priors be generalized to humanoid loco-manipulation without learning whole-body behavior from scratch?

The authors' insight is that pre-trained WAMs already capture rich manipulation priors; what is fundamentally missing is structured grounding to heterogeneous WBCs that regulates when and how whole-body behavior should coordinate. WholeBodyWAM operationalizes this with three complementary designs: a structured action representation that retains the pre-trained manipulation pathway while adding a distinct whole-body control stream, a Unified Whole-Body Controller interface (UWBC) that assigns consistent physical meanings to shared commands while accommodating controller-specific extensions, and Coordination-Aware Self-Attention (CASA), which modulates an attention bias with task-directional arm manipulability to strengthen manipulation-to-UWBC flow exactly when more coordination is indicated.

Preliminaries

A WAM backbone is a video Diffusion Transformer (DiT): observations are compressed into latent video tokens by a video VAE, future frames and actions are organized as a token sequence, and the DiT denoises them jointly under a flow-matching objective, where generation is velocity-field regression from a noisy block to the data block and inference integrates the field with a few-step sampler. WholeBodyWAM inherits DreamZero's pre-trained 14B video DiT and this joint flow-matching formulation.

On the control side, SONIC favors joint-space references and motion latents, AMO consumes planar velocity, heading, absolute height and torso orientation plus a turning flag, GEAR WBC uses velocity, height and orientation with a derived yaw rate, and HugWBC-style controllers offer general tracking interfaces. Dimensions, units and semantics differ across them: one intent such as "step forward and bend" is a sequence of lower-body joint positions in SONIC but a handful of task-space scalars in AMO. This heterogeneity is the main obstacle to reusing policies across controllers.

Manipulability quantifies how well a joint configuration can produce end-effector velocity along a direction. The damped manipulability matrix with joint speed limits, $C^{v,\lambda}$, turns this into a direction-dependent ellipsoid; following prior work the paper builds a voxelized manipulability map offline so that at runtime the maximum radial capability along the estimated task direction within the current end-effector pose voxel is a cheap lookup, providing the state signal for the coordination gate.

Method

Problem formulation. WholeBodyWAM formulates humanoid loco-manipulation as joint prediction under a task context $\mathbf{c}_{t}$ consisting of visual history $\mathbf{o}_{0:t}$, current robot state $\mathbf{s}_{t}$ and language instruction $\ell$: predict future visual dynamics $\mathbf{v}_{t:t+H}$, a manipulation stream $\mathbf{m}_{t:t+H}$ of arm joint targets and finger joint angles, and a whole-body stream $\mathbf{u}_{t:t+H}$ of UWBC commands over horizon $H$:

$$ p_{\theta}(\mathbf{v}_{t:t+H},\mathbf{m}_{t:t+H},\mathbf{u}_{t:t+H}\mid\mathbf{c}_{t}). $$

Following joint world-action modeling, the conditional distribution factorizes into a world factor and an action factor:

$$ p_{\theta}(\mathbf{v},\mathbf{m},\mathbf{u}\mid\mathbf{c}_{t}) = p_{\theta}^{v}(\mathbf{v}\mid\mathbf{c}_{t})\,p_{\theta}^{a}(\mathbf{m},\mathbf{u}\mid\mathbf{v},\mathbf{c}_{t}). $$

The authors stress that this factorization describes the dependency structure of the joint distribution, not a sequential video-then-action inference pipeline: the three streams are instantiated, optimized and sampled jointly inside one shared generative backbone.

Shared backbone and typed token streams. The model builds on the pre-trained 14B video DiT. A pre-trained T5 encoder injects instruction features through cross-attention; the pre-trained Wan video VAE compresses observations into latent video tokens; a lightweight state encoder maps the current robot state to proprioceptive conditioning tokens. Future visual tokens, manipulation tokens and UWBC tokens form three typed streams that share a prediction block at each future step and are processed jointly by the DiT. The manipulation stream retains the visual-manipulation pathway inherited from the pre-trained WAM, while distinct UWBC tokens make controller-facing whole-body behavior explicitly addressable for the first time.

Training objective. Following the joint flow-matching formulation, WholeBodyWAM minimizes a modality-weighted velocity regression loss:

$$ \mathcal{L}(\theta)=\mathbb{E}\!\left[\sum_{k}\lambda_{k}w_{k}(\tau)\left\|f_{\theta}^{k}(\mathbf{x}_{\tau},\tau,\mathbf{c}_{t})-\dot{\mathbf{x}}^{k,*}\right\|_{k}^{2}\right], $$

where $\mathbf{x}_{\tau}$ is the noisy joint block at flow time $\tau\in[0,1]$, $\dot{\mathbf{x}}^{k,*}$ is the target flow velocity for stream $k\in\{v,m,u\}$, $\lambda_{k}$ balances stream losses and $w_{k}(\tau)$ varies with flow time. The objective itself is not new; what is new is the token structure it jointly optimizes.

Unified WBC interface. Naively concatenating heterogeneous commands into a flat action vector obscures control semantics and welds the policy to one controller interface. UWBC organizes commands by shared physical semantics: task-space whole-body commands plus lower-body and waist joint positions and velocities form a 46-D shared block, and ten residual slots at indices $[46,56)$ hold controller-specific commands including discrete control modes:

$$ \mathbf{u}_{t}=[\mathbf{u}_{t}^{\mathrm{sh}},\mathbf{u}_{t}^{r}]\in\mathbb{R}^{56},\quad \mathbf{u}_{t}^{\mathrm{sh}}=[\mathbf{u}_{t}^{\mathrm{task}},\mathbf{q}_{t}^{\mathrm{lower}},\dot{\mathbf{q}}_{t}^{\mathrm{lower}}]\in\mathbb{R}^{46}. $$

Here $\mathbf{u}_{t}^{\mathrm{task}}\in\mathbb{R}^{16}$ carries task-space commands and $\mathbf{q}_{t}^{\mathrm{lower}},\dot{\mathbf{q}}_{t}^{\mathrm{lower}}\in\mathbb{R}^{15}$ specify desired lower-body and waist joint positions and velocities. The shared slots are registered with explicit semantics: $[0,3)$ base linear velocity, $[3,6)$ base angular velocity, $[6,8)$ heading as $(\sin\psi,\cos\psi)$, $[8,10)$ pelvis planar position, $[10,11)$ pelvis height, $[11,13)$ pelvis roll/pitch, $[13,16)$ torso-to-pelvis orientation, and $[16,31)$ / $[31,46)$ lower-body and waist joint positions and velocities. For a downstream controller, fields are activated according to their registered semantics and the controller capability profile, while unsupported fields are masked: SONIC uses joint-space slots $[16,46)$; AMO maps planar velocity, heading, absolute height and torso orientation to $[0,2)$, $[6,8)$, $10$ and $[13,16)$ with its turning flag in a residual slot; GEAR WBC reuses the velocity, height and orientation slots and carries its derived yaw rate in slot $5$.

Figure 1: Model architecture (paper Fig. 2). A shared DiT jointly generates future visual dynamics, manipulation actions and UWBC commands; CASA adaptively strengthens manipulation-to-UWBC attention as task-directional manipulability drops.

Coordination-Aware Self-Attention. Coordination in humanoid loco-manipulation is directional and state dependent: when arm manipulability along the task direction becomes insufficient, the body must compensate. CASA writes this physical intuition into the attention logits as a gated bias:

$$ \mathbf{L}^{\prime}_{t}=\frac{\mathbf{Q}\mathbf{K}^{\top}}{\sqrt{d}}+\mathbf{M}^{\mathrm{nat}}+\beta g_{t}\mathbf{S}^{M\rightarrow U}, $$

where $\mathbf{M}^{\mathrm{nat}}$ is the native self-attention mask, $\beta>0$ a fixed bias strength and the gate $g_{t}\in[0,1]$ decides when it opens; the selector $S_{ij}^{M\rightarrow U}=1$ only when $i$ is a UWBC query and $j$ a manipulation key within the same prediction block. Native visibility across the three streams is fully preserved; the bias only strengthens manipulation-to-UWBC flow.

Figure 2: Blockwise-causal attention with gated coordination (paper Fig. 3). Within each prediction block, UWBC queries attend more strongly to manipulation keys when the gate opens.

The gate signal is the relative reduction of task-directional manipulability. For each arm $a\in\{L,R\}$ the task direction is estimated by differencing consecutive observed end-effector positions:

$$ \mathbf{v}_{a,t}=\frac{\mathbf{p}_{a,t}-\mathbf{p}_{a,t-1}}{\Delta t},\qquad \hat{\mathbf{d}}_{a,t}=\frac{\mathbf{v}_{a,t}}{\|\mathbf{v}_{a,t}\|_{2}}. $$

The directional manipulability of the observed arm configuration is then:

$$ c_{a,t}=\left[\hat{\mathbf{d}}_{a,t}^{\top}\bigl(C_{a}^{v,\lambda}(\mathbf{q}^{\mathrm{obs}}_{a,t})\bigr)^{-1}\hat{\mathbf{d}}_{a,t}\right]^{-1/2}. $$

At runtime the reference $c^{\mathrm{ref}}_{a,t}$ is the maximum sampled radial capability along the estimated task direction among configurations in the current end-effector pose voxel of the offline voxelized manipulability map. The per-arm gate follows from the relative reduction and the global gate takes the larger of the two arms:

$$ g_{a,t}=1-\operatorname{clip}\!\left(\frac{c_{a,t}}{c^{\mathrm{ref}}_{a,t}+\epsilon_{m}},0,1\right),\qquad g_{t}^{M\rightarrow U}=\max(g_{L,t},g_{R,t}). $$

Thus "either active arm approaching its capability boundary along the task direction" is sufficient to trigger strong conditioning of whole-body commands on manipulation intent: the moment the arms cannot reach is exactly the moment the base and torso should move.

Training recipe. The model is initialized from the pre-trained 14B WAM and retains its world-action backbone. The authors collect 15K whole-body loco-manipulation demonstrations in SIMPLE with a PICO 4 Ultra headset and wrist trackers, convert each demonstration into a SMPL motion sequence, retarget it to the G1 embodiment with GMR to obtain robot motion references, and then derive supervision for every UWBC shared slot from those references according to its predefined physical semantics, coordinate convention and unit. After temporal alignment the UWBC targets join the post-training dataset. Post-training uses LoRA with rank $r=4$ and scaling $\alpha=4$, optimized for 50,000 steps at learning rate $10^{-5}$ with batch size 1 per GPU; inference uses 16 denoising steps with a second-order UniPC flow-matching sampler and a fixed noise-schedule shift of 5. The recipe is deliberately light: the trainable increment is LoRA plus the new token pathways, leaving the pre-trained prior almost untouched.

flowchart TD
  L["Language instruction"] --> T5["T5 text encoder"]
  O["Visual history o_0:t"] --> VAE["Wan video VAE"]
  S["Robot state s_t"] --> SE["Lightweight state encoder"]
  T5 --> DIT["Shared 14B pre-trained video DiT"]
  VAE --> DIT
  SE --> DIT
  DIT --> VS["Future visual stream v"]
  DIT --> MS["Manipulation stream m
arm + finger joint targets"]
  DIT --> US["UWBC stream u
46 shared + 10 residual slots"]
  P["Observed EE positions"] --> DIR["Task direction Eq.6"]
  DIR --> MAN["Directional manipulability Eq.7"]
  MAN --> GATE["Gate g_t = max of both arms Eq.8 Eq.9"]
  GATE --> CASA["CASA bias on M-to-U attention Eq.5"]
  MS --> CASA
  CASA --> DIT
  US --> SONIC["SONIC: joint slots 16-46"]
  US --> AMO["AMO: vel heading height torso + turn flag"]
  US --> GEAR["GEAR WBC: vel height orientation + yaw slot 5"]

Figure 3: Information flow drawn from the paper's method. The CASA gate is computed online from observed end-effector motion and the offline manipulability map, and only modulates the attention bias without changing the joint generation of the three streams.

Experiments

Setup and metrics. The real platform is a Unitree G1 with two BrainCo Revo 2 dexterous hands, observed by one head-mounted and two wrist-mounted RealSense D435i cameras; simulation runs on the SIMPLE benchmark. Three metrics are used: Task Success Rate (TSR), the fraction of trials completing all objectives within the horizon; Task Progress (TP), normalized progress along ordered task stages before the first failure or irreversible deviation; and cross-WBC variance, the population variance of mean TSR across controllers in pp², where lower is more robust. Every method-task combination uses 20 independent trials. Baselines span WAMs and VLAs: DreamZero, DreamZero-PT post-trained on the same data, Cosmos-3, the native whole-body VLA Psi-0, and the humanoid VLA GR00T N1.6.

Simulation main results. With the SONIC controller over six tasks and three perturbation levels (L0/L1/L2), WholeBodyWAM reaches an overall TSR of 91.9%, exceeding DreamZero, DreamZero-PT and Cosmos-3 by 26.9, 18.3 and 5.6 percentage points and beating the strongest VLA baseline Psi-0 (82.2%) by 9.7 points. The 18.3-point margin over DreamZero-PT, which sees the same post-training data, shows the gain is not merely "more humanoid supervision". Against Cosmos-3 the advantage concentrates on coordination-intensive PickBetweenTables and MoveBendPick (+10.0 and +11.7 points), while the two are comparable on manipulation-dominant Handover and TabletopGrasp: the division of labor between prior preservation and coordination grounding is visible in the numbers.

Table 1: Simulation task success rate (%) with SONIC; each cell reports L0/L1/L2 under the SIMPLE protocol (paper Table II).

Cross-controller robustness. Fine-tuning separately for SONIC, AMO and GEAR WBC, WholeBodyWAM attains the highest mean TSR across the three controllers, 89.2%, with the smallest variance, 10.5 pp²; Cosmos-3 scores 80.2% and 35.0 pp², Psi-0 75.2%/37.0, DreamZero-PT 65.7%/39.0, DreamZero 56.7%/43.0 and GR00T N1.6 39.8%/36.0. The roughly 70% variance reduction matters because models trained with a single WBC must relearn command conventions when the controller changes, whereas UWBC separates internal whole-body intent from controller-specific conventions so the same grounding prior maps onto different interfaces by semantics.

Table 2: Cross-WBC evaluation (data from paper Fig. 4). Higher mean and lower variance indicate less sensitivity to controller choice.

Figure 4: Cross-WBC evaluation (paper Fig. 4). WholeBodyWAM sits in the high-mean, low-variance corner while other methods spread along the variance axis.

Downstream fine-tuning efficiency. Under budgets of 50/100/200/300 demonstrations per task (100 is the main simulation budget), WholeBodyWAM's advantage is largest at the low-data end: with 50 demonstrations baselines collapse into the 20%-60% TSR range on several tasks while the proposed method keeps task-average success above roughly 60%-90%. This matches the prior-reuse narrative: fine-tuning only has to teach the model to translate existing intent into UWBC commands, not to relearn manipulation.

Figure 5: Downstream fine-tuning efficiency (paper Fig. 5). The gap widens as demonstrations shrink.

Real-world results. On eight real tasks, WholeBodyWAM reaches 81.3% mean TSR in-distribution versus 57.5% for DreamZero, and 68.8% versus 40.0% out-of-distribution where object configurations and language instructions change, with a smaller ID-to-OOD drop (12.5 versus 17.5 points). Task-level gaps again follow coordination intensity: on manipulation-dominant TableCleanup the margin is only 5 points (90 versus 85), while on BasketCarry and DoorEntry, which require substantial body reconfiguration, it reaches 35 points, 40 on BoxTransfer, and 45 on OOD DoorEntry. Task progress tells the same story: mean ID/OOD TP of 0.92/0.82 versus 0.73/0.59 for DreamZero, and the 0.23 OOD progress gain is exactly the number quoted in the abstract.

Table 3: Real-world task success rate (%) under in-distribution and out-of-distribution conditions (paper Table III).

Figure 6: Qualitative results on the eight real-world tasks (paper Fig. 6), spanning manipulation-dominant and coordination-intensive behaviors.

Emergent whole-body coordination. The most striking case is OOD TowelPlace: displacing the towel support 10 cm farther puts the towel beyond the initial arm workspace; after one unsuccessful local reaching attempt the robot takes a short forward step and adjusts its torso to extend reach, succeeding on the second attempt. The authors state this recovery sequence is absent from their demonstrations and interpret it as emergent behavior arising from pre-trained WAM manipulation priors coupled with enhanced whole-body coordination. A single qualitative case is not a capability proof, but it does show the coordination gate does not freeze the body under distribution shift.

Figure 7: Grasp failure and body-adjustment recovery under a 10 cm displacement (paper Fig. 8).

Ablations. The three components separate cleanly across six simulation tasks. Removing CASA (keeping both streams, restoring native self-attention) drops overall TSR from 91.9% to 86.9%, with the reduction concentrated on PickBetweenTables (-8.3) and MoveBendPick (-6.7) versus only -1.7 on Handover, consistent with state-dependent directional coordination mattering mainly on tasks that demand large body adjustment. Replacing UWBC with native SONIC commands while keeping the distinct WBC pathway and supervision drops TSR to 84.7%, indicating that explicit physical semantics improve controller grounding even when a separate pathway exists. Removing the structured action factorization (SAF), i.e. replacing the two streams with a capacity-matched monolithic stream under the same data and optimization budget, causes the largest drop to 80.8%, supporting the choice to preserve the pre-trained manipulation pathway and keep WBC commands separately addressable.

Limitations

First, the authors themselves note the gains are unevenly distributed: on manipulation-dominant tasks the margin over DreamZero is small (TableCleanup 90% versus 85%), because pre-trained manipulation priors already suffice when targets lie within the arm workspace and require little base or torso adjustment. UWBC and CASA address coordination, not pure manipulation accuracy.

Second, cross-controller portability remains conditional: the Fig. 4 protocol fine-tunes separately for each target WBC, so UWBC provides a shared semantic interface and reusable grounding prior rather than a zero-shot controller adapter; switching controllers still requires re-fine-tuning on the new interface.

Third, on the experimental design, the real-robot study compares against a single baseline (DreamZero) with 20 trials per method-task combination, while stronger baselines such as Psi-0, GR00T N1.6 and Cosmos-3 appear only in simulation, limiting how far the physical results generalize.

Fourth, the paper reports no end-to-end inference latency or control frequency: with a 14B backbone and 16 denoising steps, the closed-loop rate and visual history window on the real robot are not stated, so real-time behavior can only be inferred from project-page videos. In addition, the CASA gate depends on differencing observed end-effector positions and on the offline voxelized manipulability map, making it sensitive to observation noise, calibration error and voxel granularity; the emergent recovery rests on a single qualitative case.

Conclusion and Outlook

WholeBodyWAM's contribution is less a single module than a clean reframing: the bottleneck for humanoid loco-manipulation is not a lack of manipulation priors but the lack of structured grounding from those priors into heterogeneous whole-body controllers. UWBC decouples intent from command conventions through shared physical semantics, CASA makes "when to coordinate" explicit through a manipulability gate, and the structured action factorization keeps the pre-trained pathway unpolluted by the new stream. Together, 15K demonstrations and LoRA-scale post-training push a 14B tabletop WAM to 91.9% over six simulation tasks and 81.3% ID / 68.8% OOD over eight real tasks.

The route forward is equally clear: whether UWBC residual slots can absorb more controller families including force control and gait modes, whether the CASA gate can extend from manipulability to contact and balance constraints, and whether prior reuse still beats learning from scratch at larger humanoid data scale. If code and data ship with the final version, UWBC as a cross-controller semantic layer will prove more durable than the model itself.

Golden Quote

"These results suggest a path toward scalable humanoid whole-body intelligence by extending pre-trained world-action priors through structured WBC grounding and coordination, rather than relearning whole-body behavior from scratch." (abstract)