PAPER DEEP DIVE
CoorDex: Coordinating Body and Hand Priors for Continuous Dexterous Humanoid Loco-Manipulation
Humanoid loco-manipulation is often simplified into a stop-and-go process: walking to an object, stopping to manipulate it, and then resuming locomotion. It also commonly relies on low degree-of-freedom (DoF) end effectors that behave like an open-close grasp primitive. We introduce CoorDex, a learning pipeline that converts high-dimensional body and dexterous hand control into coordinated latent residual control, enabling high-DoF dexterous loco-manipulation on the move. Starting from simulated whole-body and hand demonstrations, CoorDex trains privileged motion tracking teachers for the humanoid body and dexterous hand, distills them into proprioception-conditioned latent priors, and uses the frozen priors as the action space for downstream residual reinforcement learning. A coordinated latent residual policy composes these priors through shared task context and separate body-hand residual heads, preserving natural whole-body motion while improving finger-level contact reliability. CoorDex enables a Unitree G1 humanoid with a 20-DoF WUJI hand to execute dexterous manipulation while in motion, including non-stop bottle grasping and carrying, fridge door opening on the move, and cube pick-and-turn. Ablations on the walk-grasp-carry task show that joint-space PPO, joint-space hand control, and monolithic latent prediction all fail under the same reward budget, while the latent-prior interface and coordinated residual structure make high-dimensional contact-rich loco-manipulation trainable. Project Page: https://skevinci.github.io/coordex/
1. Overview: Coordinating Body and Hand Priors for Continuous Dexterous Loco-Manipulation
CoorDex presents a modular learning pipeline that maps high-DoF humanoid locomotion and dexterous hand control to coordinated latent residual control. The core motivation: existing humanoid loco-manipulation is often simplified into a stop-and-go process — walking to an object, then stopping to manipulate. CoorDex separates whole-body wrist placement from finger coordination, modeling them with a body prior and a wrist-stabilized hand prior, then couples them through a coordinated latent residual policy to enable continuous grasping and manipulation while walking. Validated in Isaac Lab on a Unitree G1 (29-DoF body + 20-DoF WUJI dexterous hand) across three tasks: WalkGrab (grasp bottle while walking), OpenFridge (open door while stepping back), WalkPickTurn (pick cube and turn 180°).
2. Core Problem: Dexterous Manipulation on the Move
Humanoid robots with high-DoF dexterous hands promise a single embodiment that can both navigate human environments and manipulate everyday objects. But realizing this promise requires more than placing an end effector near a target — the robot must maintain balance, keep the object reachable, coordinate wrist and finger poses before contact, close the hand without disturbing the object, and continue transporting while hand-object interactions perturb whole-body dynamics. Existing hand-centric methods assume wrist/arm trajectories are provided by teleoperation, planning, or a fixed-base setup, and cannot handle dynamic scenarios where wrist placement emerges from whole-body motion. CoorDex's key insight: non-stop dexterous manipulation requires separated but coordinated modeling of body and hand priors — the body prior handles locomotion and wrist placement, the hand prior handles finger coordination, and downstream residual RL jointly adapts both in a low-dimensional latent space.
3. Method: Coordinated Latent Residual Control
3.1 Prior Construction
Two separate motion priors are constructed: a body prior and a hand prior, denoted $x \in \{b, h\}$. The body prior trains a whole-body motion tracking teacher $\pi_T^b$ taking body proprioception $\mathbf{s}_t^{b,p}$ and reference goal $\mathbf{s}_t^{b,g}$, outputting joint position targets. The hand prior trains a privileged tracking teacher $\pi_T^h$ in a floating-hand environment using ManipTrans-style hand-object motions. After distillation, an encoder $\mathcal{E}_x$, proprioceptive prior $\mathcal{R}_x$, and decoder $D_x$ are obtained. The distillation loss:
$$\mathcal{L}_{\text{distill}}^{x} = \mathcal{L}_{\text{action}}^{x} + \alpha_x \mathcal{L}_{\text{regu}}^{x} + \beta_x \mathcal{L}_{\text{KL}}^{x} \tag{1}$$After distillation, $\mathcal{R}_x$ and $D_x$ are frozen; the prior mean $\boldsymbol{\mu}_t^{x,p}$ provides the default latent command for downstream residual RL. The body subsystem has 29-DoF with latent dimension 16; the hand has 20-DoF with latent dimension 12.
3.2 Coordinated Latent Residual Policy
At each step, the frozen priors produce proprioception-conditioned prior means:
$$\boldsymbol{\mu}_t^{b,p} = \text{Mean}[\mathcal{R}_b(\mathbf{z}_t^b \mid \mathbf{s}_t^{b,p})], \quad \boldsymbol{\mu}_t^{h,p} = \text{Mean}[\mathcal{R}_h(\mathbf{z}_t^h \mid \mathbf{s}_t^{h,p})] \tag{2}$$The residual policy predicts residuals in latent space, not joint space:
$$\Delta\mathbf{z}_t = [\Delta\mathbf{z}_t^b, \Delta\mathbf{z}_t^h], \quad \Delta\mathbf{z}_t^b \in \mathbb{R}^{d_b}, \quad \Delta\mathbf{z}_t^h \in \mathbb{R}^{d_h} \tag{3}$$A shared coordination trunk $f_{\text{coord}}$ produces a task-level coordination feature, then body head $f_b$ and hand head $f_h$ predict residuals:
$$\mathbf{c}_t = f_{\text{coord}}(\mathbf{s}_t^{b,p}, \mathbf{s}_t^{h,p}, \mathbf{s}_t^{\text{task}}, \mathbf{s}_t^{\text{hand-object}}, \boldsymbol{\mu}_t^{b,p}, \boldsymbol{\mu}_t^{h,p}, \Delta\mathbf{z}_{t-1}) \tag{4}$$The corrected latent commands and final joint targets:
$$\tilde{\mathbf{z}}_t^b = \boldsymbol{\mu}_t^{b,p} + \Delta\mathbf{z}_t^b, \quad \tilde{\mathbf{z}}_t^h = \boldsymbol{\mu}_t^{h,p} + \Delta\mathbf{z}_t^h \tag{5}$$ $$\mathbf{a}_t^b = D_b(\mathbf{s}_t^{b,p}, \tilde{\mathbf{z}}_t^b), \quad \mathbf{a}_t^h = D_h(\mathbf{s}_t^{h,p}, \tilde{\mathbf{z}}_t^h) \tag{6}$$
3.3 Residual RL and Environment Design
Downstream policies are trained with PPO while keeping body and hand priors frozen. The policy observation contains body proprioception, hand proprioception, task state, object-relative geometry, fingertip contact features, prior means, and the previous latent residual. Since the actor only outputs $\Delta\mathbf{z}_t$, exploration happens in the learned latent space rather than full joint space. WalkPickTurn uses NoDemoRSI: state snapshots discovered by the policy itself serve as reset points for later stages, without expert states or action labels. Per-stage sampling probabilities adapt by difficulty $\max(1-\text{SR}_k, \epsilon)^p$ with $\epsilon=0.1$; stage unlocking requires the preceding stage's success rate exceeding 0.70 and the buffer satisfying minimum snapshot count (128).
4. Experiments
4.1 Task Performance
Three tasks validate that the unified interface supports diverse loco-manipulation skills. WalkGrab is hardest (continuous grasping while walking), OpenFridge allows deceleration and repositioning, WalkPickTurn is longer-horizon with NoDemoRSI support.
| Task | Success | Fall | Drop | Task Metric |
|---|---|---|---|---|
| WalkGrab | 0.55 | 0.00 | 0.40 | Velocity ~0.25 m/s |
| OpenFridge | 0.66 | 0.00 | — | Door angle 57.76°/60° |
| WalkPickTurn | 0.89 | 0.01 | 0.10 | Min heading error 9.98° |
4.2 Action-Space Ablation
Three variants compared on WalkGrab. All Joint Space removes both priors and directly explores full body+hand joint space — never grasps the bottle. Body Prior + Hand Joint Space isolates hand difficulty — reaches the bottle but fails at finger coordination, often slowing to a stop for stationary grasping. CoorDex maintains ~0.25 m/s forward velocity, grasping without stopping.
| Method | Success | Reach | Grasp | Stop | Fall |
|---|---|---|---|---|---|
| All Joint Space | 0.00 | 1.00 | 0.00 | 0.86 | 0.04 |
| Body Prior + Hand Joint Space | 0.00 | 0.96 | 0.01 | 0.90 | 0.04 |
| CoorDex | 0.55 | 1.00 | 0.55 | 0.00 | 0.00 |
4.3 Coordinated Residual Prediction
Compares Monolithic Latent Residual (single MLP predicting full latent residual, then split) vs CoorDex (shared trunk + separate heads). Both use the same frozen priors, latent dimensions, RL environment, and reward — only the policy architecture differs. Monolithic action rate 0.40 (higher = jerkier), CoorDex action rate 0.22.
| Method | Success | Action Rate | Fall |
|---|---|---|---|
| Monolithic Latent Residual | 0.00 | 0.40 | 0.02 |
| CoorDex | 0.55 | 0.22 | 0.00 |
5. Limitations
- Privileged state observations: Current policies use privileged state observations (object poses, contact signals) and do not yet address perception or visual sim-to-real transfer.
- Fixed platform: Experiments focus on a fixed G1+WUJI hand platform; broader deployment requires evaluation across more body and hand morphologies.
- High drop rate: WalkGrab's 0.40 drop rate indicates room for improvement in grasping stability during locomotion; hand-object interaction perturbing whole-body dynamics is not fully solved.
6. Conclusion
CoorDex presents a modular pipeline mapping humanoid loco-manipulation to a structured latent action space — the body and dexterous hand each have separate priors, and downstream RL jointly adapts stepping, wrist placement, and finger contact in a low-dimensional latent space. The key design is the separated-but-coordinated prior composition: the body prior handles wrist placement emerging from whole-body motion, the hand prior captures reusable finger coordination patterns, and the coordination trunk lets the policy reason about task phase and contact state. Ablations prove both body+hand priors are indispensable — all-joint-space never grasps, body-prior-only degrades to stop-and-grasp. The core insight is that continuous dexterous manipulation is not "walk then grasp" but "grasp while walking" — the body prior gets the wrist there, the hand prior coordinates the fingers, and residual RL synchronizes both during locomotion; none can be omitted.
flowchart TD
A["Whole-body tracking teacher π_T^b"] --> B["Body prior R_b, D_b (29-DoF, dim=16)"]
C["Dexterous hand tracking teacher π_T^h"] --> D["Hand prior R_h, D_h (20-DoF, dim=12)"]
E["Mocap demonstrations"] --> A
E --> C
B --> F["Frozen prior mean μ_b"]
D --> G["Frozen prior mean μ_h"]
F --> H["Coordination trunk f_coord"]
G --> H
I["Task state + object geometry + contact"] --> H
H --> J["Body residual head f_b → Δz_b"]
H --> K["Hand residual head f_h → Δz_h"]
J --> L["z̃_b = μ_b + Δz_b"]
K --> M["z̃_h = μ_h + Δz_h"]
L --> N["Body decoder D_b → joint targets"]
M --> O["Hand decoder D_h → finger targets"]
N --> P["PD controller → G1 robot"]
O --> P
P --> Q{"Task"}
Q -->|WalkGrab| R["Grasp bottle while walking 0.55"]
Q -->|OpenFridge| S["Open door stepping back 0.66"]
Q -->|WalkPickTurn| T["Pick cube and turn 0.89"]
