PAPER DEEP DIVE
AnyBody: Free-Form Whole-Body Humanoid Control from Arbitrary Keypoint Guidance
We present AnyBody, a unified whole-body humanoid controller driven by an arbitrary subset of body keypoints chosen at deploy time. Prior physics-based trackers either rely on expensive full-body motion capture and error-prone trajectory retargeting, which bottleneck scalable data collection and policy learning, or decompose upper- and lower-body control into separate hierarchical representations, sacrificing the coordinated whole-body motions that loco-manipulation requires. We close this gap by learning a single latent motion representation that any keypoint subset can address. To achieve this, we first train a privileged teacher tracker on a large unstructured motion corpus and distill it online into a deterministic encoder-decoder student whose latent space is a unit sphere. We then train a transformer keypoint encoder that admits any subset of body keypoints through masked self-attention, aligning it to the privileged latent. Additionally, we treat the frozen decoder as a motor prior and specialize downstream tasks with a lightweight residual corrector in the latent space. We demonstrate the effectiveness of AnyBody by tracking large-scale human motions from arbitrary keypoint subsets, free-form control, flexibly teleoperating, and learning downstream behaviors including locomotion, in-air writing, and obstacle-reach.
1. Overview: Free-Form Whole-Body Humanoid Control from Arbitrary Keypoint Guidance
AnyBody presents a unified whole-body humanoid controller driven by an arbitrary subset of body keypoints, enabling free-form whole-body motion control. The core motivation: existing physics-based motion tracking methods rely on expensive mocap systems, while hierarchical control frameworks decouple upper/lower body, limiting coordinated whole-body behavior. AnyBody learns a shared spherical latent motion representation that unifies diverse forms of partial-body conditioning within a single control framework. Operators can use different keypoint configurations depending on the task and deployment setting — from sparse upper-body tracking to full-body control or even single end-effector guidance. Validated on a Unitree G1 with zero-shot real-robot deployment, generalization to unseen human motions, and downstream RL finetuning (obstacle avoidance, writing, container placement, etc.).
2. Core Problem: Flexible Humanoid Teleoperation and Whole-Body Coordination
The central challenge in humanoid control is the high degrees of freedom and the difficulty of maintaining stable whole-body coordination. The dominant paradigm is physics-based motion tracking — policies first learn to imitate human motions in simulation, then are controlled using dense motion references from mocap systems. But mocap systems are expensive and cumbersome, limiting scalable humanoid teleoperation and large-scale whole-body control data collection. Hierarchical frameworks (upper-body manipulation + lower-body locomotion) allow teleoperating only the upper body but assume a fixed control interface and explicitly separate upper/lower body, limiting coordinated whole-body behavior. AnyBody's key insight: learn a shared spherical latent motion representation that unifies diverse partial-body conditioning into one whole-body control framework — different keypoint subsets map to the same latent space via a Transformer encoder, then a frozen dynamics-aware decoder produces coordinated whole-body motion.
3. Method: Three-Stage Training Pipeline
3.1 Stage One: Latent Space and Dynamics-Aware Decoder
First, a privileged teacher $\pi^T$ (trained with RL to track a large-scale motion dataset) is distilled online into a deterministic encoder-decoder. The encoder output is L2-normalized onto the unit hypersphere:
$$z = \widehat{E}_\theta(s^g, s^p) \in \mathbb{S}^{d_z-1}, \quad a = D_\phi(z, s^p) \tag{1}$$The distillation loss is a behavior-cloning term plus a cosine smoothness regularizer:
$$\mathcal{L}_1 = \|D_\phi(z, s^p) - \pi^T(s)\|_2^2 + \lambda_{\text{sm}}(1 - \cos(z_t, z_{t-1})) \tag{2}$$The spherical projection constrains all latents to the unit hypersphere, eliminating the norm-drift failure mode of unconstrained BC. $\lambda_{\text{sm}}=0.1$. The teacher PPO uses $r = \sum_i w_i r_i$ where $r_i = \exp(-e_i^2/\sigma_i^2)$ for tracking terms. After training, the decoder $D$ is frozen as a motor prior for downstream stages.
3.2 Stage Two: Partial Keypoint Encoder via Online Distillation
The joint-command encoder is replaced with a keypoint encoder $E_\psi^{kp}$ that consumes masked keypoints and proprioceptive state, emitting a latent in the same space. $E_\psi^{kp}$ is a self-attention Transformer with one token per keypoint; absent keypoints are dropped via the attention mask $m$, so any subset is admissible at test time. Latent distillation aligns the keypoint latent with the privileged latent:
$$\mathcal{L}_2 = (1 - \cos(\hat{z}^{kp}, \hat{z})) + \lambda_a \|D(\hat{z}^{kp}, s^p) - a_{\text{JC}}\|_2^2 \tag{3}$$Training follows a three-phase masking curriculum: all keypoints visible → per-keypoint Bernoulli masking (keep-probability annealed 1.0→0.4) → sampling semantic deployment modes (torso/wrists/ankles-only, etc.). The 5 keypoints (KP5): torso, left/right wrist, left/right ankle, each token is $T \times 3 = 45$-dimensional (15 log-spaced time offsets spanning a 0.5s window).
3.3 Stage Three: Task-Specific RL Finetune in Latent Space
The policy is finetuned in the latent space with PPO; the RL action is the latent $z$, and the environment applies $a = D(z, s^p)$ with $D$ frozen as a motor prior. Rather than finetuning $E_\psi^{kp}$ or inserting LoRA adapters, a small residual corrector $g_\xi$ is learned:
$$z = \widehat{\mu + \alpha \Delta z}, \quad \Delta z = g_\xi(o, \mu), \quad \mu = E_\psi^{kp}(k \odot m, s^p) \tag{4}$$Small-gain initialization ($\Delta z \approx 0$) makes the policy reproduce the distilled tracker at initialization — a safe start that avoids reward collapse from random-latent exploration on a pretrained decoder. $g_\xi$ is a shallow per-body-token Transformer with output projection Xavier-initialized at gain 0.01. The RL reward:
$$r_t = \sum_k w_k r_t^{(k)}, \quad r_i = \exp(-e_i^2 / \sigma_i^2) \tag{5}$$
4. Experiments
4.1 Quantitative Tracking Analysis
Tracking accuracy is evaluated under different keypoint command modes. Full-body (5 points) achieves 97.6% success rate, upper-body (3 points) 94.8%, wrists (2 points) 91.2%, single-wrist 88.4%. POI position error ranges 8.68-10.90 cm, velocity error 0.284-0.301 m/s.
| Command Mode | # Points | Success Rate (%) | POI Pos. Err. (cm) | POI Vel. Err. |
|---|---|---|---|---|
| Full Body | 5 | 97.6 | 10.90 | 0.301 |
| Upper Body (Wrists+Torso) | 3 | 94.8 | 9.48 | 0.284 |
| Wrists | 2 | 91.2 | 9.12 | 0.295 |
| Single Wrist | 1 | 88.4 | 8.68 | 0.290 |
4.2 Downstream RL Finetune
Five task families validate downstream RL finetuning. Without finetuning, most tasks have low or zero success rates; after finetuning, all improve dramatically. Obstacle reach rises from 54.68% to 97.09%, low-clearance from 44.36% to 99.47%, container placement from 0% to 95.56%, wrist-writing from 0% to 97.87%.
| Task | w/o RL finetune | w/ RL finetune |
|---|---|---|
| Obstacle reach | 54.68% | 97.09% |
| Barrier | 10.49% | 96.04% |
| Low-clearance | 44.36% | 99.47% |
| Container | 0% | 95.56% |
| Wrist-writing | 0% | 97.87% |
4.3 Real-Robot Experiment
AnyBody is deployed zero-shot on a physical Unitree G1. VR teleoperation is driven through an interactive UI, with operators using different keypoint configurations. Real-robot snapshots demonstrate three task families: omnidirectional locomotion (torso keypoint), in-air writing (right-wrist keypoint), obstacle reach (right-wrist + obstacle OBBs).
5. Limitations
- Minimum keypoint count: While supporting arbitrary subsets, single-keypoint mode achieves 88.4% success and lower precision than full-body mode (97.6%); whole-body coordination degrades under extremely sparse conditions.
- Real-robot perception gap: Real-robot deployment is zero-shot VR teleoperation; autonomous perception-driven whole-body control is not demonstrated; the perceptual layer (vision, touch) of sim-to-real is not addressed.
- Downstream task scope: Five RL finetune tasks are diverse but all trained in simulation; more complex real-world manipulation tasks (e.g., multi-object rearrangement, tool use) are not verified.
6. Conclusion
AnyBody presents a unified whole-body humanoid controller driven by arbitrary body keypoint subsets. The core design is a three-stage pipeline: Stage 1 distills a privileged teacher into a spherical latent space encoder-decoder; Stage 2 freezes the decoder and trains a Transformer keypoint encoder via online latent distillation to support arbitrary subsets; Stage 3 finetunes task-specific behaviors with residual latent RL. Spherical projection eliminates latent norm drift; small-gain initialization ensures a safe RL starting point. Experiments demonstrate that 1 to 5 keypoints all maintain coordinated whole-body motion (88-98% success), downstream RL finetuning raises zero-baseline tasks to 95-99%, and zero-shot real-robot deployment works. The core insight is that whole-body control does not require complete motion capture — a shared spherical latent motion manifold can unify all conditioning forms from single-point to full-body, and the key is letting the decoder learn "how to reconstruct coordinated whole-body motion from arbitrary local cues."
flowchart TD
A["Large-scale human motion dataset"] --> B["Stage 1: Privileged teacher PPO"]
B --> C["Online distillation → Encoder E + Decoder D"]
C --> D["Spherical latent space S^(dz-1)"]
D --> E["Freeze D as motor prior"]
F["Keypoints (arbitrary subset)"] --> G["Stage 2: Transformer encoder E_kp"]
G --> H["Mask curriculum: all visible → Bernoulli → semantic modes"]
H --> I["Latent distillation: align E_kp output to E output"]
E --> J["Stage 3: Residual RL finetune"]
I --> J
J --> K["Corrector g_ξ → Δz (small-gain init)"]
K --> L["z = μ + αΔz → frozen D → whole-body motion"]
L --> M{"Deployment mode"}
M -->|5-point full body| N["Success 97.6%"]
M -->|3-point upper body| O["Success 94.8%"]
M -->|1-point single wrist| P["Success 88.4%"]
L --> Q["Real-robot G1 zero-shot deployment"]
L --> R["Downstream RL: obstacle/writing/container 95-99%"]
