Learning Reusable Hybrid Motion Priors for Humanoid Locomotion from Motion Imitation

Background and Motivation

Reinforcement learning has become a standard tool for humanoid robot control, with large-scale simulation enabling robust controllers deployable on physical robots. However, standard RL controllers are typically task-specific solutions—each new objective requires a new reward design and training process, during which the policy must reacquire basic motor competence such as balance, stepping, contact timing, and whole-body coordination. Motion imitation offers a systematic way to acquire motor competence from data: given retargeted human motions, RL can train humanoid robots to track reference trajectories, learning diverse natural skills without designing separate task rewards for each behavior. But motion imitation policies are fundamentally reference trackers—they execute provided motion references but do not directly solve downstream tasks such as following commands, reaching goals, or responding to task-level observations.

From a technical trajectory perspective, this work sits at the intersection of motion imitation and motion prior research. DeepMimic pioneered the paradigm of example-guided RL synthesizing physics-based character skills from motion clips, with subsequent methods learning more general humanoid tracking policies and improving robustness and scalability. In the motion prior direction, prior work has explored reusable low-level control spaces where downstream policies query learned controllers or latent representations instead of acting directly in the full joint-action space. Neural motor primitives provided early examples, with subsequent work using VQ-VAE to encode motor skills as discrete codebooks where downstream policies select codebook entries to generate actions. This paper's key distinction is "residual quantization"—not quantizing the full motion latent but only the residual between reference intent and proprioception, dramatically reducing codebook learning difficulty and improving generalization.

This paper aims to bridge this gap by converting motion imitation skills into a reusable Hybrid Motion Prior (HMP). The core idea: first train a motion imitation expert policy, then distill it into a frozen architecture (proprioceptive encoder + residual vector-quantized codebook + action decoder), and finally train task-level policies on top of the frozen HMP by selecting discrete codebook entries to solve downstream tasks. This design enables skill reuse across tasks without retraining the low-level motor vocabulary, while the discrete codebook selection provides an interpretable action interface. The study also discovers that the codebook has an interpretable structure—the number of active RVQ stages modulates available gait patterns, and training the codebook with the rotation trick improves latent organization and reduces downstream falls compared to a standard straight-through estimator (STE).

Three-Stage Pipeline Method

The method consists of three phases. The proprioceptive observation is defined as:

$$p_t = \left[{}^b\omega_t, {}^b g_t, q_t - q_{\text{nom}}, \dot{q}_t, a_{t-1}\right]$$

where ${}^b\omega_t$ is base angular velocity, ${}^b g_t$ is projected gravity, $q_t$ and $\dot{q}_t$ are joint positions and velocities, $q_{\text{nom}}$ is nominal joint posture, and $a_{t-1}$ is the previous action. Actions $a_t$ are interpreted as position targets for joint-level PD controllers.

Phase 1: Motion Imitation Expert

The first phase trains a motion imitation expert using PPO in an asymmetric actor-critic framework. At each environment reset, a motion clip and initial time index are randomly sampled. The motion command is encoded as:

$$m_t = \left[\hat{q}_t, \dot{\hat{q}}_t, {}^b\hat{v}_{xy,t}, {}^b\hat{\omega}_{z,t}, {}^w\hat{h}_t, {}^w\hat{\phi}_t, {}^w\hat{\theta}_t\right]$$

where hatted quantities are joint positions/velocities, local base planar velocity, yaw rate, base height, and roll/pitch angles extracted from the reference. The reward combines base, body, joint, and velocity tracking terms with regularization on contacts, action smoothness, and joint limits. The expert actor uses a temporal convolutional history encoder $H_\alpha(\cdot)$ mapping the history window $[p_{t-1}, \ldots, p_{t-H}]$ ($H=10$) to a proprioceptive history embedding:

$$\eta_t = H_\alpha([p_{t-1}, \ldots, p_{t-H}])$$

The expert is trained for 30,000 iterations on 27 LaFAN motion clips (12 walking, 4 running, 2 sprinting, 3 jumping, 6 fall-and-get-up) with domain randomization and external perturbations.

The deeper logic of the residual quantization design deserves further elaboration. In standard VQ-VAE, the encoder quantizes the full input into discrete codes, requiring the codebook to cover the entire input space's diversity. But in humanoid robotics, proprioception $p_t$ already contains abundant information about the current motion state (joint angles, velocities, base posture, etc.) that is continuous and high-dimensional. If the full motion intent were quantized into discrete codes, the codebook would need to encode both "what the current state is" and "what to do next," with enormous capacity requirements and learning difficulty. The residual design subtracts $ ext{sg}(z_t^{ ext{prop}})$ (stop-gradient copy of proprioceptive latent) from $z_t^{ ext{ref}}$ (reference intent), so the codebook $y_t$ only encodes "the residual portion of reference intent beyond what proprioception can infer"—purely the motion intent increment. This decoupling makes the continuous proprioceptive encoder responsible for balance and basic locomotion competence, while the discrete codebook handles task-conditioned motion pattern modulation, with the two being complementary rather than redundant. Experiments confirm this complementary division—freezing the codebook and using only the proprioceptive encoder still produces basic standing behavior.

Phase 2: Hybrid Motion Prior Distillation

The second phase distills the expert into a reusable HMP. The frozen expert generates short on-policy rollouts, storing proprioceptive measurements, motion commands, expert actions, and termination masks, updating a student model via supervised learning. The architecture includes two encoders sharing a common latent space and an action decoder. The reference encoder $E_\phi(\cdot)$ receives the motion reference command $m_t$ (privileged during training), producing a reference latent intent:

$$z_t^{\text{ref}} = E_\phi(p_t, \eta_t, m_t)$$

The proprioceptive encoder $P_\theta(\cdot)$ outputs the proprioceptive latent:

$$z_t^{\text{prop}} = P_\theta(p_t, \eta_t)$$

Only $P_\theta$ is retained at deployment; the reference encoder is used only during distillation. Residual quantization quantizes only the residual between reference and proprioceptive latents:

$$y_t = z_t^{\text{ref}} - \text{sg}(z_t^{\text{prop}})$$

where $\text{sg}(\cdot)$ is the stop-gradient operator. This design makes the proprioceptive encoder capture what can be inferred from robot proprioception, while the codebook encodes only the residual motor intent needed to track the reference, reducing learning difficulty. The discrete module is an $M$-stage residual vector quantizer (RVQ), with each stage's codebook $C^{(m)} = \{e_1^{(m)}, \ldots, e_K^{(m)}\}$ containing $K$ entries. RVQ selects one code entry per stage and sums them:

$$\hat{y}_t = \sum_{m=1}^{M} e_{k_t^{(m)}}^{(m)}$$

During distillation, quantizer dropout evaluates the sum over a random prefix of $M' \leq M$ active stages, encouraging earlier stages to capture coarse motion structure while subsequent stages progressively refine residuals. The action decoder $D_\psi(\cdot)$ receives $\hat{z}_t = \text{sg}(z_t^{\text{prop}}) + \hat{y}_t$ and reconstructs actions. Configuration: 8-stage RVQ ($M=8$), $K=1024$ entries per stage, 128-dim latents, codebook EMA decay 0.99, 6000 distillation iterations.

flowchart TD
    subgraph P1["Phase 1: Motion Imitation Expert"]
        L["LaFAN clips"] --> RT["Retarget to G1"]
        RT --> EP["PPO Expert Policy
Track reference motion"]
    end
    subgraph P2["Phase 2: HMP Distillation"]
        EP -->|"Frozen rollouts"| PE["Proprio Encoder P_θ"]
        EP --> RE["Reference Encoder E_φ"]
        PE --> RQ["Residual Quantization
y = z_ref - sg(z_prop)"]
        RE --> RQ
        RQ --> CB["RVQ Codebook
8 stages × 1024 entries"]
        CB --> AD["Action Decoder D_ψ"]
    end
    subgraph P3["Phase 3: Downstream Task Learning"]
        CB -->|"Frozen"| TP["Task Policy F_ξ
Select codebook indices"]
        TP -->|"Output code indices"| CB
        TP --> T1["Velocity Tracking"]
        TP --> T2["Point-goal Navigation"]
        TP --> T3["Fall Recovery"]
    end

The improvement of the rotation trick (RT) over the straight-through estimator (STE) also deserves deeper analysis. STE trains VQ codebooks by passing quantization gradients directly to the encoder, but this gradient approximation can lead to uneven codebook utilization—some code entries are frequently used while others "die" (never selected). The rotation trick computes gradients by rotating encoder outputs and codebook entries in vector space, providing more accurate gradient estimates so all code entries can be effectively updated. Experimental results show STE and RT achieve similar tracking errors, but RT's fall rate is only one-fifth of STE's (2.1% vs 10.2%), indicating RT learns better codebook latent organization—code entries are more uniformly distributed, and downstream policies are less likely to select combinations leading to instability. This finding has important practical implications for VQ-based motion priors: codebook training methods affect not only reconstruction quality but more critically the robustness of downstream tasks.

Phase 3: Downstream Task Learning

The third phase freezes the history encoder and HMP, training task-level policies. Downstream tasks no longer provide reference motion $m_t$ but task-specific commands $c_t$. The task-level policy $F_\xi(\cdot)$ receives $p_t, \eta_t, c_t$ and outputs categorical logits for each quantization stage:

$$\ell_t = F_\xi(p_t, \eta_t, c_t) \in \mathbb{R}^{M \times K}$$

For each stage, one index is sampled from the categorical distribution $k_t^{(m)} \sim \text{Cat}(\text{softmax}(\ell_t^{(m)}))$; during evaluation and deployment, the highest-logit index is selected instead. The resulting indices retrieve code vectors from the frozen codebook, summed and passed to the frozen HMP to return joint-position actions. PPO optimizes the categorical code-selection policy while the environment steps with continuous actions from the frozen HMP—new task policies learn to select among motion codes acquired during imitation without modifying the low-level motor vocabulary.

Experimental Results

Distillation Evaluation

Evaluated on 1024 30-second simulation rollouts, the distilled model (with reference encoder and HMP) closely matches the expert on main tracking metrics: body position error 62.2mm, body rotation error 4.8°, joint position error 0.07rad, joint velocity error 0.48rad/s. This validates that the distillation stage effectively preserves the expert's tracking behavior.

Downstream Task Performance

All three downstream tasks run on Unitree G1 (29-DoF) at 50Hz with 4096 parallel environments. Velocity tracking command range: $v_x \in [-1.0, 3.0]$ m/s, $v_y \in [-1.0, 1.0]$ m/s, $\omega_z \in [-3.0, 3.0]$ rad/s.

The rotation trick (RT) vs straight-through estimator (STE) comparison shows similar tracking errors but dramatically different robustness—STE fall rate of 10.2% is about five times RT's 2.1%, indicating the rotation trick learns representations less prone to unstable behavior under challenging velocity commands.

Figure 1: Three-stage pipeline overview—motion imitation expert distilled to HMP for downstream tasks

Figure 2: Velocity tracking behaviors—walking, sidestepping, running, turning, and stopping

Figure 3: Gait patterns at different active RVQ depths—from standing to walking to fast walking to running

Codebook Structure Analysis

By varying the active RVQ depth $M_{\text{act}}$, an ordered gait hierarchy emerges:

This shows the RVQ codebook spontaneously forms a hierarchical gait structure: the first 1-2 stages encode coarse motion (standing, slow walk), stage 3 is the key transition (enabling fast walk), and 5+ stages produce running. This interpretable codebook structure is a significant advantage of the HMP architecture—different RVQ depths correspond to qualitatively different motion modes, analogous to the progressive hierarchy from standing to walking to running in human locomotion.

From a broader perspective, this work represents the trend of robot learning shifting from "task-specific policies" to "reusable motor foundation models." Just as large-scale pretraining in NLP and CV gave rise to foundation models, robot learning is exploring similar paradigms—first acquiring general motor competence through motion imitation, distilling it into reusable motion priors, and finally fine-tuning lightweight high-level policies on different tasks. This paradigm's advantage: motor competence acquisition (the most time-consuming phase) is done once, and subsequent new tasks only need to train codebook selection policies (with far fewer parameters than full policies), dramatically reducing task training costs. The HMP's frozen design also guarantees motor safety—downstream policies cannot modify the low-level motor vocabulary, preventing task policies from learning dangerous motion patterns in pursuit of task rewards. This "safety barrier" design is particularly important for real robot deployment. Additionally, the discrete codebook interface naturally supports skill composition—different code entry sequences correspond to different motion patterns, providing a structured action space for long-horizon task planning.

Limitations and Future Work

Limitation 1: Fixed codebook capacity. The 8-stage, 1024-entry-per-stage codebook is preset and may be insufficient for complex motion sets or redundant for simple tasks. Future work could explore adaptive codebook sizing or dynamic codebook growth mechanisms.

Limitation 2: Expert quality dependency. HMP quality is upper-bounded by the Phase 1 expert—if the expert cannot learn certain motion patterns, the HMP codebook cannot encode them either. Additionally, the 27 LaFAN clips offer limited motion diversity; broader motion datasets may produce richer codebook structures. Real robot validation covers only the velocity tracking task; navigation and fall recovery are not yet validated on real hardware.

Summary and Insights

This paper presents a three-stage pipeline converting motion imitation skills into a reusable Hybrid Motion Prior. Key technical contributions include: residual quantization design $y_t = z_t^{\text{ref}} - \text{sg}(z_t^{\text{prop}})$ separating motor intent from proprioception so the codebook only encodes residuals, reducing learning difficulty; $M$-stage RVQ $\hat{y}_t = \sum_{m=1}^{M} e_{k_t^{(m)}}^{(m)}$ with quantizer dropout forming hierarchical codebook structure; task-level policy $F_\xi$ solving downstream tasks by selecting discrete codebook indices $\ell_t \in \mathbb{R}^{M \times K}$ without modifying the low-level motor vocabulary; rotation trick reducing fall rate from 10.2% to 2.1% vs STE. On Unitree G1, the same frozen HMP supports velocity tracking (Lin. RMSE 0.46m/s), point-goal navigation, and fall recovery, with the velocity tracking policy transferring zero-shot to the real robot. Codebook structure analysis reveals an ordered gait hierarchy—active RVQ depth from 1 to 8 corresponds to standing, slow walk, fast walk, and running progressive modes. This work provides a new paradigm for reuse and modularization of humanoid locomotion skills, with the HMP as a frozen action interface reusable by arbitrary downstream policies being significant for reducing task training costs.