PAPER DEEP DIVE
Extreme-RGMT: Continual Learning of Highly Dynamic Skills for Robust Generalist Humanoid Control
Humans can progressively acquire highly dynamic motor skills while preserving reliable everyday motor abilities. In contrast, existing humanoid controllers face a trade-off between generalist and specialist capabilities: generalist motion tracking policies struggle to reliably execute rare highly dynamic motions, whereas specialist training can degrade previously acquired behaviors. We introduce Extreme-RGMT, a two-stage continual learning framework for robust generalist humanoid control. The method first learns a generalist motion-tracking base policy from diverse multi-source motion data, then employs an asymmetric skill acquisition and capability consolidation mechanism to constrain policy drift on mastered motions while emphasizing difficult dynamic segments. To address the scarcity of highly dynamic motions, their high failure rates, and the resulting shortage of informative samples, Extreme-RGMT combines difficulty-aware sampling with advantage-prioritized trajectory resampling to emphasize critical segments. Experiments show that Extreme-RGMT achieves state-of-the-art generalist whole-body motion-tracking performance, including substantially improved completion of challenging highly dynamic motions. The resulting controller directly executes diverse unseen highly dynamic motions under fixed references and online inertial motion-capture inputs, advancing generalist whole-body motion-tracking controllers toward highly dynamic motor capabilities at the human-expert level.
1. Background and Motivation
A central goal of humanoid robot control is enabling robots to progressively acquire whole-body motor capabilities approaching human experts. Human experts perform highly dynamic, contact-rich, strongly coordinated complex motions built upon stable fundamental motor skills. Through long-term practice, humans expand motor boundaries while retaining previously acquired abilities. This progressive development from fundamental to expert skills provides natural inspiration for robot learning.
However, existing generalist motion tracking methods fall short. Prior methods train unified controllers from large-scale motion libraries supporting basic locomotion, simple pose following, and regular whole-body motions, but these capabilities mainly cover relatively smooth motion regimes. For expert-level highly dynamic motions (backflips, aerial twists), existing controllers still struggle with reliable execution.
The key reason: effective control requirements of highly dynamic motions are concentrated in a small number of short temporal segments (aerial posture adjustment, landing recovery). Once the policy fails in these segments, resulting rollouts contain few informative successful transitions. In large-scale generalist training, these sparse but critical learning signals are further diluted by abundant regular motion samples. Meanwhile, specialist optimization methods rely on preprocessed high-quality data lacking real-world noise, and specialist training degrades previously acquired behaviors — the stability-plasticity dilemma.
2. System Overview
2.1 Problem Formulation
Formulated as a POMDP $\mathcal{M}=(\mathcal{S},\mathcal{O},\mathcal{A},P,r,\gamma)$. At each step $t$, policy $\pi_\theta$ receives proprioceptive observation history $o^{\mathrm{prop}}_{t-H:t}$, past actions $a_{t-H-1:t-1}$, and local reference-motion window $g_{t-L:t+L}$, outputting residual joint-position command $a_t$.
Proprioceptive observation:
where $\mathbf{g}_t^{\mathrm{proj}}$ is gravity projected into the base frame, $\omega_t$ is base angular velocity, $q_t-q_0$ is joint-position offset, $\dot{q}_t$ is joint velocity. The 29D residual action is added to reference joint pose:
Asymmetric actor-critic: actor uses only deployable observations; critic additionally receives privileged information (reference base height, tracked body-link positions/orientations, base linear velocity).
2.2 Policy Architecture
Proprioceptive and action histories are encoded separately, interleaved, and processed by a causal history encoder to form the query. The reference-motion window is independently encoded as keys and values for cross-attention:
This state-dependent reference aggregation enables the policy to attend to reference-window portions most relevant to its current control state. A finite scalar quantization (FSQ) bottleneck $\hat{u}_t=\mathcal{Q}_{\mathrm{FSQ}}(u_t)$ constrains information to a discrete bounded latent, reducing sensitivity to local inconsistencies in highly dynamic reference trajectories.
3. Stage I: Generalist Motion Tracking Training
3.1 Motion Stratification
The motion set $\mathcal{D}$ includes LAFAN1, AMASS, and in-house Xsens inertial motion-capture data, retargeted to Unitree G1 at 50 Hz. After training base policy $\pi_{\mathrm{base}}$, each motion sequence is split into 10s clips evaluated with 5 randomized rollouts. Clips with ≥80% completion rate form mastered set $\mathcal{D}_m$ (2.82h), the rest form challenging set $\mathcal{D}_c$ (0.28h).
| Motion Set | Symbol | Hours | Training Role |
|---|---|---|---|
| Mastered | $\mathcal{D}_m$ | 2.82 | Consolidation & broad coverage |
| Challenging | $\mathcal{D}_c$ | 0.28 | Highly dynamic acquisition |
3.2 Adaptive Motion Sampling
Adaptive sampling allocates more rollout initializations to difficult temporal bins. Each bin maintains an exponential moving average of failure statistics:
4. Stage II: Progressive Highly Dynamic Skill Expansion
4.1 PACE: Progressive Acquisition and Consolidation
PACE addresses the stability-plasticity dilemma through asymmetric environment allocation: $\xi=0.8$ of environments for skill acquisition on $\mathcal{D}_c$, $1-\xi=0.2$ for capability consolidation on $\mathcal{D}_m$. Consolidation environments sample $\mathcal{D}_m$ uniformly for broad coverage; acquisition environments sample $\mathcal{D}_c$ adaptively to focus on difficulties.
The key innovation is the consolidation constraint: in consolidation environments, policy drift from the base policy is constrained via consolidation loss $\mathcal{L}_{\mathrm{con}}$, with weight $\lambda_{\mathrm{con}}$ adaptively adjusted by training progress — stronger early to protect mastered capabilities, gradually relaxing later to allow more充分 skill acquisition.
4.2 STAR: Segment-Aware Trajectory Advantage Resampling
Adaptive sampling allocates opportunities only at the bin level, unable to distinguish trajectory quality within the same difficult region. STAR transforms bin-level difficulty into transition-level weights with difficulty-conditioned advantage normalization:
High-difficulty group $H$ and remaining group $E$ are normalized independently, preventing shared statistics from obscuring relative learning potential in difficult regions. STAR then evaluates contiguous trajectory fragments by average raw advantage $q_{b,\tau}$, retaining top 5% per difficult bin:
Selected fragments are mixed into PPO mini-batches at $\rho_{\mathrm{star}}=0.25$ ratio, focusing acquisition-side updates on trajectory experience with greater improvement potential in high-difficulty regions.
5. Experimental Results
5.1 Generalist and Specialist Motion Tracking
| Method | In-source Succ.↑ | Unseen Succ.↑ | Challenging Succ.↑ |
|---|---|---|---|
| ExBody2 | 85.63 | 66.78 | - |
| BeyondMimic | 94.72 | 85.53 | 76.5 |
| Extreme-RGMT (Stage I) | 96.12 | 89.47 | 78.2 |
| Extreme-RGMT (Full) | 96.05 | 89.82 | 90.9 |
The full method maintains generalist capability (96.05% in-source, 89.82% unseen) while boosting challenging-motion success from 78.2% to 90.9%, significantly outperforming Stage I alone. This validates PACE+STAR's effectiveness in acquiring highly dynamic skills while retaining generalist capability.
5.2 STAR's Cross-Source Effect
| Source | w/o STAR | Full Method | Gain |
|---|---|---|---|
| AMASS Motions | 82.2 | 90.9 | +8.7 |
| In-house Xsens Motions | 45.5 | 86.3 | +40.8 |
STAR's gain on Xsens motions far exceeds AMASS (+40.8 vs +8.7). Xsens inertial capture data has lower quality, more susceptible to root drift, local pose inconsistencies, and contact-timing errors. Their highly dynamic segments rely more heavily on effective use of limited informative experience. STAR strengthens effective learning signals in critical temporal regions by prioritizing high-advantage fragments.
5.3 Real-World Deployment
| Evaluation | Unified Enc. | w/o FSQ | w/o STAR | Ours |
|---|---|---|---|---|
| AMASS Replay | 85.0 | 80.0 | 80.0 | 90.0 |
| Xsens Teleop (Dynamic) | 75.0 | 65.0 | 45.0 | 85.0 |
| Xsens Teleop (General) | 90.0 | 100.0 | 100.0 | 100.0 |
The full method achieves strongest performance in fixed replay and highly dynamic teleoperation while maintaining 100% generalist teleoperation success. The w/o STAR variant degrades most severely on highly dynamic Xsens teleoperation (45.0%), highlighting the importance of prioritizing effective trajectory experience when inertial motion-capture inputs contain noise.
6. Method Workflow
7. Limitations and Discussion
8. Conclusion and Implications
Extreme-RGMT's core contribution is introducing the continual learning paradigm into humanoid highly dynamic skill acquisition. The two-stage design is profound: first establish broad-coverage generalist foundations, then progressively expand capability boundaries through asymmetric acquisition-consolidation. PACE's consolidation constraint directly corresponds to knowledge retention in continual learning — preventing new skill learning from eroding mastered capabilities. STAR addresses the core contradiction of extremely scarce effective experience in highly dynamic motions — not simply increasing sampling frequency of difficult bins, but identifying and prioritizing genuinely informative experience at the trajectory-fragment level.
STAR's remarkable +40.8% gain on Xsens data reveals an important insight: the lower the data quality and the greater the noise, the more critical intelligent experience utilization becomes. Xsens inertial capture data's root drift and local inconsistencies make its highly dynamic segments more failure-prone, but these failure-recovery boundaries also contain the densest learning signals. The FSQ bottleneck similarly reflects robustness considerations for real-world noise — constraining command representation to a discrete bounded space reduces sensitivity to local reference errors.
In real-world deployment, the policy directly executes highly dynamic motions absent from the training corpus (via online Xsens input), indicating it learns not simple motion memory but generalizable highly dynamic control capability — advancing generalist whole-body motion-tracking controllers toward human-expert-level dynamic motor capabilities.
SOURCE LINKS
