ADP: Adversarial Dynamics Priors for Physically Grounded Humanoid Locomotion

ADP: Adversarial Dynamics Priors for Physically Grounded Humanoid Locomotion

Paper: ADP: Adversarial Dynamics Priors for Physically Grounded Humanoid Locomotion

Authors: Seokju Lee, Jeongtae Lee, Jeonghyeok Lim, Jeonguk Kang, Byungwook Lee, Seungho Han, Keun Ha Choi, Dongil Park, Kyung-Soo Kim (KAIST / KIMM / Samsung / Hanyang Univ.)

Links: Project Page · arXiv:2607.03454

Code: No public code repository; project page with demo videos only

---

One-Sentence Summary

ADP replaces kinematic motion-style features with trajectory-optimization-derived dynamics features (CoM motion, centroidal momentum, contact forces, contact states) as the adversarial regularization target, training a discriminator to evaluate whether policy-generated dynamics windows are consistent with the reference distribution, improving the 80%-success impulse threshold by 16.7% and reducing recovery time by 47.9% under four-direction external push perturbations.

---

Background and Motivation

Humanoid robots have recently attracted significant attention as platforms capable of operating in human-centric environments. Learning-based control has become the dominant paradigm for controlling such high-dimensional systems, building on its success in quadrupedal locomotion. However, humanoid locomotion remains substantially more challenging than quadrupedal locomotion due to the increased degrees of freedom, the underactuated floating-base dynamics, and the need to maintain balance with intermittent contacts.

Recent approaches have moved beyond purely hand-designed reward functions to leverage motion datasets for guiding policy learning. A representative method uses imitation rewards to directly track reference motions (e.g., DeepMimic), with tracking terms for pose, velocity, end-effector, and root-state matching. While effective for generating natural motions, the reliance on phase variables or target poses limits policy flexibility when perturbations drive the robot away from the reference trajectory. Another approach, AMP, formulates motion imitation as a distribution matching problem using a discriminator-based style reward to encourage the policy to generate a kinematic motion-style distribution similar to the motion dataset.

However, the AMP prior is still based on kinematic features (joint pose, velocity, end-effector state), and therefore does not directly regularize the dynamics-level recovery behaviors required after perturbations—momentum regulation, contact-force modulation, and contact timing. This is the core problem ADP addresses: replacing kinematic-style imitation with dynamics-level adversarial regularization.

The paper's key insight is that external impulse perturbations first excite floating-base momentum and contact forces before being expressed through joint-level kinematics. Therefore, adversarial regularization directly in the dynamics feature space exposes perturbation transients earlier and more strongly than in the kinematic feature space. The representation sensitivity analysis quantitatively validates this: the dynamics representation reaches 6.0x baseline deviation within 20ms, while the kinematic representation takes 160ms to reach 4.3x.

---

Method

Problem Formulation

ADP uses an asymmetric actor-critic architecture for training the humanoid locomotion policy. The actor receives proprioceptive observations (IMU signals, projected gravity, joint states, previous actions, and velocity commands), outputting target position offset actions for active joints. Base linear velocity is excluded from the actor input for real-world deployability but provided to the critic as privileged information. The total reward is defined as:

$$r_t = w_G r_t^G + w_D r_t^D + w_R r_t^R$$

where $r_t^G$ is the task reward, $r_t^D$ is the proposed adversarial dynamics prior reward, and $r_t^R$ denotes regularization terms.

Reference Trajectory Generation

To construct the reference dataset for ADP training, the paper uses SRBD-based trajectory optimization. SRBD represents the robot through CoM dynamics and foot contact forces:

$$m\ddot{\mathbf{p}}_t^{\text{com}} = m\mathbf{g} + \sum_{i \in \{L,R\}} \mathbf{f}_t^i$$

$$\dot{\mathbf{L}}_t^{\text{com}} = \sum_{i \in \{L,R\}} (\mathbf{p}_t^i - \mathbf{p}_t^{\text{com}}) \times \mathbf{f}_t^i$$

where $m$ is the total robot mass, $\mathbf{p}_t^{\text{com}}$ is the CoM position, $\mathbf{L}_t^{\text{com}}$ is the centroidal angular momentum, $\mathbf{p}_t^i$ is the contact position of foot $i$, and $\mathbf{f}_t^i$ is the corresponding contact force. TO generates reference trajectories for multiple locomotion modes (forward running, walking, turning, lateral stepping, backward walking), from which dynamics quantities are extracted. The key design choice: $\mathcal{D}_{\text{dyn}}$ stores dynamics features rather than motion-tracking targets.

Dynamics Feature Representation

ADP compares TO-derived reference trajectories and policy rollouts in a shared feature space. The timestep-level dynamics feature is defined as:

$$\mathbf{z}_t = [\mathbf{v}_{t,h}^{\text{com}}, \mathbf{L}_{t,h}^{\text{com}}, \bar{\dot{\mathbf{L}}}_{t,h}^{xy,\text{com}}, \hat{\mathbf{f}}_{t,h}^L, \hat{\mathbf{f}}_{t,h}^R, c_t^L, c_t^R]$$

where subscript $h$ denotes the heading frame, $\mathbf{v}_{t,h}^{\text{com}}$ is the heading-frame CoM velocity, $\mathbf{L}_{t,h}^{\text{com}}$ is the centroidal angular momentum, $\bar{\dot{\mathbf{L}}}_{t,h}^{xy,\text{com}}$ is the planar angular momentum rate, $\hat{\mathbf{f}}_{t,h}^L$ and $\hat{\mathbf{f}}_{t,h}^R$ are weight-normalized foot contact forces, and $c_t^L, c_t^R$ are binary contact indicators.

For policy rollouts, the centroidal angular momentum is computed by summing link-wise spin and orbital momentum over all links:

$$\mathbf{L}_t^{\text{com}} = \sum_i \left(\mathbf{I}_{i,w}\boldsymbol{\omega}_i + (\mathbf{p}_i - \mathbf{p}^{\text{com}}) \times m_i(\mathbf{v}_i - \mathbf{v}^{\text{com}})\right)$$

The discriminator input is conditioned on both the velocity command and tracking error:

$$\boldsymbol{\xi}_t = [\mathbf{z}_t, \mathbf{v}_t^{\text{cmd}}, \mathbf{e}_t]$$

where $\mathbf{e}_t$ is the planar velocity and yaw-rate tracking error. This aligns the reference distribution with the current command rather than serving as a tracking-error reward term.

Temporal Window Construction

Contact switching, force modulation, and momentum recovery cannot be sufficiently represented by a single timestep. ADP constructs a short temporal window:

$$\boldsymbol{\Xi}_t = [\bar{\boldsymbol{\xi}}_{t-K+1}, \bar{\boldsymbol{\xi}}_{t-K+2}, \ldots, \bar{\boldsymbol{\xi}}_t]$$

where $\bar{\boldsymbol{\xi}}_t$ is the normalized dynamics feature and $K$ is the window length. The discriminator receives the flattened vector $\text{vec}(\boldsymbol{\Xi}_t)$ as input. The reference dataset is $\mathcal{D}_{\text{dyn}} = \{\boldsymbol{\Xi}_j^{\text{ref}}\}_{j=1}^N$.

Adversarial Training

The discriminator $D_\phi$ is trained to distinguish reference windows from policy-generated windows using binary cross-entropy:

$$\mathcal{L}_D = -\mathbb{E}_{\boldsymbol{\Xi}^{\text{ref}}}[\log D_\phi(\boldsymbol{\Xi}^{\text{ref}})] - \mathbb{E}_{\boldsymbol{\Xi}^\pi}[\log(1 - D_\phi(\boldsymbol{\Xi}^\pi))]$$

The dynamics prior reward encourages the policy to generate windows classified as reference:

$$r_t^D = -\log(1 - D_\phi(\boldsymbol{\Xi}_t^\pi))$$

The policy is optimized via PPO, maximizing the weighted sum of task and dynamics prior rewards.

Method Pipeline

graph TB
    subgraph Reference Gen
        TO["SRBD Trajectory Optimization
Forward/Walk/Turn/Lateral/Backward"] --> REF["Reference Dynamics Features
D_dyn"]
    end
    subgraph Policy Training
        OBS["Proprioceptive Obs
IMU+Joint State+Command"] --> POLICY["Actor Policy
PPO Training"]
        POLICY --> ROLL["Policy Rollout"]
        ROLL --> DYN_FEAT["Dynamics Feature Extraction
CoM+Momentum+Contact Force+Contact State"]
        DYN_FEAT --> WINDOW["Temporal Window K=8"]
        WINDOW --> DISC["Discriminator D_phi"]
        REF --> DISC
        DISC --> REWARD_D["Dynamics Prior Reward r_D"]
        TASK["Task Reward r_G"] --> TOTAL["Total Reward r=wG*rG+wD*rD+wR*rR"]
        REWARD_D --> TOTAL
        TOTAL --> POLICY
    end

---

Experimental Results

The paper applies instantaneous velocity impulses in four directions (lateral ±y, forward +x, backward -x) to assess perturbation recovery. $J_{80}$ is the largest impulse for which the direction-averaged success rate remains above 80%, identified by sweeping $\Delta v$ from 1.5 to 4.5 m/s at 0.5 m/s intervals.

Figure 1: Dynamics-feature alignment for perturbation recovery. An external force at t₂ perturbs the humanoid, driving dynamics features away from the reference distribution. ADP encourages recovery by regularizing perturbed windows toward this distribution, achieving stable locomotion at t₃.

Method	Success Rate (%) ↑	J₈₀ (N·s) ↑	Recovery Time (s) ↓	Vel. Error (m/s) ↓
Vanilla RL	5.5	49.5	10.52	2.89
Dynamics Reward	14.1	66.0	9.73	2.68
AMP	73.4	99.0	4.76	1.30
ADP (ours)	91.4	115.5	2.48	0.84

Table 1: Quantitative perturbation recovery performance under four-direction external pushes. Data from paper Table I.

ADP achieves the best performance across all metrics: $J_{80}$ of 115.5 N·s, a 16.7% improvement over AMP (99.0); direction-averaged success rate of 91.4%, an 18 percentage point improvement over AMP (73.4%); recovery time reduced from 4.76s to 2.48s (47.9% reduction); velocity error from 1.30 to 0.84 (35.4% reduction). The comparison with the Dynamics Reward baseline isolates the effect of the adversarial formulation—distribution-level adversarial regularization is more effective than point-wise feature matching on the same feature set.

Figure 2: Time-series perturbation recovery under a lateral push. ADP rapidly attenuates transient velocity and attitude errors within approximately 1-1.5 seconds.

Representation Sensitivity Analysis

Representation	Peak fold change ↑	50% rise time (ms) ↓	Post-push AUC ↑	Pre/post sep. AUC ↑
Kinematic	4.3×	160	2.2	0.98
Dynamics	6.0×	20	3.9	1.00

Table 2: Perturbation sensitivity of dynamics-feature vs kinematic-feature representations. Data from paper Table II.

The dynamics representation reaches 6.0x baseline deviation within 20ms, while the kinematic representation takes 160ms to reach 4.3x. This quantitatively validates the core hypothesis: impulses directly excite floating-base momentum and contact forces before being expressed through joint-level kinematics.

Figure 3: ADP framework overview—(a) SRBD trajectory optimization generates reference trajectories and extracts dynamics features; (b) discriminator evaluates policy windows during training.

Ablation Study

Method	Success Rate (%) ↑	Vel. Error (m/s) ↓	Dyn. Dist. ↓	Recovery Time (s) ↓
ADP	91.4	0.84	1.13	2.48
ADP w/o Momentum	45.3	1.92	3.04	6.74
ADP w/o Contact Force	82.0	1.06	1.51	3.32
ADP w/o Contact Indicator	30.5	2.36	3.82	8.82

Table 3: Dynamics feature component ablation results. Data from paper Table III.

The ablation reveals each component's contribution: removing centroidal angular momentum drops success rate from 91.4% to 45.3% and increases recovery time from 2.48s to 6.74s; removing contact force moderately drops success to 82.0%; removing the binary contact indicator causes the largest degradation—success drops to 30.5% with recovery time of 8.82s. This indicates contact timing is the most critical signal for alignment with the reference distribution. For window length, $K=8$ achieves the best trade-off (91.4% success), while $K=1$ yields only 28.1% and $K=16$ degrades to 33.6%.

Figure 4: Qualitative hardware demonstration of ADP on the real robot under repeated external pushes.

---

Limitations

1. No Explicit Joint-Level Motion Naturalness (author-stated): ADP does not explicitly enforce joint-level motion naturalness since it replaces kinematic style features with dynamics features. Future work will investigate combining dynamics and kinematic priors to improve both robustness and naturalness.

2. Baseline Comparison Fairness (author-stated): ADP is compared against an AMP baseline using the same TO-derived reference source for fairness, rather than against a high-quality mocap-based AMP baseline. Characterizing the gap between TO-derived and high-quality mocap-based kinematic priors is left to future work.

3. Limited Sim-to-Real Validation: While qualitative hardware demonstrations are provided, quantitative evaluation is conducted only in simulation. Real-world success rates, recovery times, and other metrics are not reported, and the hardware demo only shows lateral push recovery.

---

Conclusion and Future Directions

ADP's core contribution is shifting adversarial regularization from the kinematic motion-style space to the dynamics feature space, directly regularizing physical quantities relevant to perturbation recovery—centroidal momentum, contact forces, and contact timing. This shift enables the policy to recover faster toward the reference dynamics distribution after perturbations, rather than merely imitating kinematic appearance.

The representation sensitivity analysis provides strong theoretical support: dynamics features respond to perturbations within 20ms (6.0x deviation), far faster than kinematic features' 160ms (4.3x), because impulses first excite floating-base momentum and contact forces. The ablation study further reveals that the contact indicator and centroidal momentum are the most critical feature components, while the $K=8$ temporal window provides the best trade-off between local temporal context and training stability.

The "TO-derived reference + dynamics features + adversarial distribution matching" paradigm established by ADP offers a new approach for incorporating physics-based prior knowledge into learning-based locomotion control. Combining it with kinematic priors and adapting to real mocap data are valuable future directions.

"External impulses directly excite floating-base momentum and contact forces before being expressed through joint-level kinematics—therefore adversarial regularization in the dynamics feature space exposes perturbation transients earlier and more strongly than in the kinematic feature space."