Multi-Gait Learning for Humanoid Robots Using Reinforcement Learning with Selective Adversarial Motion Prior

1. Background and Motivation

Humanoid robots hold promise for operating in environments designed for humans, but realizing this potential requires the ability to execute multiple locomotion behaviors adaptively. Practical deployment demands not only walking for efficient traversal but also running for rapid movement, jumping for gap crossing or obstacle negotiation, stair climbing for multi-level navigation, and specialized gaits such as goose-stepping for structured or ceremonial scenarios. Each gait imposes fundamentally different requirements: walking emphasizes energy efficiency and steady balance, running requires rapid force generation and aerial phase management, jumping demands maximal vertical impulse in a short time window, stair climbing needs precise foot placement and clearance control, and goose-stepping requires sustained leg extension with strict posture constraints.

The central challenge is not merely acquiring each gait in isolation but learning them within a unified framework sharing policy architecture, observation space, and reward structure. RL has emerged as a compelling approach for learning complex motor skills, but extending RL to multiple gaits introduces difficulties: the policy must reconcile conflicting dynamics across behaviors, the reward function must remain effective across gaits with vastly different motion characteristics, and the observation space must encode sufficient information for all behaviors. Adversarial Motion Prior (AMP) regularizes the policy toward demonstrated human-like behavior through a discriminator, but existing work applies AMP uniformly without exploring whether it is always beneficial.

The core insight is that AMP's effectiveness depends critically on the target gait's nature. AMP excels at reinforcing periodic, repetitive motion patterns with moderate joint amplitudes (walking, goose-stepping, stair climbing), but for gaits requiring large joint excursions and explosive dynamics (running, jumping), AMP's regularization over-constrains the motion. The selective AMP strategy applies AMP where it helps and omits it where it hinders.

2. Core Contributions

Three core contributions: (1) A unified RL framework learning five practical gaits—walking, goose-stepping, running, jumping, and stair climbing—using a consistent policy architecture and reward function based on sinusoidal joint reference tracking. All gaits share the same observation space, network structure, and reward formulation, differing only in reference trajectory parameters and reward weights. (2) A selective AMP strategy applying AMP to periodic, stability-critical gaits while omitting it for highly dynamic, agile gaits. (3) Full real-robot validation of all five gaits with quantitative comparisons including tracking error, convergence speed, success/fall rate, and AMP versus non-AMP performance.

Figure 1: Multi-gait learning pipeline overview. Policies trained in simulation with or without AMP depending on gait type, then deployed on the physical robot.

3. RL Formulation

Each gait learning task is formulated as an MDP defined by $(\mathcal{S}, \mathcal{A}, r, \gamma, \rho_0)$. All five gaits share the same MDP structure, differing only in reference trajectory generation, reward weight parameters, and AMP configuration.

State Space: The policy receives a 50-dimensional observation vector $\mathbf{o}_t \in \mathbb{R}^{50}$ at each timestep, including base linear velocity (scaled 2.0), velocity commands, phase encoding $[\sin(2\pi\phi_t), \cos(2\pi\phi_t)]$, base angular velocity (scaled 0.25), projected gravity vector, joint position error, joint velocities (scaled 0.05), and previous action. The phase variable updates as:

The policy receives $N=21$ consecutive observation frames, stacked into a $50 \times 21 = 1050$-dimensional input for implicit velocity and acceleration estimation.

Action Space: The policy outputs a 12-dimensional action vector corresponding to target joint position offsets for 12 actuated DOFs. The target joint position is computed as:

where $k_{\text{action}}$ is the action scale factor (0.25 for walking/stair climbing, 0.3 for running, 0.4 for goose-stepping) and $k_{\text{gain}} = \tau_{\text{max}}/K_P$ is a per-joint gain. The target is sent to a PD controller:

The policy executes at 50Hz while the PD controller runs at 200Hz. An exponential moving average filter smooths actions, with 5-10 timestep random action delay simulating real-world latency.

4. Reference Trajectory Design

For periodic gaits (walking, goose-stepping, running, stair climbing), the reference is sinusoidal. The phase signal $s = \sin(2\pi\phi_t)$, left hip reference is $q_{\text{ref}}^{\text{L,hip}} = -s \cdot \mathds{1}_{s<0}$, and left knee reference is:

where $\sigma$ is the target joint position scale (0.26 for walking and running, 0.28 for goose-stepping), with the right leg mirroring with opposite sign when $s>0$. Each gait modulates through its $(T_{\text{cycle}}, \text{stance\_ratio})$ pair: walking (0.8s, 0.6), goose-stepping (0.7s, 0.45), running (0.4s, 0.35), stair climbing (0.7s, 0.6).

Jumping uses a 4.0s piecewise cycle trajectory with five phases: squat ($\phi<0.30$, $\sin^2$ knee flexion), takeoff (0.30-0.42, rapid extension), flight (0.42-0.48, hold extension), landing (0.48-0.75, impact absorption and recovery), and stand ($\phi \geq 0.75$, default pose). A curriculum progressively increases squat depth over 2000-10000 PPO iterations.

5. Unified Reward Function

All five gaits share a unified reward formulation, with total reward as a weighted sum:

Reward terms are organized into four categories: (1) Joint tracking reward using a dual-exponential sum for hip and knee joints:

where $e_{ij} = q_{ij} - q_{\text{ref},ij}$ is the joint error. (2) Base balance and posture reward including orientation, height, and velocity tracking. (3) Motion smoothness reward penalizing action rate, joint velocity, and torque. (4) Safety and gait-specific rewards with fall termination (base contact or roll $>0.8$ rad, pitch $>1.0$ rad), and jumping using jump height targeting ($w=150.0$, Gaussian centered at 0.275m) and vertical takeoff velocity ($w=120.0$, piecewise nonlinear).

The key design principle: all five gaits use the same reward formula—diversity arises from reference specification, weight tuning, and gait-specific terms rather than structural differences.

6. Selective AMP Strategy

AMP introduces a discriminator $D_\phi$ trained to distinguish expert motion clips from policy-generated state transitions. The discriminator outputs a scalar logit $D(s_t, s_{t+1})$ converted to a reward signal:

This reward is linearly interpolated with the task reward:

where $\alpha$ is the per-gait AMP reward magnitude coefficient and $\beta \in [0,1]$ controls the balance. The discriminator is trained with binary cross-entropy loss plus gradient penalty:

with gradient penalty coefficient $\lambda=10$.

Selective configuration: walking and stair climbing use $\alpha=0.3, \beta=0.8$; goose-stepping uses $\alpha=0.6, \beta=0.7$ (higher $\alpha$ to enforce straight-leg style); running and jumping use $\alpha=0.0, \beta=1.0$ (AMP fully disabled), with jumping additionally disabling symmetry loss to allow asymmetric force generation during takeoff.

7. Training Pipeline and Domain Randomization

Trained using PPO in Isaac Gym with 4096 parallel environments. The actor network uses an MLP (hidden layers [512,256,128], ELU activation), with PPO running 5 learning epochs, 4 mini-batches, learning rate $5 \times 10^{-4}$, discount factor $\gamma=0.99$, GAE parameter $\lambda=0.95$.

Extensive domain randomization bridges the sim-to-real gap: friction [0.2,1.5], base mass perturbation [-2,+5]kg, link inertia scaling [0.5,1.8], center of mass displacement [-0.1,0.1]m, motor strength [0.7,1.0], PD gain scaling [0.8,1.2], joint friction and damping, 5-10 timestep action delay, and external disturbance impulses (max linear velocity 2.0m/s).

8. Zero-Shot Sim-to-Real Transfer

Trained policies are deployed on the physical humanoid robot using zero-shot transfer—no fine-tuning on real hardware. The pipeline: (1) Policy exported to ONNX format (including observation normalizer), loaded via ONNX Runtime on the onboard computer; (2) Policy runs onboard at 50Hz, receiving proprioceptive observations from an external IMU and joint encoders—since base linear velocity is not directly measurable on hardware, corresponding channels are set to zero, relying on observation history to implicitly capture velocity; (3) PD controller computes joint torques at 500Hz using policy output target positions, with a 50Hz cutoff first-order low-pass filter smoothing actions.

9. Overall Performance of Five Gaits

Under a unified policy structure, shared observation design, and common reward organization, the framework learns and deploys five locomotion modes. The table summarizes quantitative performance:

Goose-stepping achieves the highest 97% success rate and lowest 0.94 tracking error with AMP assistance, effectively reinforcing the distinctive high leg lift and straight-knee swing style. Stair climbing converges slowest (5200 steps) but has the lowest training error, reflecting terrain curriculum effects. Jumping has the lowest success rate (83%) and worst posture stability (8.6), reflecting the inherent difficulty of explosive motion.

Figure 2: Representative real-robot image sequences for five learned gaits: walking, stair climbing, goose-stepping, jumping, and running.

10. AMP Effect Analysis

AMP is most effective for gaits with clear periodic structure and stability-first objectives, but can become restrictive for highly dynamic motions. Using goose-stepping (with AMP) and jumping (without AMP) as representative cases: under AMP, goose-stepping exhibits rapid total reward increase, low training error, and prompt failure rate decay—AMP's main contribution is not merely higher terminal return but reinforcing periodic structure and style consistency.

Figure 3: AMP comparison training curves. (a) Goose-stepping with AMP. (b) Jumping without AMP. Each panel reports total reward, training error, and failure rate.

For running, a strong AMP prior would not necessarily prevent locomotion but would increase behavioral conservatism, biasing toward a smooth fast walk rather than a genuinely explosive gait with clear aerial phases. Jumping under the non-AMP setting still achieves sustained reward growth and low failure rate with steadily improving joint tracking—demonstrating that task rewards alone suffice for effective takeoff, flight, and landing recovery when expressive exploration is preserved.

11. Method Pipeline

12. Real-Robot Robustness

Real-robot robustness is assessed from three perspectives: long-horizon continuity, consistency across repeated trials, and recovery under mild external disturbances. Walking and goose-stepping emphasize rhythm consistency and controlled torso oscillation; stair climbing emphasizes foot clearance, step placement, and posture recovery after stair transitions; running emphasizes stability during rapid short-stance switching; jumping emphasizes synchronized takeoff, aerial posture control, landing absorption, and return to the next motion cycle. Deployments demonstrate that learned policies exhibit the continuity and repeatability required for physical execution, not just visually plausible simulation motions.

13. Limitations

Limitations include: (1) Jumping has the lowest success rate (83%) and worst posture stability (8.6), reflecting inherent explosive motion difficulty and landing recovery challenges; (2) All gaits validated on a 12-DOF lower-body platform, not extended to full-body humanoid; (3) Base linear velocity is not directly measurable on hardware, set to zero with reliance on observation history for implicit estimation, potentially introducing errors in high-speed motion; (4) Training configuration requires manual per-gait parameter tuning (e.g., $k_{\text{action}}$, $\sigma$, $T_{\text{cycle}}$), with limited automation.

14. Conclusion

This paper presents a unified RL framework for multi-gait humanoid locomotion with a selective AMP strategy. Under a common policy architecture and reward formulation, the method learns five representative gaits—walking, goose-stepping, stair climbing, running, and jumping—and transfers them to a physical humanoid platform. Results show that motion priors benefit periodic, stability-oriented gaits but should be weakened or removed for highly dynamic behaviors. The framework provides a practical and reproducible baseline for multi-gait learning through unified RL formulation, gait-dependent prior selection, and real-robot validation. The study suggests that practical humanoid mobility depends not only on stronger policies but also on principled methods for organizing diverse behaviors within a shared learning framework.