WARL: Wrench-Augmented Reinforcement Learning for Task-Agnostic Learning in Legged Robots

1. Overview: Wrench-Augmented RL for Task-Agnostic Legged Robot Learning

WARL (Wrench-Augmented Reinforcement Learning), proposed by Keita Yoneda, Kento Kawaharazuka, and Kei Okada at the University of Tokyo in July 2026, addresses the limited exploration capability of actions confined to joint space in legged robot RL by introducing a wrench (force and torque) into the action space. The method combines wrench-guided exploration with a success rate-based curriculum mechanism to expand exploration capabilities in early learning, with the ultimate goal of acquiring behaviors based solely on joint control. Experiments on a quadruped robot demonstrate that WARL learns robustly across diverse terrains and motor tasks without terrain-specific reward adjustments or complex curriculum designs. An ablation study verifies the effectiveness of the Switching Curriculum that gradually eliminates the wrench. The study also shows that introducing a wrench can encourage behaviors that do not sufficiently exploit the robot's physical embodiment—designing wrench exploration consistent with the robot's physical structure is a critical future challenge.

2. Core Problem: Limitations of Joint-Space Exploration

Legged robot RL has achieved high motor performance but is constrained by the limited exploration capability of joint-space actions. Conventional frameworks involve numerous heuristics in learning environment and reward design for specific tasks, making new task design costly. For example, traversing uneven terrain with large steps typically uses a heuristic curriculum that gradually increases step height; even without terrain curricula, task-specific meticulous reward design (e.g., fine-grained reward shaping for jumping) is often necessary. These methods succeed on specific tasks but lack versatility across a wide range. The root cause: joint-space actions have limited exploration range, making it hard to discover motions requiring large center-of-mass displacement or instantaneous force impulses.

3. Background: Action Space Design and Spatial Exploration

The paper analyzes exploration capability differences from an action space design perspective. With joint-space actions only, the policy outputs $\boldsymbol{q}^{\text{ref}}_t = \pi_j(\boldsymbol{s}_t)$, and exploration is confined to a small neighborhood around joint angles. Introducing wrench extends the action space:

$$\boldsymbol{a}_t = (\boldsymbol{q}^{\text{ref}}_t, \alpha\, \boldsymbol{w}_t), \quad \boldsymbol{q}^{\text{ref}}_t = \pi_j(\boldsymbol{s}_t), \quad \boldsymbol{w}_t = \pi_w(\boldsymbol{s}_t)$$

where $\boldsymbol{w}_t$ is the force/torque output of wrench policy $\pi_w$, and $\alpha \in [0,1]$ is the curriculum decay coefficient. The wrench provides direct force/torque application capability, greatly expanding spatial exploration. The environment executes:

$$\boldsymbol{s}_{t+1}, r_t = \text{Step}(\boldsymbol{q}^{\text{ref}}_t, \alpha\, \boldsymbol{w}_t)$$

4. Method: WARL Framework

4.1 Overview

WARL alternates between two phases: (i) Joint+Wrench learning, where both wrench policy $\pi_w$ and joint policy $\pi_j$ are optimized; (ii) Joint-only adaptation, where motion is reproduced by $\pi_j$ under attenuated previously saved wrenches. The Switching Curriculum enables efficient spatial exploration early and gradually removes wrench dependency through a decaying curriculum coefficient $\alpha$. A Teacher-Student framework is adopted, using privileged observations during training removed via student distillation.

4.2 Switching Curriculum

Initialize $\pi_w, \pi_j$, $\alpha=1$. While $\alpha > \epsilon$: Joint+Wrench learning until success rate $\zeta \geq \gamma$ or change $|\Delta\zeta| \leq \delta$; freeze $\pi_w$, $\alpha \leftarrow \sigma\alpha$; Joint-only adaptation using saved wrench sequences $\mathcal{W}_{\text{saved}}$. The decay sequence is:

$$\alpha_k = \sigma^k \alpha_0, \quad \alpha_0 = 1, \quad k = 0, 1, 2, \dots$$

When $\alpha \to 0$ the wrench is fully removed and the policy degenerates to pure joint control $\pi_j$.

4.3 Saving Wrench Sequences

During the Joint+Wrench phase, wrench sequences from successful episodes are saved for use in the Joint-only adaptation phase: $\boldsymbol{w}_t = \mathcal{W}_{\text{saved}}(t)$. This lets the joint policy learn to reproduce motion under gradually decaying external force assistance.

4.4 Teacher-Student Learning

Training uses privileged observations (e.g., true terrain height maps), removed through a student distillation process so the deployed policy relies only on proprioception and onboard sensing. The distillation objective is:

$$\min_{\theta_S} \mathbb{E}\left[\left\| \pi_S(\boldsymbol{s}_t^{\text{onboard}}; \theta_S) - \pi_T(\boldsymbol{s}_t^{\text{priv}}) \right\|_2^2\right]$$

where $\pi_T$ is the privileged teacher policy, $\pi_S$ the student, $\boldsymbol{s}^{\text{priv}}$ includes privileged observations and $\boldsymbol{s}^{\text{onboard}}$ only onboard sensing.

5. Experimental Setup and Results

5.1 Tasks and Robots

Trained on KLEIYN and Unitree Go2 quadrupeds across six tasks (four goal-reaching + two dynamic), with no robot-specific tuning other than changing standing height $z^*$. Goal-reaching: FlatGround, HurdleJump, IslandTraverse, RoughTerrain; Dynamic: FlipBack, BipedWalk.

Category	Task	Challenge
Goal-reaching	FlatGround	Flat walking baseline
Goal-reaching	HurdleJump	Hurdle crossing
Goal-reaching	IslandTraverse	Island gap crossing
Goal-reaching	RoughTerrain	Rough terrain
Dynamic	FlipBack	Backflip
Dynamic	BipedWalk	Bipedal walking

5.2 Simulation Learning Results

Both robots successfully learned diverse goal-reaching and dynamic motion tasks. Learning curves show overall reward improvement, with temporary drops from curriculum-driven wrench decay followed by recovery through adaptive learning. In IslandTraverse, the robot acquired a motion that directly jumps over the 1.4m gap without using intermediate footholds—likely because wrench-based exploration dominated learning, and motions utilizing footholds were not discovered. At iteration 500 of HurdleJump, the learned wrench and joint motion show the hurdle-crossing motion is largely generated by the learned wrench, while coordinated leg motions lift the feet to clear the hurdle.

5.3 Ablation Study

Comparing four conditions on HurdleJump: (1) WARL, (2) w/o Switch (omit adaptation, always explore), (3) w/o Wrench Reward (remove wrench penalty), (4) Baseline (PPO + joint control only). WARL showed the most stable success rate improvement; w/o Switch's policy depended on wrench and success rate decreased in later curriculum stages; Baseline had insufficient exploration. Success rate is defined as:

$$\zeta = \frac{\#\text{success episodes}}{\#\text{total episodes}}, \quad \text{switch condition: } \zeta \geq \gamma \text{ or } |\Delta\zeta| \leq \delta$$

6. Discussion

Advantages of wrench exploration: accelerated learning (higher exploration capability shortens target motion discovery time); reduced heuristic learning design (broader exploration reduces reliance on fine reward tuning and step-by-step curricula). The exploration improvement can be approximated as:

$$P_{\text{find}}(\text{wrench}) \geq P_{\text{find}}(\text{joint}), \quad \Delta t_{\text{learn}} \propto \frac{1}{|\mathcal{E}|}$$

Challenges: introducing a wrench may encourage behaviors that do not sufficiently exploit physical embodiment (e.g., jumping over gaps directly rather than using footholds). Framework improvement: using the same reward for wrench and joint policies is simple but their roles are fundamentally different—the wrench policy primarily facilitates joint policy learning and should evaluate how effectively it promotes autonomous achievement by the joint policy. The ideal wrench policy optimizes:

$$\pi_w^* = \arg\max_{\pi_w} \mathbb{E}\left[\Delta \zeta_j \mid \pi_j, \pi_w\right]$$

Ablation Condition	Description	Result
WARL (full)	Switching curriculum + wrench reward	Most stable improvement
w/o Switch	Omit adaptation phase	Later success rate drops (3/10 success)
w/o Wrench Reward	Remove wrench penalty	Excessive wrench use
Baseline	PPO + joint control only	Insufficient exploration

7. Limitations

• Insufficient embodiment exploitation: Introducing a wrench may encourage behaviors bypassing the robot's physical structure (e.g., jumping over gaps without footholds), making motions less natural.
• Shared wrench/joint reward: Using the same reward for both despite different roles may lead the wrench policy to ignore embodiment.
• Single wrench decay strategy: Only linear decay $\alpha \leftarrow \sigma\alpha$ is used; smarter decay or imitation learning alternatives deserve exploration.
• Limited real-robot transfer: Although the real robot policy is executed in MuJoCo, full sim-to-real robustness needs more experiments.
• Computational overhead: Maintaining both wrench and joint policies during training increases cost.

8. Conclusion

WARL is a framework that enhances legged robot exploration capability by introducing a wrench into the action space, combined with a Switching Curriculum that gradually removes wrench dependency to ultimately acquire a joint-only control policy. On quadrupeds KLEIYN and Go2, it robustly learned across six tasks (goal-reaching and dynamic motion) without task-specific tuning, demonstrating task-agnostic capability. Ablations verify the effectiveness of the Switching Curriculum and wrench reward. The discussion reveals the acceleration and design-reduction advantages of wrench exploration, and the challenge of insufficient embodiment exploitation. The core insight: use external force (wrench) to expand exploration and discover target motions, then gradually withdraw the force through curriculum so the joint policy learns to reproduce autonomously—exploration enhancement and embodiment preservation are the two ends that wrench-augmented learning must balance.

flowchart TD
    A["Initialize π_w, π_j, α=1"] --> B{"α > ε?"}
    B -->|yes| C["Joint+Wrench Learning Phase"]
    C --> D["q_ref = π_j(s), w = π_w(s)"]
    D --> E["Step(q_ref, α·w)"]
    E --> F["Update π_j, π_w"]
    F --> G["Save successful episode wrench sequences"]
    G --> H{"ζ≥γ or |Δζ|≤δ?"}
    H -->|no| C
    H -->|yes| I["Freeze π_w, α ← σ·α"]
    I --> J["Joint-only Adaptation Phase"]
    J --> K["w = W_saved(t)"]
    K --> L["Step(q_ref, α·w)"]
    L --> M{"ζ≥γ or |Δζ|≤δ?"}
    M -->|no| J
    M -->|yes| B
    B -->|no| N["Output joint-only policy π_j"]