MLM: Learning Multi-task Loco-Manipulation Whole-Body Control for Quadruped Robot with Arm

MLM: Learning Multi-task Loco-Manipulation Whole-Body Control for Quadruped Robot with Arm

Paper: MLM: Learning Multi-task Loco-Manipulation Whole-Body Control for Quadruped Robot with Arm
Authors: Xin Liu, Bida Ma, Chenkun Qi, Yan Ding, Nuo Xu, Zhaxizhuoma, Guorong Zhang, Pengan Chen, Kehui Liu, Zhongjie Jia, Chuyue Guan, Yule Mo, Jiaqi Liu, Feng Gao, Jiangwei Zhong, Bin Zhao, Xuelong Li
Affiliation: School of Mechanical Engineering, Shanghai Jiao Tong University; Shanghai AI Laboratory; Lenovo Corporation; Institute of AI (TeleAI), China Telecom
Link: arXiv:2508.10538

One-Sentence Summary

This paper proposes the MLM reinforcement learning framework, driven by both real-world and simulation data, enabling a quadruped robot with a 6-DOF arm (Go2 + Airbot Play) to perform multi-task whole-body loco-manipulation with a single policy. Core innovations include: a real-world trajectory library with adaptive curriculum sampling to balance multi-task learning, and a Trajectory-Velocity Prediction (TVP) policy network that predicts future targets from historical trajectories, supporting both teleoperation and diffusion policy deployment modes with zero-shot sim-to-real transfer.

Research Background and Motivation

Quadruped robot motion control, especially through reinforcement learning (RL), has gained significant attention and development, enhancing their diverse and impressive capabilities. Integrating a 6-DOF robotic arm onto a quadruped robot offers more possibilities for manipulation — combining locomotion with manipulation capabilities enables a broader range of tasks. However, achieving multiple loco-manipulation tasks with a single policy presents a major challenge for quadruped robots, and balancing multiple tasks remains unresolved in existing methods, requiring an adaptive mechanism.

In simulation-driven paradigms, task-specific assets and rewards demand heavy manual design with poor generalizability. The visual gap for manipulation (texture, color, lighting differences between simulation and reality) also remains challenging. A feasible solution is distilling multiple tasks into multiple trajectories learning — abstracting tasks into trajectories and defining a unified reward, avoiding incorporating task visual data into control, thus guiding quadruped robots to develop coordinated whole-body behaviors under a single RL policy. Real-world multi-task trajectories provide spatial trajectories consistent with physical dynamics and distributed across the robot's workspace, essential for learning whole-body behaviors and generalizing to unseen trajectories — while random spatial curves sampled in simulation lack realistic velocity and acceleration dynamics.

Trajectory acquisition can be achieved through various methods including UMI-like approaches, teleoperation, and exoskeleton systems. Notably, this paper does not collect whole-body trajectory data (distinguishing it from imitation learning), due to challenges in species selection for data collection and the inability to obtain such data from real quadruped robots with arms. Therefore, the UMI-like approach is used to obtain only end-effector 6D manipulation trajectories — a lightweight solution independent of robots.

In deployment scenarios with only historical trajectories (e.g., teleoperation), future trajectory information cannot be used as observation. However, future information is crucial for whole-body task stability — it indicates the next action direction. Therefore, a trajectory prediction mechanism is necessary. This paper introduces Fast-UMI for real-world manipulation trajectories and proposes the MLM framework to address these challenges.

Figure 1: Framework overview. RL trains a whole-body control policy. A trajectory library with adaptive sampling curriculum provides multi-task real-world manipulation trajectories. NAE encodes historical trajectories to predict future targets, supporting teleoperation and automatic DP deployment modes.

Whole-Body Loco-Manipulation Policy Formulation

This work eliminates the requirement for object and environment perception in simulation, accelerating training and avoiding the large sim-to-real visual gap. Since information is partially observable, the manipulation task is modeled as a Partially Observable MDP (POMDP), with the objective of maximizing expected cumulative reward $\mathbb{E}_{\tau(\pi_\theta)}[\sum_{t=0}^{T} \gamma^t r_t]$, where $\gamma = 0.99$ is the discount factor. An asymmetric actor-critic framework is introduced to address the POMDP problem while avoiding two-stage training complexity.

State Space. The actor (policy network) input includes: quadruped proprioception five-frame history $\boldsymbol{o}_t^q \in \mathbb{R}^{30}$ (body angular velocity, gravity vector, leg joint positions and velocities); 6-DOF arm proprioception $\boldsymbol{o}_t^a \in \mathbb{R}^{12}$ (arm joint positions and velocities); last action $\boldsymbol{a}_{t-1} \in \mathbb{R}^{18}$; and end-effector pose trajectory $o_{t-3:t+4}^{tcp} \in \mathbb{R}^{72}$ (TCP poses over past and future steps, each including 3D position and first 6 elements of rotation matrix). The critic additionally accesses privileged state $\boldsymbol{s}_t^p \in \mathbb{R}^{21}$ (body linear velocity, contact forces, external forces and positions) and terrain elevation $\boldsymbol{s}_t^e \in \mathbb{R}^{187}$ (training-only).

Action Space. The whole-body policy outputs 12 leg joint position offsets $\boldsymbol{a}_t^q \in \mathbb{R}^{12}$ and 6 arm joint position offsets $\boldsymbol{a}_t^a \in \mathbb{R}^{6}$, added to default positions as PD control target commands.

Reward Function Design

Maintaining coordinated whole-body behaviors is challenging. Building on gait prior work, adversarial motion prior (AMP) is used to encourage quadruped gait learning via a style reward. The adversarial discriminator is trained concurrently during policy training, with the style reward defined as:

The primary reward $r^g$ is tracking the arm's TCP position and orientation, using the product of position and orientation rewards:

Where position and orientation tracking rewards are computed via exponentially decaying multi-task error concatenation:

$\oplus$ denotes concatenation, $N$ is the total number of tasks, and $\sigma_n^p$, $\sigma_n^o$ are per-task reward factors (updated via curriculum). The constraint reward $r^l$ regulates reasonable actions, energy limits, and hardware constraints, including torque penalty $-\|\boldsymbol{\tau}\|_2$, acceleration penalty $-\|\ddot{\boldsymbol{q}}\|_2$, action smoothness penalty $-\|\boldsymbol{a}_{t-1} - \boldsymbol{a}_t\|_2$, collision penalty $-n_{collision}$, and torque/velocity limit exceedance penalties.

Trajectory-Velocity Prediction Policy Network (TVP)

The TVP policy network consists of an NAE (Normalized Autoencoder), Memory Encoder (ME), Estimator, and low-level network. Historical trajectories are processed by the NAE to obtain embeddings and predict future poses; historical proprioception is encoded by the ME, whose embeddings feed the estimator to predict body linear velocity. Predicted future poses, NAE embeddings, predicted velocity, ME embeddings, current proprioception, and last action are fed into the low-level network to generate joint actions.

Trajectory Prediction. A trajectory encoder $E_{\theta_1}$ extracts features from historical poses, and a decoder $D_{\theta_2}$ predicts the subsequent trajectory sequence:

The NAE reconstruction loss measures the ability to reconstruct future trajectories from embeddings $\hat{z}_t^{tcp}$:

Velocity Estimation. The memory encoder encodes the last five frames of proprioceptive observations, and an estimator predicts body linear velocity through supervised learning:

Velocity estimation is crucial because loco-manipulation tasks involve both long-range movement and short-range local operations — the estimated body linear velocity effectively distinguishes these two task types, enabling more accurate execution. The estimation loss is $\mathcal{L}_{est} = \text{MSE}(\hat{v}_t^p, v_t^p)$.

Low-Level Network. Encoded embeddings and predictions, along with current proprioception and last action, are fed into a low-level network outputting the mean vector $\boldsymbol{\mu}_t^b \in \mathbb{R}^{18}$ of a Gaussian distribution $\boldsymbol{a}_t^b \sim \mathcal{N}(\boldsymbol{\mu}_t^b, \boldsymbol{\sigma}^b)$, optimized by PPO.

flowchart TD
  HT["Historical Trajectory o_t-3:t"] --> NAE["NAE Normalized Autoencoder"]
  NAE -->|Embedding z_hat| DEC["Decoder D"]
  NAE -->|Predicted Future Trajectory| LLN["Low-Level Network"]
  HP["Historical Proprioception 5 frames"] --> ME["Memory Encoder ME"]
  ME -->|Embedding| EST["Velocity Estimator"]
  EST -->|Predicted Velocity v_hat| LLN
  CP["Current Proprioception"] --> LLN
  LA["Last Action"] --> LLN
  LLN -->|Joint Action a_t| ROBOT["Robot Execution"]

Trajectory Generation and Sampling

Training Trajectory Library. A trajectory library containing real-world TCP poses from multiple manipulation tasks is introduced, covering typical tasks such as "clean table," "pour coke," "open container," "pick bear," "pick cup," and "unplug charger," with 200 trajectories per task, plus a "pushing" task. These tasks cover repetitive motions, pick-and-place, rotational opening/closing, and long-distance movements, representing common real-world manipulation tasks. During training, trajectory initial points are set as end-effector poses following episode reset to prevent low-quality early samples.

Adaptive Trajectory Sampling. Task trajectory complexity varies — some involve more position and orientation changes. If trajectories are sampled uniformly across all tasks, the policy may focus on tracking easier trajectories (yielding higher rewards) while performing poorly on harder tasks, getting trapped in local optima. An adaptive sampling and reward factor updating mechanism is proposed. Initially, all tasks are sampled with equal probability, with average tracking rewards recorded per task. When a task's tracking reward changes, its sampling probability is updated accordingly, prioritizing more challenging trajectories:

where $\bar{r}_n^g$ is the average tracking reward for task $n$, and $\lambda_n$ is the reward performance threshold, adjustable to achieve similar reward levels across tasks. To prevent forgetting, a minimum probability is retained for each trajectory.

Per-task pose errors (Euclidean distance and angular error) are computed, and reward factors $\sigma_n$ are updated accordingly:

Orientation error is computed via the trace of the rotation matrix. Reward factors are adaptively updated based on whether average errors meet predefined threshold sets $\{(T_{k,n}, \sigma_{k,n})\}$ — progressively increasing tracking requirements for each task as training progresses.

Figure 2: The robot demonstrates multi-task whole-body loco-manipulation capabilities in the real world, including cleaning papers, picking up trash, storing toys, unplugging chargers, and opening doors.

Experimental Setup

Isaac Gym is used to collect data from 4096 agents for training, with policy updates every 50 simulation steps, completing 10000 iterations in about 8 hours on an RTX 3090Ti. The deployment platform is a 12-DOF Uniree Go2 quadruped with a 6-DOF Airbot Play arm, equipped with a 1-DOF parallel gripper, weighing 19.9kg total. The whole-body policy runs at 50Hz on Go2's Jetson Orin NX. DP and handheld gripper run on an RTX 3090Ti PC, transmitting target trajectories via Ethernet. A Livox Mid-360 LiDAR establishes the world frame.

Simulation Ablation Studies

Four policy networks are compared: TVP (ours), TVP without NAE, single MLP (sMLP, from UMI on Legs baseline), and DeepWBC. Results show TVP achieves the smallest tracking errors across all tasks.

TVP reduces errors by 40-65% compared to sMLP baseline and approximately 85% compared to DeepWBC across all tasks. Key findings: the world frame definition makes the policy insensitive to body motion; continuous trajectory reference targets and prediction mechanisms significantly improve tracking performance. The NAE-predicted future trajectories provide "where to go next" guidance, and without them, tracking rewards are notably lower. TVP's joint torque mean squared derivative (MSD) is lower than sMLP, indicating smoother torque control.

Adaptive Sampling Validation. Uniform sampling cannot effectively balance task performance — rewards for tasks with different pose variations diverge over time, with more varied tasks converging to lower rewards due to insufficient attention. Adaptive sampling maintains similar rewards across tasks, balancing tracking performance. Tasks with frequent trajectory changes (e.g., "open container") see significant accuracy degradation under uniform sampling (1.69 vs 0.74 cm).

Prediction Error Robustness. Injecting increasing Gaussian noise into predicted trajectories (position 1cm to 10cm, orientation 0.05 to 0.5 rad), tracking errors increase with noise but the increase remains acceptable, and tasks are still completed. Even under extreme noise (mean 10cm/0.5rad), manipulation remains functional because the method does not solely rely on predicted trajectories but also on historical ground-truth trajectories for robustness.

Real-World Experiments

Feasibility and effectiveness are validated through teleoperation and DP inference. All tasks use a unified whole-body policy, demonstrating strong cross-task performance and zero-shot sim-to-real transfer.

Figure 3: Multi-computer, multi-thread deployment architecture. The host captures handheld gripper trajectory (100Hz) or runs DP (10Hz), transmitting via Ethernet. Jetson Orin NX handles multi-thread tasks.

Real-world deployment achieves tracking performance comparable to simulation. Teleoperation demonstrates exceptional whole-body coordination: for ground-level tasks (e.g., picking up trash), the quadruped assists the arm by leaning downward; tasks above body height require lifting the body; when targets exceed the workspace, the quadruped leverages locomotion to reposition. The DP inference mode generates target trajectories via a pre-trained DP, with the RL whole-body policy controlling joints for execution, limited primarily by DP inference quality.

Limitations and Future Directions

Limitation 1: DP automatic mode success rate limited. DP automatic execution success rates (80-85%) are lower than teleoperation (98-100%), mainly limited by DP inference quality. The DP's visual encoding and trajectory generation precision directly determine whole-body manipulation execution effectiveness.

Limitation 2: Task trajectories require pre-collection. The trajectory library depends on Fast-UMI handheld device collection of real manipulation trajectories, 200 per task. Adding new tasks requires additional collection, limiting scalability by data collection costs.

Limitation 3: Insufficient support for highly dynamic tasks. The current method's response to highly dynamic tasks is limited by sensor update rates and actuation response capabilities. Faster sensors and better actuators may improve dynamic task performance.

Future work includes: extending to more tasks in new environments via higher-level decision-making (e.g., vision-language models), and improving sensor update rates and actuation response for highly dynamic tasks.

Conclusion

This paper proposes the MLM reinforcement learning framework, leveraging both real-world and simulation data to address multi-task whole-body loco-manipulation control for quadruped robots with arms. Core innovations include: a real-world trajectory library with adaptive curriculum sampling balancing multi-task learning — prioritizing challenging trajectories via sigmoid probability updates to avoid local optima; a Trajectory-Velocity Prediction (TVP) policy network predicting future targets and body linear velocity from historical trajectories, supporting teleoperation deployment with only historical information; and world-frame trajectory tracking avoiding body motion interference. In simulation ablations, TVP reduces errors by 40-65% vs sMLP baseline and ~85% vs DeepWBC, with variance reduced from 0.133 to 0.050. Real-world experiments achieve 98-100% teleoperation success and 80-85% DP automatic success, demonstrating exceptional whole-body coordination and zero-shot sim-to-real transfer. This work provides a unified policy framework for multi-task manipulation of armed quadruped robots, laying the foundation for future task scope expansion via vision-language models.