Autonomous Legged Mobile Manipulation for Lunar Surface Operations via Constrained Reinforcement Learning

Autonomous Legged Mobile Manipulation for Lunar Surface Operations via Constrained Reinforcement Learning

Paper: Autonomous Legged Mobile Manipulation for Lunar Surface Operations via Constrained Reinforcement Learning
Authors: Alvaro Belmonte-Baeza, Miguel Cazorla, Gabriel J. García, Carlos J. Pérez-Del-Pulgar, Jorge Pomares
Affiliation: Department of Computer Sciences and AI / Department of Physics, Systems Engineering, and Signal Theory, University of Alicante, Spain; Department of Systems and Automatics Engineering, University of Málaga, Spain
Link: arXiv:2510.12684

One-Sentence Summary

This paper proposes a constrained reinforcement learning (CRL) framework for lunar surface operations, enabling a quadrupedal mobile manipulator (ANYmal + Interbotix WX250s 6-DOF arm) to achieve whole-body coordinated 6D end-effector pose tracking in 1/6 gravity and rough lunar terrain via the Constraints-as-Terminations (CaT) mechanism, achieving 4cm positional accuracy and 8.1° orientation accuracy while satisfying safety constraints including collision avoidance, dynamic stability, and power efficiency, with emergent jumping gaits that exploit low gravity to reduce energy consumption.

Research Background and Motivation

Robotic systems are indispensable for modern planetary exploration. With the advancement of NASA's Artemis and China's Chang'e programs, establishing permanent lunar bases and utilizing in-situ resources requires robots to perform complex mobility and manipulation tasks in extremely harsh environments. Lunar surface challenges include: approximately 1/6 Earth gravity, highly irregular terrain, loose regolith, and the absence of atmosphere — conditions where small control instabilities can lead to catastrophic failure, while limited communication bandwidth and time delays preclude direct human supervision.

Traditional wheeled rovers degrade severely on steep slopes, uneven surfaces, and soft regolith, while legged robots offer superior terrain adaptability and accessibility. Research prototypes including SpaceClimber, ATHLETE, and SpaceBok have explored various aspects of legged mobility for space exploration, from statically stable climbing to highly dynamic gaits optimized for reduced-gravity locomotion. These developments demonstrate that legged systems can extend planetary mission reach into areas previously inaccessible to wheeled rovers.

However, mounting a manipulator arm on a quadruped robot introduces a high-dimensional control problem — requiring real-time whole-body reasoning about balance, forces, and safety constraints. Coordinating the locomotion and manipulation subsystems demands that the policy simultaneously handle balance, contact forces, joint limits, and other coupled factors. Reinforcement learning (RL) has demonstrated powerful capabilities in quadrupedal and humanoid locomotion, dexterous manipulation, and integrated navigation, with multiple works beginning to study whole-body loco-manipulation — jointly learning locomotion and manipulation for coordinated tasks. However, these frameworks typically focus on terrestrial environments and lack formal safety guarantee mechanisms.

Constrained reinforcement learning (CRL) offers a principled framework combining RL's adaptive power with explicit safety guarantees. Unlike conventional RL that only optimizes a reward function, CRL incorporates task-specific constraints into the learning process that the policy must satisfy during both training and execution. This paradigm has shown promise for improving safety and robustness in terrestrial legged locomotion and manipulation tasks, but has not been applied to the planetary robotics domain. This paper is the first to apply CRL to legged mobile manipulators operating in lunar environments, bridging the gap between adaptive learning-based control and the safety reliability required for planetary robotics.

Constrained RL and the CaT Framework

Standard RL models sequential decision-making as a Markov Decision Process (MDP), defined by the tuple $(\mathcal{S}, \mathcal{A}, \mathcal{R}, \mathcal{P})$, with the goal of maximizing expected cumulative reward:

where $\gamma \in [0,1)$ is the discount factor weighing short-term versus long-term rewards. The Constrained MDP (CMDP) extends the MDP by introducing a constraint set $\mathcal{C}$ with associated limits $\mathcal{L}$, where each constraint $c_i$ maps a state transition to a cost, and the discounted cumulative cost must remain within limit $l_i$:

The Constraints-as-Terminations (CaT) framework aims to reduce the implementation complexity of constrained RL by introducing stochastic terminations to enforce desired constraints. Any constraint violation implies a probability of terminating future rewards from that timestep onward. CaT reformulates the learning objective as:

where $\delta_t \in [0,1]$ is a random variable representing whether the episode is terminated at timestep $t$, with its value depending on constraint violations:

Here $c_i^+ = \max(0, c_i(s,a))$ is the violation of constraint $c_i$, $c_i^{max}$ is a moving average of the maximum constraint violation updated empirically during training, and the clipping operation bounds the ratio to $[0,1]$. $p_i^{max}$ is the maximum termination probability for that constraint. This mechanism allows state-of-the-art RL algorithms like PPO to easily integrate constrained learning by simply weighting rewards by a factor of $(1-\delta)$ and updating termination flags, dramatically lowering the implementation barrier.

Figure 1: Method overview. The policy receives task-space end-effector pose commands, outputs whole-body joint position targets executed by PD controllers; the constraint module monitors safety conditions and produces stochastic termination signals.

Whole-Body Legged Locomanipulation Problem Formulation

End-effector Pose Command. The command is defined as a desired end-effector pose $p_{ee} \in \mathbb{SE}(3)$ in task space. During training, a random 6D pose is sampled within a vicinity of the robot, such that it may need to traverse some terrain but won't need to move very far. This policy is not aimed at long-distance locomotion but is deployed when the robot is in a local area of interest for inspection or sample retrieval. The task-space definition is crucial — previous works typically define the tracking problem in the body frame to simplify learning, but this causes the end-effector position to shift when the body displaces, unsuitable for applications requiring precise manipulation or inspection of specific task-space areas.

Action Space. The action is the desired joint positions for both the arm and quadrupedal legs. The arm has 6 DOF, the quadruped has 4 legs with 3 joints each, totaling $a_t \in \mathbb{R}^{18}$. The raw policy action is multiplied by a scaling factor $\sigma$ and added to the default joint position:

The target positions are sent to two separate low-level PD controllers for the arm and legs.

Observation Space. Includes proprioceptive information (projected gravity vector $\phi_b$, body linear and angular velocities $v_b, \omega_b$, joint positions and velocities $q, \dot{q}$, previous action $a_{t-1}$, four-dimensional boolean contact state vector for quadruped feet) and exteroceptive perception (chin-mounted LiDAR height scan $h$ providing local terrain elevation), plus the current desired end-effector pose $p_{ee}^*$. The key design is transforming the task-space target pose to the body frame before feeding it to the policy — preserving the advantages of local information for policy learning while maintaining the task-space definition, enabling a mission planner to provide task-space targets while the policy learns in local coordinates.

Reward Function Design

Mobile manipulation is inherently a two-stage hierarchical problem: first using mobility to approach the target until the end-effector pose becomes reachable, then maintaining a stable base position for the manipulator to reach the target. Literature typically handles these tasks separately since each is complex on its own, and combining them introduces additional difficulties. However, separation fails to leverage whole-body coordination (WBC) capabilities — such as positioning the quadruped to help the manipulator reach difficult poses, using the manipulator while the base moves to save operation time, or even helping the base balance on rough terrain. The second challenge is 6D tracking itself — position and orientation errors have different magnitudes, and precisely tracking 3D orientation is especially difficult. This paper proposes a continuous reward formulation that implicitly handles both stages and inherently balances position and orientation tracking.

6D End-Effector Tracking Reward. The position error is the sum of squared norms of the difference between current and desired positions, and the orientation error is the magnitude of the axis-angle representation of the relative quaternion rotation:

Individual rewards are computed via exponential kernels, where smaller errors yield rewards closer to 1:

where $\sigma_{pos}, \sigma_{rot}$ are parameters adjusting the sensitivity of exponential decay. To inherently balance position and orientation tracking (avoiding bias toward one), the pose reward is the product of both:

This way, only simultaneous improvement in both position and orientation can maximize the reward — sacrificing one for the other does not yield higher rewards, encouraging the policy to steadily improve tracking accuracy in both aspects.

Legged Locomanipulation Reward. To leverage quadrupedal mobility, a body-to-target gated reward is designed, encouraging the body to approach the target end-effector position in the XY plane within radius $r$. Considering $d_{base} = \| p_{body,xy} - p_{ee,xy}^* \|$, the base position reward is:

The combined task reward is the product of the pose reward and the gated base reward:

where the gating factor $g = \text{Sigmoid}(k \cdot (d_{base} - r)) \in (0, 1)$ modulates the importance of the base distance reward. The intuition: the robot first moves to within distance $r$ of the target, then obtains the maximum pose reward value. This continuous formulation requires no explicit state machines or thresholds, favoring learning and leading to smooth autonomous transitions between task stages.

Power Minimization Reward. Under the CaT framework, penalty terms (negative rewards) would be reduced by the $(1-\delta)$ coefficient when constraints are violated, paradoxically undermining constraint enforcement — violating constraints would help minimize penalty loss. Therefore, the penalty is flipped into a positive reward — higher rewards when power consumption is low, applying an exponential kernel to the mechanical power of each joint:

where $\omega_{legs}, \omega_{arm}$ are reward weights for each term, $\dot{\mathbf{q}}$ and $\tau$ represent joint velocities and torques for legs and arm, and $\mu$ is a normalization parameter corresponding to average maximum power of each joint group.

Figure 2: Whole-body end-effector pose tracking examples. The legged mobile manipulator demonstrates precise and safe EE pose tracking in different configurations, suitable for sample collection or surface inspection tasks.

Constraint Design

Constraints are divided into two types: soft constraints, whose violation results in probabilistic partial termination of future rewards (allowing brief violations), and hard constraints, whose violation causes immediate termination of cumulative reward (must always be avoided). The conceptual distinction: some situations must be avoided every time (e.g., non-foot body parts colliding with the environment), while others are desirable to avoid but can tolerate brief violations (e.g., exceeding velocity limits).

flowchart LR
  subgraph Soft["Soft Constraints"]
    S1["Joint position/velocity/torque limits"]
    S2["Body linear velocity ≤ 0.25 m/s"]
    S3["Base rotation ≤ 0.3 rad"]
    S4["Foot force distribution std dev"]
  end
  subgraph Hard["Hard Constraints"]
    H1["Non-foot contact force limit"]
    H2["Fall constraint roll/pitch > 90°"]
    H3["Minimum body height"]
    H4["Max foot impact force
(Lunar low-gravity specific)"]
    H5["Max jump height
(Lunar low-gravity specific)"]
  end
  Soft -->|Probabilistic termination| T["CaT stochastic termination δ"]
  Hard -->|Immediate termination| T
  T --> Policy["Policy Learning"]

Soft constraints include: joint position $c_{q_j}$, joint velocity $c_{\dot{q}_j}$, and joint torque $c_{\tau_j}$ limits within operational range and maximum nominal values (avoiding peak torques and velocities); body linear velocity $c_v$ limited to 0.25 m/s; base rotation $c_{rot}$ limited to 0.3 radians to avoid excessive rotation; and foot force distribution standard deviation $c_{f_{std}}$ constraining the maximum standard deviation in the distribution of forces applied by each foot, favoring even mass distribution across feet — crucial in lunar operations where feet might sink into regolith and cause catastrophic failure.

Hard constraints include: non-foot contact force $c_{contact}$ restricting high contact forces on any non-foot body part; fall constraint $c_{fall}$ triggered when body roll or pitch exceeds 90° to prevent falls and damage; minimum body height $c_{h,min}$ preventing the body from hitting the ground. The lunar-specific constraints are particularly critical: $c_{f,max}$ limits the maximum impact force exerted by robot feet, avoiding high-force impacts that might cause the robot to lose ground contact and float to dangerous heights or damage legs causing emergency stops; $c_{h,max}$ limits the maximum body height, allowing some floating and jumping motion leveraging low-gravity conditions but preventing loss of control. These two constraints are specifically designed for the lunar low-gravity environment, reflecting deep understanding of planetary operation peculiarities.

Experimental Setup

The system uses an ANYmal quadruped robot with an Interbotix WX250s 6-DOF manipulator mounted on top. Lunar conditions are simulated: gravity reduced to 1/6 of Earth's, with rough uneven terrain resembling the lunar surface morphology. Each episode samples a target EE position within a cylinder of 1.2m radius and 0.7m height centered at the robot's XY position, with orientation sampled uniformly within ±30° from the default EE orientation. Domain randomization includes: quadruped mass randomized within ±10% of nominal, delayed PD controller with random delay up to 40ms, and Gaussian noise on observations — to robustify the control policy against uncertain environments.

Soft constraints employ a probability curriculum — each constraint has minimum and maximum termination probabilities, increasing linearly as training progresses, reaching maximum at 60% of training time, allowing more exploratory behavior early and becoming more restrictive as training advances and base skills are learned. NVIDIA Isaac Sim serves as the physics simulator, with Isaac Lab as the learning framework. A customized constrained PPO algorithm modifies the RSL-RL library's PPO implementation. The policy network is a 3-layer MLP (512/256/128 neurons, ELU activation), running at 100Hz with PD controllers at 200Hz. Training runs for 10000 iterations with 4096 parallel environments, completing in approximately 5 hours on a single NVIDIA RTX 3090 GPU.

Experimental Results

Testing runs a single episode across 4096 environments. For end-effector pose tracking, the system reports an average position error of 4cm and orientation error of 8.1°, comparable to state-of-the-art terrestrial loco-manipulation works (such as Portela et al. and Ha et al. for whole-body EE pose tracking). The position error density is concentrated between 2-6cm with maximum around 10cm, indicating the policy maintains high-precision tracking in most cases, with larger deviations only at extreme target poses.

Figure 3: Distribution of position and orientation errors for 4096 evaluated samples. Position error density is concentrated at 2-6cm with maximum around 10cm.

Emergent Behaviors. The policy exhibits emergent behaviors exploiting low-gravity conditions — slight jumping gaits appear to minimize quadruped joint power consumption, while using small movements of arms and legs to stabilize during these maneuvers. This adaptive behavior is a natural result of the CRL framework combining task rewards with power efficiency constraints, arising spontaneously without manual design and demonstrating the advantage of learning approaches in discovering environment-specific optimal policies. Under Earth gravity, jumping gaits would be uneconomical due to high power consumption; but under lunar low gravity, jumping can reduce sustained ground contact friction while leveraging low gravity to extend air time, becoming an efficient locomotion mode.

Constraint Satisfaction Analysis. All hard constraints (contact force, fall, minimum height, maximum impact force, maximum height) show 0% violation time, ensuring the policy is always reliable under the most restrictive safety conditions. For soft constraints, the worst case is the body velocity constraint at 0.08% violation time, with an average violation rate of approximately 0.01% across all soft constraints, indicating the policy rarely truly violates constraints while exploring safety boundaries. This result validates the probability curriculum design — lenient early, restrictive later — in guiding the policy to learn safe behaviors.

Limitations and Future Directions

Limitation 1: Simulation-only validation. All experiments are conducted in NVIDIA Isaac Sim simulation, without validation on real lunar-analog terrain or hardware. The sim-to-real transfer gap (e.g., complex contact dynamics of regolith, sensor noise characteristics, actuator delay models) may affect policy deployment effectiveness. While domain randomization partially mitigates this issue, the uncertainty of the real lunar environment far exceeds simulation coverage.

Limitation 2: Insufficient terrain diversity. Testing is limited to uneven terrain, without covering more challenging lunar landforms such as craters or steep slopes. These scenarios impose higher demands on quadrupedal stability and manipulation capability, potentially requiring more complex perception and planning strategies.

Limitation 3: Limited operation range. Target poses are sampled within a 1.2m radius of the robot, making the policy unsuitable for long-distance mobile manipulation. Real lunar missions may require a multi-level system combining navigation planning with local manipulation, where the current policy is only applicable to the local fine manipulation phase.

Future work includes: evaluating the policy on more challenging terrains (craters, steep slopes), conducting real-world experiments in lunar-analog terrains combined with task planning algorithms to validate adaptability, and demonstrating the potential of constrained reinforcement learning in challenging real-world scenarios.

Conclusion

This paper presents the first constrained reinforcement learning framework for legged mobile manipulators operating on the lunar surface. The core innovation lies in unifying whole-body locomotion and manipulation control within a CRL formulation, embedding safety constraints including collision avoidance, dynamic stability, and power management into the learning process via the CaT mechanism. Technical highlights include: the product reward design for inherently balanced position-orientation tracking, gated body distance reward for continuous stage transitions, flipping power penalty to positive reward to adapt to the CaT framework, and lunar-specific maximum impact force and maximum height hard constraints. The 4cm position accuracy and 8.1° orientation accuracy match terrestrial SOTA levels, with zero hard constraint violations and soft constraint violation rates below 0.08%. The emergent jumping gaits exploiting low gravity demonstrate CRL's ability to unify task objectives with environmental adaptation. This work lays the foundation for safe learning-based control in future lunar autonomous manipulation missions, effectively bridging the gap between adaptive learning and the safety reliability required for planetary robotics.