Contact-Guided Exploration for Non-Prehensile Locomanipulation with Multi-Critic RL

Paper Information

Title: Contact-Guided Exploration for Non-Prehensile Locomanipulation with Multi-Critic RL

Authors: Simone Tolomei, Mayank Mittal, Franco Angelini, Manolo Garabini, Paolo Salaris, and Marco Hutter
Date: 2026-08-28
Paper: arXiv 2608.28140
Project page: tolomeis.github.io/contact-guided-exp
Code status: The paper does not provide a public code repository. The training stack is based on Isaac Lab and a modified RSL-RL PPO implementation.

One-Sentence Summary

The paper converts general-purpose grasp candidates into a temporary exploration objective, gives that objective its own critic, and then decays its contribution to the policy gradient so a quadrupedal mobile manipulator first discovers contact and later optimizes stable object transport.

Background and Motivation

Transporting a chair, pushing a box, or opening a dishwasher does not require a hand to enclose an object, but it does require sustained control over a contact state that can disappear without warning. Unlike a rigid grasp, a non-prehensile interaction is governed by unilateral constraints and friction cones. The robot must reason about where force can enter the object, how the base should move while the arm remains within its workspace, and whether the resulting wrench will transport the object or tip it over. A controller that only solves the next instant of contact can still fail because the object slips, rotates, or becomes unreachable.

Model-based controllers have been successful in mobile manipulation because they can express stability, collision, and task constraints explicitly. Yet the contact dynamics that dominate these tasks are discontinuous. To keep an MPC problem tractable, the model often smooths or approximates contact transitions. When the real object does not match the model, or when a planned contact is lost, the controller must replan under time pressure. In cluttered furniture scenarios, precise meshes and friction parameters are not always available.

Reinforcement learning offers a different route. A policy can acquire reactions to slipping and unexpected contact by experiencing many randomized episodes. The obstacle is exploration. A successful transport trajectory requires the end effector to reach a useful point, the base to position itself correctly, the arm to apply a suitable force, and the object to remain stable. Before the policy knows any of this, random actions rarely generate a complete positive reward. The earliest useful signal is usually a penalty for moving too quickly or using too much energy.

This creates a well-known but especially destructive local optimum. The regularizers are dense; task success is sparse. An agent can reduce immediate cost by staying away from the object. It has not misunderstood the objective; it has simply discovered that avoiding contact is the easiest way to avoid penalties. In locomanipulation, this problem is amplified because base motion and arm motion must be coordinated before object motion is even possible.

Prior work has used uniform surface sampling to guide pushing on boxes and cylinders. That works when large flat regions provide reliable force transmission. Chairs, folding chairs, and backrest-free tables are less forgiving. Many uniformly sampled points lie in open space between legs, on thin edges, or in regions that are kinematically awkward for the arm. Exploration guidance must therefore respect object structure without requiring manual contact annotations for every mesh.

The authors exploit an unexpected source of structure: a grasping algorithm. Grasp candidates are optimized to transfer force into an object, even though the final task is not grasping. A point selected on a chair leg, handle, or panel may be useful for hooking, pulling, or pushing. The central question is how to use that prior without making the robot permanently obsessed with touching it.

System Context

The robot is ALMA, a quadrupedal mobile manipulator. The learned controller is hierarchical. Instead of emitting commands for every leg and arm joint, a high-level policy produces four base quantities and six arm joint targets. The base quantities are forward velocity, lateral velocity, yaw rate, and base height. A separately pretrained and frozen locomotion policy tracks those base commands while observing arm positions and emits low-level leg joint targets.

Base height is treated as an active control variable rather than a constant. When the arm reaches downward and forward to hook a low contact point, the shoulder can approach a joint limit and trigger protection. Lowering the base gives the arm a better kinematic configuration. In hardware, the learned policy settles on a base height about 15 centimeters below its standard locomotion height.

Figure 1: Object-centric contact priors guide exploration, and the learned policy uses them for contact-rich chair transport.

Method

Contact candidates come from an existing grasping algorithm. The authors adapt Grasp It Like a Pro 2.0 to process an object mesh and return candidate interaction points. For chair transportation, they take a set of 25 returned points. At every episode reset, one candidate becomes the target point for that episode. This changes the exploration problem from "search the whole object" to "reach and exploit a physically meaningful region." The same object can expose many different candidates over training.

Figure 3: Red markers show candidates generated by the grasping algorithm; one is sampled as the end-effector target in each episode.

The number of candidates is a real design parameter. With too few, the policy overfits a small set of positions and may follow a contact target at the expense of moving the object. With too many, the candidate set approaches uniform surface sampling and loses its structural bias, especially on non-convex furniture. Twenty-five points gives the policy enough variety to avoid memorizing one chair while concentrating exploration near legs and panels.

Box pushing does not use the grasping sampler. The paper follows prior work and samples contact points uniformly over visible surfaces. A box has large, stable faces suitable for pushing. The contrast is important: contact priors are not treated as a dogma, but as a way to compensate for object geometries where uniform sampling is inefficient.

The reward is divided into three groups. The task group contains object velocity toward the goal and object position tracking. The exploration group contains end-effector tracking of the sampled contact point. The regularization group contains base action rate, arm action rate, hook-height penalty, and action-limit penalty. The task group weight is fixed at 0.75. The exploration weight starts at 0.10 and ends at 0.01. The regularization weight rises from 0.15 to 0.24.

Both scheduling changes occur linearly between 5000 and 10000 training steps:

$$w_{\mathrm{exp}}(k)=\begin{cases}0.10,&k\le 5000\\0.10-0.09\frac{k-5000}{5000},&5000<k<10000\\0.01,&k\ge 10000\end{cases}$$

$$w_{\mathrm{reg}}(k)=\begin{cases}0.15,&k\le 5000\\0.15+0.09\frac{k-5000}{5000},&5000<k<10000\\0.24,&k\ge 10000\end{cases}$$

$$w_{\mathrm{task}}=0.75$$

Outside that interval, the weights remain constant. The schedule was tuned on chair transportation, then reused unchanged for box pushing and dishwasher opening. The starting interval gives the policy time to discover contact; the decay prevents the contact-seeking term from dominating once reliable contact is available.

Multi-critic PPO separates value estimation. Instead of adding all reward terms into one scalar and learning a single return, the environment emits a three-dimensional reward vector. A shared value backbone branches into three heads, each estimating the return of one reward group. This is the core distinction from ordinary reward shaping: exploration has its own value estimate, and its influence is adjusted at the advantage level rather than hidden inside a mixed scalar return.

For each value head $V_{\phi_h}$, the temporal-difference error is:

$$\delta_t^h=r_t^h+\gamma V_{\phi_h}(s_{t+1})-V_{\phi_h}(s_t)$$

The corresponding advantage is accumulated over future TD errors:

$$A_t^h=\sum_{k=0}^{\infty}\gamma^k\delta_{t+k}^h$$

The policy update uses the weighted sum of group advantages:

$$A_t=\sum_{h=1}^{3}w_h A_t^h$$

Each value head is trained toward its own empirical return $R_t^h$. The collective value loss is therefore:

$$\mathcal{L}^{\mathrm{VF}}(\phi)=\sum_{h=1}^{3}w_h\left|V_{\phi_h}(s_t)-R_t^h\right|^2$$

The actor still uses the PPO clipped surrogate objective. With probability ratio $\rho_t(\theta)$, the update maximizes:

$$\mathcal{L}^{\mathrm{CLIP}}(\theta)=\hat{\mathbb{E}}_t\left[\min\left(\rho_t(\theta)A_t,\mathrm{clip}\left(\rho_t(\theta),1-\epsilon,1+\epsilon\right)A_t\right)\right]$$

This formulation preserves PPO's stability while making the source of policy improvement explicit. Task advantages do not have to fight exploration advantages inside one critic; the weights say how much each group should influence this particular phase of training.

The hook-height penalty requires rescaling. Its raw form is:

$$r_{\mathrm{hk}}=e^{\max(0,\,0.15-z_{\mathrm{ee}})}-1$$

It is zero when the end-effector height $z_{\mathrm{ee}}$ is at or above 0.15 meters and grows exponentially when the end effector moves below that threshold. Because the raw value is several orders of magnitude smaller than the other reward terms, Table I applies a coefficient of -5000. The large coefficient is a scaling correction, not an independent new behavior term.

The action space is deliberately compact. The high-level action consists of 4 base dimensions and 6 arm joint targets. This keeps the locomotion skill in the frozen low-level policy while allowing the manipulation policy to decide where the base and end effector should go. The final Gaussian action distribution outputs the high-level command, and the low-level controller realizes legged motion.

The actor observation contains proprioception and object information available on hardware: joint states, projected gravity, base velocity, object position, object orientation, and object-to-goal error. All quantities are expressed in the robot base frame. The critic observes privileged quantities that are useful for learning but impractical at deployment: object linear velocity, object angular velocity, and the selected target contact point. This actor-critic split keeps the deployed observation set realistic.

The networks share temporal features. The actor uses an LSTM of hidden size 256 followed by an ELU MLP with layers 256, 128, and 64. The multi-critic also uses a shared LSTM and initial MLP layers. Only the final output layer branches into three value heads. According to the paper, this shared design has negligible additional memory and wall-clock training cost compared with the same training setup.

Training uses high-throughput simulation. Isaac Lab runs 4096 parallel environments. Simulation time is advanced at 0.005 seconds, while control commands are issued every 0.02 seconds. Every episode randomizes robot initial pose, joint state, object mass, and friction. Object friction is sampled from 0.2 to 1.5, object mass from 2 to 4 kilograms, and robot base mass is perturbed by up to 5 kilograms in either direction.

The chair corpus is broad enough to test morphology. Training uses 15 IKEA CAD chairs and 100 procedurally generated chairs. The procedural model varies total height from 0.70 to 1.10 meters, seat height from 0.35 to 0.55 meters, seat width and depth from 0.40 to 0.60 meters each, seat and backrest thickness from 0.02 to 0.05 meters each, and leg cross-section from 0.03 to 0.06 meters. These ranges cover light, tall, wide, and structurally distinct chairs.

The goal is a 2D position relative to the object's initial pose, sampled uniformly from a disk of radius 2 meters. PPO uses a learning rate of 0.0001 for actor and critic, discount factor 0.99, GAE parameter 0.95, clip range 0.2, KL target 0.01, 5 epochs per rollout, and 4 mini-batches per epoch.

Figure 2: Contact sampling, three value heads, weighted advantage mixing, and hierarchical control across three non-prehensile tasks.

Figure 4: With different goal positions around the same object, the policy selects a contact side aligned with the intended motion.

flowchart TD
  Mesh[Object mesh] --> Grasp[Grasp It Like a Pro 2.0]
  Grasp --> CandidateSet[25 contact candidates]
  CandidateSet --> Reset[Sample target contact point]
  Reset --> Actor[High level LSTM actor]
  Actor --> BaseCommand[4D base command]
  Actor --> ArmTarget[6D arm joint target]
  BaseCommand --> Locomotion[Frozen locomotion policy]
  Locomotion --> LegTargets[Leg joint targets]
  ArmTarget --> EndEffector[End effector interaction]
  LegTargets --> BaseMotion[Base motion]
  BaseMotion --> EndEffector
  EndEffector --> TaskReward[Task reward group]
  EndEffector --> ExploreReward[Exploration reward group]
  Actor --> Regularizer[Regularization reward group]
  TaskReward --> TaskHead[Task value head]
  ExploreReward --> ExploreHead[Exploration value head]
  Regularizer --> RegularizerHead[Regularization value head]
  TaskHead --> AdvantageMix[Weighted advantage mixing]
  ExploreHead --> AdvantageMix
  RegularizerHead --> AdvantageMix
  AdvantageMix --> PPO[PPO clipped update]
  PPO --> Schedule[Decay exploration and raise regularization]

Results

Simulation compares four learning formulations. The first is vanilla PPO with a single critic and scalar reward weights. The second is PPO with weight scheduling, where reward-term weights are decayed but value estimation remains scalar. The third is multi-critic PPO with fixed group weights. The fourth is the full method: multi-critic PPO plus weight scheduling. All training runs include the contact-guided exploration reward, so the comparison isolates how the guidance is represented and phased out.

The full method reaches 94.1% success, 4.4% tipover, and 9.2 seconds average completion. PPO with scalar reward scheduling is only slightly faster at 9.1 seconds, but it is much less stable. The success-rate standard deviation over five random seeds is 0.98% for the full method. Fixed-weight multi-critic PPO has 4.2%, scalar scheduling has 1.2%, and vanilla PPO has 9.7%.

The ablations separate two failure modes. Vanilla PPO often avoids contact; on chair transportation its missed-contact rate is 9.1%. Adding a schedule directly to scalar reward terms lowers missed contact to 4.0%, proving that contact guidance helps discovery. However, the single value function still mixes task, exploration, and regularization, so the learned behavior remains unstable. Conversely, fixed multi-critic PPO finds contact but leaves the exploration term active throughout training, causing the policy to prioritize touching the point over maintaining transport stability.

The most blunt result is the variant without any exploration reward: 0% success. Even with 4096 simulated environments and task rewards, the policy does not reliably discover meaningful non-prehensile interaction. This places the contact prior, rather than simulation volume alone, at the center of the method's benefit.

The metrics are also explicit. Success means the object reaches within 0.2 meters of the goal. Missed contact means the object moves less than 0.2 meters. Tipover means the object tilts beyond 35 degrees. The authors state that 35 degrees is a labeling threshold rather than a physical tipping angle for every object, and that trends remain consistent when the threshold varies between 30 and 40 degrees.

Qualitative rollouts show object-aware control. The authors fix robot and object poses and sample goals around the object. The policy consistently chooses a contact point on the side aligned with the desired motion, increasing control authority for pulling. It also positions the mobile base so the manipulator remains in a favorable workspace instead of approaching a singularity. This is not visible in the success rate alone, but it explains why the policy can generalize across goal directions.

Hardware tests use four unseen IKEA objects and 58 trials. The aggregate result is 40 successes, or 69.0%. ADDE is the primary benchmark with 27 out of 37, or 72.90%. SANDSBERG succeeds 8 out of 14 times, or 57.14%. The case studies include VIHALS, a folding chair, with 3 out of 3, and LOVBACKEN, a three-legged table without a backrest, with 2 out of 4.

Figure 6: Zero-shot transportation tests on diverse physical IKEA geometries with the quadrupedal manipulator.

The drop from 94.1% in simulation to 69% in hardware is not treated as noise. The authors identify a concrete chain: to avoid frontal self-collision, the policy sometimes approaches laterally; lateral approach induces abrupt yaw commands; yaw motion degrades real-robot state estimation and odometry; inaccurate state then produces aggressive motion that tips the object. The policy has learned a contact strategy, but its dependence on accurate object and base state becomes visible under real odometry error.

Recovery is an emergent behavior. If the first hook slips or misses, the robot does not remain stuck. It repositions the base, increases end-effector trajectory amplitude, and re-establishes contact before resuming transport. Figure 7 illustrates this sequence. For deployment, this behavior matters as much as the mean success rate because non-prehensile contact will occasionally fail even with a good policy.

Figure 7: After an initial slip, the policy repositions the base, enlarges arm swing, and re-establishes a stable contact.

Dynamic goal tracking tests the controller's local autonomy. A human updates the goal while the robot is transporting the chair. The policy adjusts base velocity and contact forces to follow the changing target. This supports using the learned controller as a local whole-body controller under a separate high-level planner, rather than requiring a static goal for the entire episode.

Payload experiments show that the arm is not the only load path. The authors add 5 kilograms to a chair, bringing total mass to 6.5 kilograms, which exceeds the arm's rated static payload when fully extended. The robot still transports it. The non-prehensile hook, base motion, and ground support redistribute the mechanical burden. This is a practical argument for locomanipulation over lift-and-carry when the object is heavy but not necessarily fragile.

Figure 8: The robot transports a chair with an added 5 kg payload and recovers after a human shifts the object.

Disturbance rejection tests closed-loop behavior. A person pushes the chair during transport. Training did not explicitly provide a human pusher, but the policy repeatedly recovers. When the disturbance forces contact to disengage, it replans its approach and hooks the object again. This suggests that randomized physical properties and contact guidance produce a policy with corrective behavior, not merely a memorized trajectory.

The dishwasher task reveals contact sequencing. Candidates include both the handle and door panel. The policy first uses the handle to initiate opening, then transfers contact to the panel and pushes the door into a fully open horizontal position. The contact sequence emerges from the object's changing kinematics rather than being scripted.

Simple PPO can also open the dishwasher, but it tends to settle into a quasi-static strategy: maintain contact with the handle and pull until the door opens. That solution succeeds in simulation but keeps the arm near joint limits for longer. The proposed method reduces, on average by 59%, the fraction of time steps in which an arm joint lies within 10% of its position limit. The benefit is not only task success but a more feasible whole-body interaction.

Taken together, the experiments show a progression from contact discovery to task refinement. The candidate sampler supplies meaningful interaction regions; the exploration critic turns those regions into a dense early objective; weight scheduling removes that objective once transport competence appears. The real-robot trials then test whether the learned contact physics transfers beyond the exact simulation meshes.

The comparison also clarifies what should not be concluded from reward scheduling. A schedule applied to reward weights and a schedule applied to critic-head advantages can produce similar coefficients during training, but they are not the same operation. In scalar scheduling, the environment reward changes over time, while one critic is forced to represent a return whose composition is nonstationary. Early in training it must value contact-seeking behavior; later it must forget part of that objective and value transport stability with the same state features. In the multi-critic formulation, each return stream remains structurally identifiable. The schedule changes how much each identifiable stream contributes to the actor update, not what each critic is asked to estimate.

This distinction matters for diagnostics. If a scalar-reward run fails, it is difficult to determine whether task return is poorly estimated, exploration return is overrepresented, or regularization dominates the gradient. With grouped critics, an engineer can inspect the separate value heads and their advantage magnitudes. That does not make reward design automatic, but it converts a monolithic failure into a testable decomposition. The ablation results are consistent with this interpretation: scalar scheduling improves contact discovery, while decoupled critics plus scheduling improve the entire trajectory distribution.

The hardware experiments should be viewed as a transfer stress test rather than a product benchmark. The simulation policy starts from randomized poses and physical properties, but it receives privileged object state in the critic and is deployed with external motion capture for object pose. That design allows the authors to evaluate whether the contact behavior itself transfers, before asking whether the full perception stack can transfer. The distinction is important: a 69% aggregate success rate is not yet a robust autonomous system, but it is strong evidence that the learned hooking, repositioning, and recovery patterns are not tied to simulation artifacts.

The object set reinforces that point. ADDE and SANDSBERG are conventional four-legged chairs, yet their different dimensions and masses already expose state-estimation sensitivity. VIHALS tests folding geometry. LOVBACKEN removes the backrest and changes the contact topology to a three-legged table. The policy does not require a new training run for LOVBACKEN; it reorients the base and hooks the legs. This suggests the candidate-based representation captures reusable object affordances rather than mesh-specific key points.

There is also a systems-level lesson in the payload and disturbance tests. A common assumption is that payload capacity belongs to the end effector alone. Here, the learned behavior uses the whole machine as a mechanical system: the arm establishes and maintains contact, the base supplies locomotion and pushing authority, and the ground closes the force chain. That is why a load beyond the arm's static rating can still be moved. It also explains why disturbances are recoverable: a contact loss changes the local force path, but the policy can reorganize base and arm coordinates to re-enter a useful interaction state.

These observations do not eliminate the need for better sensing or uncertainty handling. They do show that learned whole-body control can exploit the robot's locomotion substrate in ways that are difficult to encode in a static whole-body optimization objective. The combination of structured contact candidates and a scheduled learning signal gives the policy a curriculum for acquiring that skill.

Limitations

The authors explicitly acknowledge limited domain-shift coverage. Three tasks, even with diverse chairs, are not enough to prove robustness across arbitrary environments, lighting conditions, object materials, or clutter. They also state that external motion capture is required for object pose tracking, which prevents direct field deployment. Thus, the current result is a zero-shot hardware transfer from simulation meshes to real furniture, not a fully autonomous home robot system.

The hardware failure chain reveals a sensing gap. Lateral approaches create abrupt yaw, yaw degrades odometry, and degraded state estimation leads to aggressive motion. If the policy cannot observe or correct object pose accurately, it may choose a feasible simulated contact but execute it incorrectly on hardware. Future systems will need onboard object-level estimation, contact sensing, or training-time odometry disturbances to close this gap.

The fixed schedule is another boundary. The 5000-to-10000-step decay works for these tasks, but it assumes contact is discovered in that interval. A harder geometry or a different robot may need a later decay, while an easy task may waste time if exploration remains active too long. The authors propose an adaptive performance-based curriculum as a way to reduce this hyperparameter sensitivity.

The sim-to-real performance gap remains substantial. Simulation reaches 94.1%, while the two primary real objects reach 72.9% and 57.1%. The paper provides a plausible mechanism through odometry and lateral approach, but it does not quantify every contributor. Real friction, deformations, mesh error, timing, and sensing noise all interact, so the reported hardware number should be read as an honest deployment result rather than an upper bound.

Finally, no public code repository is listed. Researchers can reproduce the high-level recipe, but the modified multi-critic PPO implementation, procedural chair generator details, reward implementation, and hardware state estimator are not available from the paper itself. This limits independent verification until the authors release code or an extended technical report.

Conclusion and Outlook

The paper's contribution is architectural as much as reward-design. It treats exploration as a temporary competence with its own value head and explicitly schedules its influence out of the policy gradient. Grasp candidates provide structure; multi-critic PPO prevents that structure from contaminating task value; weight decay returns control to the transport objective. This is more direct than manually crafting a single reward and hoping the policy learns when to stop touching the object.

The evidence is convincing because the ablation chain is complete. No exploration reward yields no success. Scalar scheduling improves contact discovery but remains unstable. Fixed multi-critic learning improves discovery but creates tipover. Scheduled multi-critic learning improves both success and stability. Hardware results then show that the same mechanism transfers to unseen chairs, a folding chair, and a three-legged table without fine-tuning.

The next step is to replace privileged object state with deployable perception and contact feedback. An adaptive schedule driven by contact discovery rate, task success, or joint-limit occupancy would make the method less task-specific. Injecting odometry drift and lateral-approach disturbances during training could directly attack the identified hardware failure mode. With those additions, contact-guided multi-critic exploration could become a practical local controller for long-horizon mobile manipulation.

Golden Line

"Grasp candidates can teach a robot where to make contact, but the policy only becomes a transporter when exploration is allowed to retire."