Cross-Embodiment Robot Manipulation via a Unified Hand Action Space

One-Sentence Summary

UHAS represents dexterous hand actions as deformations of a normalized canonical sphere and uses Cascade Inverse Kinematics to map those deformations back to embodiment-specific joint commands, allowing one reinforcement learning policy to transfer across hands with different kinematics and finger counts.

Background and Motivation

Large-scale robot learning has made rapid progress in grasping, pick-and-place, and long-horizon manipulation, but most of that progress is built around simple parallel-jaw grippers. A gripper can often be treated as a point, a direction, and an opening width, so policies and datasets can share an action interface even when the physical hardware differs.

Dexterous hands do not fit that interface. Allegro, LEAP, Shadow, and MANO differ in the number of fingers, the number of joints per finger, the axes of rotation, the range of motion, and the actuator interfaces. A vector of joint positions in one hand has no direct semantic counterpart in another hand. As a result, dexterous manipulation systems tend to be trained separately for each platform.

The paper starts from a simple observation: even though robotic hands look very different, they interact with objects through spatially organized contact patterns around the object. If an action can describe that spatial pattern instead of the joint configuration that produces it, then different hands can share the same action semantics.

The proposed solution is the Unified Hand Action Space, or UHAS. Actions are defined as geometric deformations of a canonical sphere. The sphere gives a continuous, closed, and topology-agnostic surface that can be normalized across hands. A Cascade Inverse Kinematics algorithm, or CIK, then recovers the concrete joint configuration for a particular hand.

The result is a controller-level abstraction rather than a new learning algorithm. Policy learning happens in the shared sphere space, while hardware-specific details are pushed into precomputed lookup tables and cascaded one-dimensional IK solves.

Prerequisites

The evaluation task is in-hand cube reorientation. The simulation environment is adapted from NVIDIA Isaac Lab's Repose Cube. Each episode requires the hand to reach ten sequential target cube orientations. If the cube falls, the environment resets and the remaining targets are attempted until all ten have been evaluated.

Policies are trained with the RSL-RL implementation of Proximal Policy Optimization. The actor-critic network is deliberately lightweight. The actor receives normalized observations in the canonical sphere frame and outputs sphere deformation parameters, while the critic can use privileged state information.

The geometric representation relies on spherical coordinates. A point on the sphere is described by the azimuthal angle $\theta$, the polar angle $\phi$, and the radius $r$. UHAS assigns every relevant hand surface point a fixed spherical-coordinate label. Finger motion can then be interpreted as moving those labels inward, outward, or sideways relative to the object-centered sphere.

Method

Automatic Sphere Construction from URDF

UHAS does not require a hand-specific sphere to be hand-designed. Given a URDF, the pipeline first identifies the palm frame and fingertip frames in an open-hand configuration where the fingers are fully extended. It computes the palm center as the average of finger root positions and measures the average distance $l$ from the palm center to the fingertips.

$$r=\frac{2l}{\pi}$$

This radius places the sphere within the natural grasping workspace of the hand and covers approximately a 90-degree arc from the palm center to the fingertips. The sphere center lies along the outward palm normal. The positive $z$-axis aligns with the outward palm normal, and the positive $x$-axis aligns with the middle finger direction; the $y$-axis follows the right-hand rule.

All distances are normalized by the hand-specific sphere radius. This produces a unit-sphere coordinate system that is scale-invariant across hands. The original hand-specific sphere parameters are preserved for CIK, so normalization does not discard the information needed to reconstruct real joint commands.

Figure 2: Automatic sphere creation for a robotic hand from its URDF.

Dense Sphere-Hand Correspondence

The sphere becomes useful only when its surface can be related to the hand. The method uniformly samples points on the canonical sphere, computes spherical coordinates and outward surface normals, and projects the sphere points onto nearby points on the interior hand surface.

This projection creates dense correspondences between the sphere and the palm and finger surfaces. The important property is configuration invariance: the 3D locations of hand surface points change when fingers move, but their associated spherical coordinates remain unchanged. Policies can therefore learn in a fixed spherical coordinate domain.

Figure 3: Uniform sphere sampling, normal computation, projection to the hand surface, and configuration-invariant spherical labels.

A Compact Deformation Action Space

Deforming every point on the sphere independently would produce an impractically large action space. UHAS therefore uses only two deformation components: lateral angular displacement $\Delta\theta$ and radial expansion or contraction $\Delta r$. A reference point on the undeformed sphere is described by $(\theta,\phi,r)$ with $r=1$.

$$(\theta,\phi,r),\quad r=1$$

The deformation field is parameterized by sparse control primitives. Each finger is assigned a driving plane through the sphere center at a fixed azimuthal angle $\theta_{\text{plane}}$. Rotating these planes controls $\Delta\theta$. Radial displacements at control points within each plane, called driving vectors, control $\Delta r$.

The full field is reconstructed by interpolation. The $\Delta\theta$ component is interpolated across neighboring driving planes, while $\Delta r$ is interpolated in the two-dimensional $(\theta,\phi)$ parameter space. The default configuration uses five driving planes and two driving vectors per plane, giving a 15-dimensional action space.

$$\text{dim}(a)=N_{\text{plane}}+N_{\text{vector}}\times N_{\text{plane}}=5+2\times 5=15$$

For four-finger hands, an extra driving plane is placed at the ring-finger azimuthal position. Deformations for the merged fingers are averaged before interpolation. This keeps the policy output identical across four-finger and five-finger embodiments.

Figure 4: Driving planes control $\Delta\theta$, driving vectors control $\Delta r$, and interpolation reconstructs the deformed sphere.

Cascade Inverse Kinematics

CIK takes a deformed sphere as input and produces a joint configuration $\mathbf{q}$ for a specific hand. The first step is joint classification. Each joint is swept over its full range in an open-hand base configuration while forward kinematics records the effect on fingertip spherical coordinates. Lateral joints predominantly change $\theta$; encompassing joints predominantly change $r$ and $\phi$.

Lateral joints are solved with a precomputed lookup table. Offline, the pipeline sweeps each lateral joint across its range, re-solves encompassing joints on the undeformed sphere, and records the resulting fingertip azimuthal angle. At inference time, the target fingertip angle is

$$\theta_{\text{fingertip}}=\theta_{\text{initial}}+\Delta\theta$$

and the table returns the corresponding lateral joint value. This makes lateral solving constant-time at runtime.

Encompassing joints are solved sequentially from the finger root toward the fingertip. For each joint, forward kinematics transforms the target sphere points associated with that joint and its descendants into the joint's local frame. A one-dimensional inverse kinematics subproblem then places those points onto the deformed sphere. Each joint is solved once in a single forward pass, so CIK can run at approximately 150 Hz.

The lookup implementation is compact. The function below comes from sphere_torch_utils.py:311-329 and maps normalized lateral offsets to entries in a precomputed joint-angle table.

def torch_solve_for_A_joints_ohne_interpolation(
    type_A_joints, anchor_offsets, res, q_A_zero_idx,
    q_A_max, q_anchor_dist, q_list_dict):
    joint_values = torch.zeros_like(anchor_offsets, ...)
    offset_idx = torch.round(
        anchor_offsets / res[None, :]
        + q_A_zero_idx[None, :]
        + (q_anchor_dist / res)[None, :])
    offset_idx = torch.clamp(offset_idx, 0, q_A_max[None, :] - 1).int()
    for i, q_A in enumerate(type_A_joints):
        joint_values[:, i] = q_list_dict[q_A][offset_idx[:, i]]
    return joint_values

The environment-side action preprocessing in multi_manipulation_env.py:652-700 follows the same logic. The policy action is split into lateral offsets and vector offsets, scaled by hand-specific lookup limits, merged for four-finger embodiments, and then passed into deformed-sphere generation and per-joint solving.

main_joint_offsets = self.processed_actions[:, :self.max_fingers]
main_joint_offsets = torch.where(
    main_joint_offsets < 0,
    main_joint_offsets * -self.finger_lateral_min,
    main_joint_offsets * self.finger_lateral_max)
torch_vector_offsets = self.processed_actions[:, self.max_fingers:]
torch_vector_offsets = torch_vector_offsets.view(
    self.num_envs, self.max_fingers, self.vector_phis.size(2))
torch_vector_offsets = scale(
    torch_vector_offsets,
    self.min_vector_offsets[None, :],
    self.max_vector_offsets[None, :])

Figure 5: Lateral and encompassing joint classification, plus the cascaded solve on a deformed sphere.

Shared Observations and Reward

A unified action space is not sufficient if the observations are still hand-specific. The paper discretizes each finger into seven equally spaced points from root to fingertip. Positions are computed with forward kinematics, and velocities use the corresponding Jacobians. The final policy uses two points per finger, the midpoint and the fingertip, and duplicates ring-finger observations for four-finger hands.

Point positions, object positions, and velocities are transformed into the canonical sphere frame and divided by the hand-specific radius. The normalization appears in multi_manipulation_env.py:1367-1372. This gives the actor a homogeneous proprioceptive representation across hands.

The reward is adopted from the original Isaac Lab environment with two additional regularization terms that penalize deviation of lateral and encompassing joints from reference positions. The per-step reward is

$$r=w_{d}\,d+w_{r}\,r_{\text{rot}}+w_{\text{lat}}\,p_{\text{lat}}+w_{\text{rad}}\,p_{\text{rad}}+b_{\text{success}}+p_{\text{fall}}$$

Here $d$ is the object-to-goal distance, $r_{\text{rot}}$ is the orientation alignment reward, $p_{\text{lat}}$ and $p_{\text{rad}}$ are joint position penalties, $b_{\text{success}}$ is a large bonus for reaching the target within 0.1 radians, and $p_{\text{fall}}$ is a penalty when the cube falls. The implementation is in multi_manipulation_env.py:1615-1628.

Training Details and Domain Randomization

All models are trained on a single NVIDIA A5000 GPU in the custom Isaac Lab cube-reposing environment. The network uses hidden dimensions $[512,512,256,128]$, ELU activations, a learning rate of $5.0\times 10^{-4}$, an entropy coefficient of $0.005$, and empirical normalization. The actor-critic is trained with PPO using a discount factor of $0.99$ and GAE parameter $0.95$.

Domain randomization covers object scale, mass, friction, robot mass, joint friction, armature, effort limits, stiffness, damping, hand base inclination, and driving-vector azimuthal angle. Randomizing hand base inclination prevents the policy from relying on a particular cube-palm sliding behavior. Randomizing object scale helps avoid finger entrapment caused by geometric differences between hands. Randomizing PD gains, effort limits, and driving-vector angles changes the in-distribution dynamics and improves zero-shot transfer.

The real-world training recipe differs from simulation-only training. The authors increase the domain randomization range, randomize velocity limits, and use an asymmetric actor-critic design. The actor receives only positional information about the object and hand joints, while the critic receives full state including velocities. This design is intended to make the policy rely on position feedback, which is more reliable under real sensing and communication noise.

Negative radial deformations also play an important role. Before CIK, all sphere points with negative radial coordinate $r$ are discarded. If an encompassing joint has no remaining reachable points, it is commanded to its fully closed configuration. The paper reports that allowing negative radial deformations substantially improves performance on the dynamic reorientation task, because it enables rapid finger closing during aggressive reposes.

System Identification for the Real LEAP Hand

Sim-to-real transfer on the LEAP Hand required modeling the physical current-based servo. The paper writes the control law as

$$\text{current}(t)=K_{p}\Delta\theta(t)-K_{d}\Delta\dot{\theta}(t)$$

where $\Delta\theta(t)$ is the joint position error. Mapping this law directly to implicit PD torque actuators in simulation did not reproduce the real motion. The authors commanded random target positions, varied $K_p$ and $K_d$, recorded joint position, velocity, and current trajectories, and solved for effective gains that matched the data.

The identification found that the LEAP Hand motors are nearly undamped. The effective proportional gain is approximately $0.0786\ \text{Nm/rad}$ per 100 motor units, and the effective derivative gain is approximately $0.0014\ \text{Nm/(rad/s)}$ per 100 motor units. Because serial communication bandwidth is limited, the policy runs at 20 Hz during training and real deployment.

flowchart LR
    A[URDF] --> B[Automatic Sphere]
    B --> C[Dense Correspondence]
    C --> D[Policy in UHAS]
    D --> E[Sphere Deformation]
    E --> F[CIK Lateral Lookup]
    E --> G[CIK Encompassing Cascade]
    F --> H[Joint Positions q]
    G --> H
    H --> I[Low-level PD Control]

Experiments

Main Simulation Results

Simulation evaluation uses 1000 parallel environments across four hands: Allegro, LEAP, Shadow, and MANO. Table 1 reports Success Rate and Average Consecutive Reorientations for single-hand training, the joint-control baseline, multi-hand training, and zero-shot evaluation where the target hand is excluded from training.

The most informative row is Shadow. A policy that never sees Shadow during training still reaches 85.7% success and 4.4 consecutive reorientations, while the Shadow-specific policy reaches 9.6. The gap shows that zero-shot transfer is real but not free; morphology differences still reduce the average run length.

MANO transfers with almost no loss. This is consistent with the idea that high-degree-of-freedom five-finger hands share a more similar geometric action structure. The 4-finger hands also transfer well to each other, while cross-finger-count transfer is harder.

Cross-Morphology Transfer and Fast Finetuning

The paper also trains policies on pairs of hands with the same finger count and evaluates them on the opposite morphology. A policy trained on Shadow and MANO reaches 66.2% success on Allegro and 80.8% on LEAP. A policy trained on Allegro and LEAP reaches 83.2% on Shadow and 95.0% on MANO.

These numbers are lower than in-distribution results, but they still demonstrate meaningful transfer across different finger counts. The duplicated ring-finger plane lets a five-finger action interface run on four-finger hands without extra processing.

The finetuning experiments start from a MANO-only policy and adapt it to each unseen hand for only 500 iterations. Training from scratch typically requires about 4500 iterations. After 500 iterations, Allegro improves from 7.7 to 8.1 consecutive reorientations, LEAP from 7.7 to 8.0, and Shadow from 4.4 to 7.8. The Shadow improvement is the strongest evidence that UHAS preserves reusable policy knowledge.

One-to-Many Transfer Analysis

Appendix F reports one-to-many zero-shot experiments in which a single source hand trains the policy and all other hands are tested without finetuning. The results expose a strong asymmetry that is not visible from the average zero-shot numbers in the main table.

An Allegro-trained policy fails almost completely on Shadow and MANO. The paper attributes this to exploitative use of Allegro's distinctive lateral joints. LEAP has a similar lateral capability when fingers are flexed, which explains the partial transfer to LEAP.

The LEAP-trained policy transfers best overall. LEAP has the largest range of motion among the four hands, and its source performance is also high. Shadow transfers well to MANO, but MANO transfers poorly to Shadow. This asymmetry suggests that constrained source hands produce policies overfitted to their joint limits, while high-motion source hands learn more general behaviors.

The paper leaves an open question: whether explicitly limiting the source range of motion during training could improve robustness, or whether the asymmetry is intrinsic to cross-embodiment transfer. This is a useful framing because it separates representation capacity from training distribution.

Observation Ablation

Appendix E ablates the number of homogeneous observation points per finger. The paper tests one, two, three, and four points sampled along each finger. Success rates are 98.8, 98.7, 99.0, and 99.1, while average consecutive reorientations are 9.3, 9.3, 9.4, and 9.5. Training time is 4.9, 4.5, 4.8, and 4.7 hours.

The improvements are marginal and mostly disappear once domain randomization is applied. The paper therefore uses two points per finger in the final model, which balances performance with inference cost. The implementation keeps the observation dimension stable by duplicating ring-finger observations for four-finger hands.

Driving-Plane Ablation

The number of driving planes is another design choice. The paper compares the standard five-plane UHAS with a four-plane variant in which the ring and pinky planes are merged by averaging their deformations. Shadow reaches 99.5 / 9.7 with four planes and 99.3 / 9.6 with five planes. MANO reaches 99.6 / 9.8 with four planes and 99.8 / 9.9 with five planes.

Three-plane models did not converge to meaningful policies. The paper interprets this as evidence that a minimum action-space expressiveness is required for the task. It also cautions that the near-equivalence of four and five planes may be specific to cube reposing, the chosen reward, and the reference cube position. Removing the pinky plane may matter more in other tasks where the pinky is actually needed.

Cube Pose Estimation

Real-world object tracking uses a 3D-printed cube with four distinct AprilTags on each face, for 24 tags total. One or two Intel RealSense cameras stream rectified infrared images at 848 by 480 pixels and 60 Hz. A second camera reduces occlusion and increases the number of visible tags during fast finger motion.

Each visible tag is localized with perspective-n-point pose estimation from its four corners. Tags are accepted only when the Hamming decoded value is zero and the decision margin is at least 50. A separate tag on the base mount provides a common reference frame. In dual-camera mode, messages within 100 ms are paired, duplicate tags keep the higher detection margin, and inconsistent estimates are dropped if translation deviates more than 10 mm from the median or orientation deviates more than 0.075 rad from a consensus quaternion. Remaining estimates are fused by mean position and hemisphere-aligned quaternion averaging, and the pose is published only when at least three separate tags are visible.

Physical Hardware Handling

The real LEAP Hand deployment required additional engineering beyond policy training. Serial communication was unreliable, with frequent missed reads and writes, so the authors added software-level communication management. Some motor state readings consistently failed through the manufacturer API.

Several joint ranges were limited because the physical joints overshoot and the structural components deform under load. A lower velocity limit was used to reduce overshoot. Repeated use loosened the fingers, so the authors 3D-printed a reinforced base for the root joints of the index, middle, and ring fingers and designed a custom attachment for the thumb root. These mechanical fixes are planned for public release alongside the paper.

Ablation on Driving Vectors

The number of driving vectors per plane is one of the main action-space hyperparameters. The paper trains with one, two, three, and four vectors per finger while keeping all other components fixed. Training time is measured as the time to reach 90% of the maximum average consecutive reorientations.

Two vectors are the chosen operating point because they match the performance of three vectors while training fastest. One vector lacks enough control flexibility, and four vectors increase the action dimension without improving the final result.

Real-World LEAP Hand Results

The real-world setup uses an AprilTag-instrumented cube and a pose estimator built from multiple visible tags. The authors also performed system identification on the LEAP Hand actuators because the physical current-based servos do not match the implicit PD torque model in simulation.

All UHAS variants outperform the joint-control baseline on the physical LEAP Hand. The best model is the LEAP-specific policy, with a mean of 2.0 consecutive reorientations and a best trial of 6. The zero-shot model still reaches 0.9, above the baseline's 0.6.

The multi-hand model underperforms the LEAP-specific model in the real world. The paper attributes this to a more conservative policy: multi-hand training must avoid motions that are unsafe or infeasible on any hand, which constrains the solution in the simple actor-critic architecture.

Appendix F reports real-world Allegro Hand results as well. The zero-shot UHAS model reaches a mean of 0.8 consecutive reorientations, while the multi-hand model and the Allegro-specific model both reach 2.1. The best multi-hand trial completes 8 reorientations, and the best Allegro-specific trial completes 4. Trial variance is high, with immediate failures in some runs and longer successes in others. This pattern is consistent with the LEAP experiments and confirms that cross-embodiment zero-shot transfer survives sim-to-real contact dynamics, but with a noticeable performance penalty.

Figure 6: The four-hand simulation environment and the real LEAP Hand setup.

Discussion

The main distinction between UHAS and prior cross-embodiment approaches is that the shared action space is geometric. It does not map joint values to joint values, and it does not rely on latent actions learned implicitly by a network. Instead, the hand is abstracted as a sphere-like interaction surface around an object.

The public simulation repository stores per-hand CIK metadata in sphere_cik.json. This includes driving planes, joint classification, lookup tables, and reference sphere information. Adding a new hand follows a practical path: convert the URDF, generate the sphere and CIK configuration, then test the pretrained multi-hand policy without retraining.

The README also references a companion real-world deployment repository, but the public clone was not available when this analysis was prepared. Code-based verification is therefore strongest for the simulation pipeline; the real-world communication and system-identification details are taken from Appendix D of the paper.

From a code perspective, the simulation repository is structured around three main pieces. The URDF processing pipeline in process_urdf/ builds spheres, classifies joints, and generates sphere_cik.json. The Isaac Lab task in tasks/UHAS_inhand/ loads those configurations, generates deformed spheres, applies CIK, and computes rewards. The pretrained models and evaluation scripts let a user reproduce the multi-hand policy without writing new network code.

This separation is useful for future deployments. Adding a new hand does not require changing the policy network or the action dimension. It requires producing a valid CIK configuration for the new hand and then either deploying the shared policy zero-shot or finetuning it with a small number of iterations.

Limitations

The authors list three explicit limitations. Performance is sensitive to low-level PD controller parameters; the in-hand reorientation task is sensitive to reward design and RL hyperparameters; and transfer degrades for substantially different hand morphologies such as four-finger versus five-finger hands.

The real-world numbers reinforce these concerns. The LEAP-specific UHAS model reaches only 2.0 consecutive reorientations on average, compared with roughly 9.8 in simulation. System identification and domain randomization narrow the gap but do not eliminate it.

The one-to-many results in Appendix F expose a more subtle limitation. Transfer is asymmetric: LEAP is a strong source hand, while MANO-to-Shadow transfer is poor. Hands with larger ranges of motion tend to produce more transferable policies, while constrained hands overfit their specific joint limits. The paper leaves open whether explicitly limiting training range of motion can fix this.

Multi-hand training also does not automatically dominate single-hand training. The real-world multi-hand policy is more conservative and performs worse than the LEAP-specific policy. A unified action space is therefore necessary but not sufficient for building a better generalist policy.

Conclusion and Outlook

UHAS reframes dexterous hand control as the problem of deforming a shared geometric surface. The paper shows that this representation supports joint training across heterogeneous hands, zero-shot deployment to unseen hands, rapid finetuning, and real-world operation on a physical LEAP Hand.

The strongest next steps are to test UHAS with larger policies and more hands, to study whether constrained training ranges can reduce transfer asymmetry, and to extend CIK to contact-rich tasks with non-cubic objects. If those directions hold, UHAS could serve as a practical action head for cross-embodiment robot foundation models.