Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data

Background and Motivation

Vision-language-action (VLA) models have made impressive progress in robotic manipulation, but their generalization ability is fundamentally constrained by the scale and diversity of available robot data. Despite major data collection efforts such as Open X-Embodiment, DROID, and AgibotWorld, robot teleoperation data remains expensive, labor-intensive, and limited by hardware availability and interaction diversity. A single robot trajectory costs orders of magnitude more to collect than a video of a human hand manipulating an object—yet the latter can be captured at massive scale in virtually any environment.

Egocentric human manipulation videos offer a compelling alternative. Compared to robot teleoperation, human hand interactions can be collected at massive scale across diverse objects, environments, and task variations. These videos capture rich manipulation priors—how objects are approached, when grasps are initiated, how tools are used with hand-eye coordination—that are difficult to obtain with robots alone. However, directly using human videos to train robot policies is infeasible due to the substantial embodiment gap between human hands and robot end-effectors.

Prior work has explored two paths. The first is co-training: mixing human video with robot data to let the model implicitly learn shared visual-semantic representations. The second is retarget-and-render: mapping human hand motions to robot kinematics and visually replacing human arms with rendered robot embodiments. The latter has shown strong results at small scale for individual tasks. But a critical question remains unanswered: can large-scale ego2robot-synthesized data serve as effective pretraining data for VLA models, particularly for improving out-of-distribution generalization?

Ego2Robot is a systematic answer to this question. The authors' core hypothesis is that, despite embodiment differences, egocentric manipulation videos contain transferable interaction regularities—trajectory patterns of object approach, timing of grasp initiation, hand-eye coordination in tool use—that can complement robot data if properly aligned. To test this, they build a complete pipeline supporting both annotated and unannotated input paths, processing ~1,940 hours of egocentric video from four sources across 15 robot morphologies, yielding 18,561 hours of synthetic training data—the largest ego-to-robot dataset to date.

Preliminaries

Understanding Ego2Robot requires several key concepts. First, hand pose estimation: recovering the 3D coordinates of 21 hand keypoints from RGB images. The paper uses WiLoR for per-frame reconstruction and DynHaMR for temporal optimization. Second, inverse kinematics (IK): given an end-effector pose, solving for robot joint angles that position the end-effector at the target. The paper uses MuJoCo's IK solver. Third, camera-frame relative actions: expressing end-effector displacements in the observer's (camera) coordinate frame rather than the world frame, which naturally unifies data from different camera setups and robot morphologies without explicit extrinsic calibration.

Another key concept is flow matching, the training objective for the action head. Similar to diffusion models, it learns a vector field from noise to data to generate action sequences, but uses linear interpolation paths rather than stochastic differential equations, requiring only a few Euler steps at inference.

Method: The Ego2Robot Pipeline

The Ego2Robot pipeline consists of three stages: action alignment, visual alignment, and quality curation. Each input video passes through these stages to produce parallel training data streams for 15 robot morphologies.

Pipeline Overview

flowchart TB
    A[Egocentric Video
~1940h] --> B{Input Path}
    B -->|Path A: Annotated| C[Existing hand poses]
    B -->|Path B: Unannotated| D[WiLoR per-frame
DynHaMR temporal opt
Qwen3.5 subtask segmentation]
    C --> E[Action Alignment]
    D --> E
    E --> E1[Hand-to-gripper retargeting]
    E --> E2[Temporal smoothing
Savitzky-Golay + SLERP]
    E --> E3[Action speed alignment]
    E1 --> F[Visual Alignment]
    E2 --> F
    E3 --> F
    F --> F1[SAM3 arm segmentation]
    F --> F2[ProPainter hand removal]
    F --> F3[Robot base pose search
Grid search + IK feasibility]
    F --> F4[MuJoCo IK per-frame solving]
    F --> F5[Depth-aware compositing]
    F1 --> G[Quality Curation]
    F5 --> G
    G --> G1[L1 Pipeline-internal]
    G --> G2[L2 Statistical filtering]
    G --> G3[L3 VLM consistency audit]
    G1 --> H[15 morphology parallel output
18561h training data]
    G2 --> H
    G3 --> H

The diagram above shows the complete Ego2Robot pipeline. Input video first passes through hand pose estimation (Path B for unannotated videos), then through action alignment, visual alignment, and quality curation, generating a training data stream for each of 15 robot morphologies.

1. Action Alignment

The action alignment stage converts human hand poses into parallel-gripper end-effector trajectories. The core is extracting a compact gripper representation from 21 hand keypoints. The paper defines a virtual fingertip as a weighted blend of the index and middle fingertips:

$$\mathbf{p}_{\text{vf}} = 0.7 \cdot \mathbf{p}_{\text{index}} + 0.3 \cdot \mathbf{p}_{\text{middle}}$$

This virtual fingertip represents the grasping center of the gripper. The tool center point (TCP) and gripper opening width are defined by the thumb tip and virtual fingertip:

$$\mathbf{p}_{\text{tcp}} = \frac{\mathbf{p}_{\text{thumb}} + \mathbf{p}_{\text{vf}}}{2}, \quad w = \|\mathbf{p}_{\text{thumb}} - \mathbf{p}_{\text{vf}}\|$$

TCP is the midpoint between thumb and virtual fingertip; the opening width is the Euclidean distance between them. This design elegantly maps the human grasping configuration to a single scalar opening of a parallel gripper.

The grasp orientation computation is more intricate. The paper constructs a right-handed orthonormal frame $\mathbf{R} = [\mathbf{x}\ \mathbf{y}\ \mathbf{z}]$ with three axes defined as follows: the grasp axis $\mathbf{z}$ lies along the thumb-to-virtual-fingertip direction; the wrist-to-fingertip direction $\mathbf{d}$ and $\mathbf{z}$ span the jaw plane, whose normal is the gripper-normal axis $\mathbf{y}$; the approach axis $\mathbf{x}$ completes the frame:

$$\mathbf{z} = \frac{s(\mathbf{p}_{\text{thumb}} - \mathbf{p}_{\text{vf}})}{w}, \quad \mathbf{y} = \frac{\mathbf{z} \times \mathbf{d}}{\|\mathbf{z} \times \mathbf{d}\|}, \quad \mathbf{x} = \mathbf{y} \times \mathbf{z}$$

where $s = +1$ for the right hand and $s = -1$ for the left hand, ensuring $\mathbf{z}$ points consistently regardless of handedness and both hands map to the same gripper frame.

Temporal smoothing addresses high-frequency noise from per-frame detection. The paper applies Savitzky-Golay filtering to positions and widths, and Gaussian-weighted SLERP (spherical linear interpolation) to orientations, producing smooth trajectories while preserving motion structure.

Action speed alignment is a subtle but impactful design. Egocentric human manipulation moves significantly faster than robot teleoperation. The paper applies per-source frame subsampling: ANT and EgoDex are downsampled to 60% of their original frame rate (~1.7× slower), EgoVerse to 45% (~2.2× slower), and ViTRA to 25% (~4× slower). This per-source tuning ensures the speed distribution of synthesized data matches real robot data.

2. Visual Alignment

Visual alignment replaces human arms in the original video with rendered robot arms, transforming the video from depicting human manipulation to robot manipulation.

Arm segmentation uses SAM 3 to segment human arm regions in each frame, providing temporally consistent masks. Hand removal uses ProPainter for temporally consistent video inpainting to remove human arms and reconstruct the background.

Robot base pose search is the most challenging step in visual alignment. Unlike robot-to-robot transfer—where a source robot's base position is available—egocentric hand trajectories are embodiment-free with no physical robot base to reference. The paper must find a base pose $\mathbf{T}_{\text{base}} = (\mathbf{t}, \mathbf{R}) \in SE(3)$ such that the retargeted trajectory remains kinematically feasible for the target robot morphology. Feasibility is assessed by solving IK at representative keyframes:

$$\max_{\mathbf{T}_{\text{base}}} \sum_{k \in \mathcal{K}} \mathbb{1}\left[\text{IK}(\mathbf{T}_{\text{base}}^{-1}\mathbf{T}_{k}^{\text{ee}}) \text{ is feasible}\right]$$

where $\mathcal{K}$ is a set of representative keyframes covering the spatial extremes of the trajectory (positions with maximum displacement or orientation change), and $\text{IK}(\cdot)$ is the MuJoCo inverse kinematics solver. Candidate base placements are generated via grid search around the trajectory centroid, constrained by per-morphology kinematic reach $r_{\text{max}}$. This search is performed independently for each of 15 robot morphologies, as different arm lengths and joint configurations require different base placements for the same trajectory.

Depth-aware compositing renders the robot from the original camera viewpoint and composites it into the inpainted scene using depth ordering:

$$I_{\text{final}}(u,v) = \begin{cases} I_{\text{robot}}(u,v) & \text{if } D_{\text{robot}}(u,v) < D_{\text{scene}}(u,v) \land M_{\text{robot}}(u,v) = 1 \\ I_{\text{inpaint}}(u,v) & \text{otherwise} \end{cases}$$

where $D_{\text{scene}}$ is obtained from depth sensors or monocular depth estimation. This ensures the robot arm correctly occludes or is occluded by objects in the scene. The process is applied independently for 15 morphologies (Panda, UR5e, ARX-L5, xArm7, Sawyer, Kinova Gen3, IIWA, Jaco, FR3, UR10e, ViperX, WidowX, Piper, YAM, Aloha-Agilex).

3. Quality Curation

Quality curation is a three-level filtering mechanism. L1 (pipeline-internal): frames with IK failures, self-collisions, action outliers, or insufficient workspace coverage are flagged during processing. L2 (statistical): trajectories with extreme action values, sudden discontinuities, or excessive invalid frame ratios are removed. L3 (VLM consistency): a vision-language model (Qwen3.5) audits synthesized videos for semantic consistency between rendered robot actions and original manipulation intents. This layer ensures the synthesized data is semantically coherent, not just kinematically valid.

Camera-Frame Relative Actions

Since egocentric videos are captured with diverse, unknown camera placements, a world-frame action representation would require per-video calibration and produce incompatible action spaces across sources. The paper adopts camera-frame relative end-effector actions. Each action step is 7-dimensional: 3D position delta $\Delta\mathbf{p}$, 3D rotation delta $\Delta\boldsymbol{\omega}$ (rotation vector), and 1D gripper. Given the EEF-to-camera transformation $\mathbf{T}_{ce} = \mathbf{T}_{wc}^{-1}\mathbf{T}_{we}$ with rotation $\mathbf{R}_{ce}$, base-frame deltas are transformed to camera-frame:

$$\Delta\mathbf{p}_{cc} = \mathbf{R}_{ce}\,\Delta\mathbf{p}_{ee}, \quad \Delta\mathbf{R}_{cc} = \mathbf{R}_{ce}\,\Delta\mathbf{R}_{ee}\,\mathbf{R}_{ce}^{\top}$$

This transformation naturally unifies data from different camera setups and robot morphologies without explicit extrinsic calibration.

Model Architecture and Training Objective

The VLA model uses Qwen3.5-4B as the vision-language backbone with a Diffusion Transformer (DiT) action head. The model predicts 32-step action chunks with 8 diffusion steps. The action head uses a flow matching training objective. Given clean actions $\mathbf{a}_0$ and noise $\boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$, the noised sample at time $t \in [0,1]$ is:

$$\mathbf{a}_t = (1-t)\,\mathbf{a}_0 + t\,\boldsymbol{\epsilon}$$

The training objective predicts the vector field from noise to data:

$$\mathcal{L} = \mathbb{E}_{t,\mathbf{a}_0,\boldsymbol{\epsilon}} \left\|\mathbf{v}_\theta(\mathbf{a}_t, t, \mathbf{c}) - (\boldsymbol{\epsilon} - \mathbf{a}_0)\right\|^2$$

where $\mathbf{c}$ is the visual-language conditioning. Inference uses 4 Euler steps ($\Delta t = 1/4$).

Evaluation Framework

The paper introduces a disentangled evaluation protocol to answer a key question: on which generalization dimensions does synthesized data help, and where is it limited by domain gaps? Existing benchmarks (e.g., RoboTwin 2.0) apply multiple perturbation factors simultaneously, producing a single aggregate OOD metric that conflates visual, spatial, embodiment, and semantic shifts.

Ego2Robot extends RoboTwin 2.0 with 11 independent perturbation settings and camera-frame relative EEF support, complemented by EBench (7 precision tabletop tasks with a higher-mounted camera closer to the egocentric perspective). The evaluation covers four generalization dimensions: visual appearance (background texture, lighting, robot color), scene layout (height, clutter, camera offset), embodiment morphology (ARX, UR5, Franka), and task semantics (unseen objects, paraphrased instructions).

Experimental Results

Main Results

The paper compares four pretraining configurations: robot-only data (DROID + AgibotWorld + InternData, ~6,565h), and Ego2R mixed with robot data at 1:3, 3:1, and 1:1 ratios. All configurations train for the same number of steps (200K steps), processing the same total number of training samples (~19.2M frames), ensuring fair comparison regardless of dataset size.

Table 1: Success rates (%) on RoboTwin2.0 and EBench. Green = gain > 5% over robot-only.

The 1:1 ratio leads in five of seven columns, reaching 68.1% on Clean (+5.9) and 53.5% on Randomized (+2.6). The 3:1 ratio achieves the best EBench score (51.7%, +12.1 over robot-only). The 1:3 ratio yields marginal gains, indicating that synthesized data needs to reach a certain proportion to produce substantial benefits.

Per-Perturbation Breakdown

Table 2: Per-perturbation success rate breakdown (%).

Visual appearance benefits most. The 1:1 ratio improves on all three visual factors: background +4%, lighting +8%, and robot color +6%. This is attributed to scene diversity in egocentric videos and multi-morphology rendering across 15 configurations. Embodiment transfer benefits from multi-morphology data: ARX improves from 44 to 51 (1:1), and UR5 peaks at 31 at 3:1 (+11.2). However, Franka remains below 7%, reflecting its large kinematic gap from training embodiments. Task semantics improve consistently: unseen object generalization improves from 29% to 40% (+11 at 3:1), and paraphrased instruction robustness reaches 69% at 1:1 (+5.4).

Ablation Studies

The ablation uses ego-only pretraining (no robot data) to isolate the pipeline's contribution. Raw ego co-training achieves only 28.1%. Processing through the pipeline (single-morphology Ego2R) improves this to 31.7% (+3.6), validating the visual and action alignment stages. Increasing from 1 to 15 morphologies steadily improves performance (31.7 → 33.5). Adding raw ego data alongside 15-morphology Ego2R data yields a further jump to 37.3%—the raw ego data effectively acts as a 16th "morphology" with slightly different visual appearance and action distribution, further enriching pretraining diversity.

Real Robot Experiments

On the ARX ACone dual-arm platform, five long-horizon tasks are evaluated: putting fruits into a basket, putting blocks into a drawer, folding a towel, sweeping trash into a bin, and inserting a screw. Three configurations are compared: robot-only pretraining, Ego2R+Robot 1:1 pretraining, and finetuning that also incorporates Ego2R Play data. Mix+Ego2R Play achieves the best results across all five tasks, with the largest gains on Put Blocks (+14 over robot-only) and Insert Screw (+13), demonstrating that casually recorded egocentric videos can be converted into effective training signal via the pipeline.

Limitations

The paper acknowledges three limitations. First, retargeting maps hand poses to parallel-jaw grippers, discarding fine-grained finger articulation. Extending to dexterous multi-finger hands could broaden the range of transferable skills. Second, visual alignment relies on inpainting and depth-aware compositing, which may introduce artifacts under heavy occlusion or complex lighting. Improving rendering fidelity with generative models could further reduce the visual domain gap. Third, evaluation is limited to the task scope of RoboTwin2.0; extending to broader tasks and embodiment configurations would strengthen the generality of conclusions.

From an independent assessment, another potential limitation lies in the reliance on VLM consistency auditing. The L3 layer uses Qwen3.5 to audit synthesized videos for semantic consistency, but the VLM's judgment itself may err—if it incorrectly labels a synthesized video as semantically consistent when it is not, noisy data enters the training set. The paper does not report the false positive and false negative rates of the L3 layer, leaving its quality control effectiveness quantitatively unassessed.

Furthermore, the grid resolution of base pose search may affect data quality. The grid search resolution determines the precision of base placement; too coarse a grid may lead to suboptimal placements, while too fine a grid incurs excessive computational cost. The paper does not discuss the trade-off between grid resolution and data quality in detail.

Conclusion and Outlook

Ego2Robot demonstrates that egocentric human manipulation videos can serve as a scalable robot data source. Through a three-stage pipeline of action alignment, visual alignment, and multi-level quality curation, 1,940 hours of first-person video are transformed into 18,561 hours of training data across 15 robot morphologies. The disentangled evaluation reveals that synthesized data provides the most value in visual robustness, embodiment transfer, and semantic generalization—precisely the dimensions where robot-only data is most scarce. Real robot experiments further validate that even a small amount of casually recorded egocentric video can be converted into effective training signal.

This work opens a path toward leveraging the vast supply of human manipulation video for scalable robot learning. As hand pose estimation, video inpainting, and generative rendering continue to improve, the quality of ego2robot-synthesized data will further increase, potentially breaking the robot data scale bottleneck entirely.