EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data

Background And Motivation

Dexterous manipulation is not an extended version of pick-and-place. Card sorting requires isolating one card from a tightly stacked deck without lifting several cards at once. Bottle unscrewing requires a stable grasp, coordinated finger motion, and at least three continuous rotations. Tongs use begins before the tool contacts the fruit: the fingers must establish a sufficiently firm grasp on the tong, rotate it into a usable configuration, squeeze at the right time, and release only after the object reaches the target. Failure can occur at contact timing, wrist orientation, tool geometry, or sequencing.

Teleoperation can capture these behaviors, but it turns every new task into a human labor problem. Collecting 100 demonstrations for one tabletop task may be feasible in a lab; collecting enough demonstrations to cover deformables, tools, containers, fasteners, and long-horizon service tasks is not. Robot-collected data is accurate, but its acquisition cost caps diversity. That cap matters because generalization is not a property of a single trajectory; it depends on the breadth of objects, viewpoints, hand configurations, and task structures that the policy has seen.

Human video is attractive because humans perform manipulation continuously across homes, stores, repair shops, kitchens, workshops, and public spaces. A person can fold clothing, wrap a package, sort cards, use a tool, prepare food, or repair a device without a robot station. These videos contain rich hand-object contact and natural long-horizon structure. They are also messy: the camera moves, the hand may be occluded, the scene changes, and the exact action intention is not always labeled.

The field already has evidence that human-to-robot transfer works, but the evidence has often been narrow. Many prior systems use tens or hundreds of hours. Some transfer only wrist motion and collapse the hand to a gripper. Others align observations but remain tied to small task suites. Those results demonstrate a useful mechanism but leave the scaling question open. If human data is truly a scalable source of physical supervision, increasing it should produce a measurable improvement in fine-grained robot control, not just a qualitative resemblance to human motion.

EgoScale takes that question seriously. The authors assemble more than 20,000 hours of egocentric human data, supervise both wrist motion and finger articulation, and measure validation loss and downstream robot performance as data volume changes. They find a log-linear relationship between data scale and hand-action validation loss, with an R squared of 0.9983 in the explored regime. More importantly, that offline loss tracks real robot task completion. This is what separates the paper from a single successful demo: it treats human video as a quantifiable training resource.

The method still needs alignment. Human videos do not provide proprioception, calibrated robot viewpoints, or a control interface. EgoScale therefore adds a small mid-training set in which humans and robots perform the same tabletop play tasks with matched cameras and motion capture. The large data layer supplies diversity and reusable motion structure; the small aligned layer makes that structure executable. This division of labor is the central design insight.

Foundation: Coordinate Systems In Egocentric Video

A naive model trained on first-person video may learn the wrong causal structure. When the camera is mounted on the head, head motion changes both the image and the coordinates of the hand in camera space. If the policy predicts absolute hand position in camera coordinates, it can confuse voluntary manipulation with involuntary body motion. EgoScale avoids that trap by explicitly separating camera pose, hand pose, and relative wrist motion.

Let $\mathcal{F}_{w}$ denote the world frame and $\mathcal{F}_{c}^{t}$ the camera frame at time $t$. SLAM estimates the camera pose $\mathbf{T}_{w\leftarrow c}^{t}\in\mathbb{SE}(3)$. A human hand is represented by 21 keypoints, and each keypoint has a rigid transform $\mathbf{H}_{c,i}^{t}\in\mathbb{SE}(3)$ in camera coordinates. The wrist is the first keypoint, so its world-frame pose is:

$$\mathbf{W}_{w}^{t}=\mathbf{T}_{w\leftarrow c}^{t}\mathbf{H}_{c,1}^{t}$$

Within an action chunk, wrist motion is represented relative to the initial wrist pose:

$$\Delta\mathbf{W}^{t}=(\mathbf{W}_{w}^{0})^{-1}\mathbf{W}_{w}^{t}$$

This relative transform is the primary arm action. It is invariant to global camera displacement, while remaining a physically meaningful end-effector displacement. A robot arm can also execute incremental end-effector commands, so human wrist motion and robot arm control meet in a shared abstraction. The shared abstraction is more valuable than forcing both domains into absolute joint angles.

Finger control uses a different boundary between observation and action. The human keypoint stream is an observation; the Sharpa hand consumes 22 joint angles. EgoScale retargets 21 human hand keypoints into the 22-DoF joint space through an optimization-based procedure that enforces joint limits and kinematic constraints. Appendix D says the robot hand is modeled with URDF-based forward kinematics, a nonlinear program is solved per hand per frame, CasADi and IPOPT are used, and a first-order exponential filter smooths the result.

This choice is not cosmetic. A fingertip trajectory preserves geometry but does not guarantee a valid joint configuration after inverse kinematics. A joint-space target is directly executable and preserves the high-dimensional structure of pinch, grasp, and release. The paper later compares wrist-only, fingertip-based, and retargeted joint-space supervision; joint-space actions perform most consistently.

Figure 2: The aligned mid-training setup uses a head camera, two wrist cameras, Vive trackers, and Manus gloves so human and robot signals share the same collection logic.

Method

Data Layer: Scale And Alignment Are Separate Problems

Stage I uses 20,854 hours of egocentric human video. Most recordings are in the wild and span 9,869 scenes, 6,015 tasks, and 43,237 objects. EgoDex contributes 829 hours of tabletop manipulation with comparatively accurate wrist and hand tracking. All recordings are RGB at 30 FPS. Off-the-shelf SLAM recovers camera motion, and hand-pose estimation recovers hand trajectories. The paper is explicit that these estimates are noisy because the data is unconstrained.

That noise does not invalidate the signal. Each short clip may have tracking drift, missing fingers, or imperfect camera pose. At scale, however, the same manipulation concepts recur under many object geometries, hand sizes, viewpoints, and lighting conditions. The model has an opportunity to learn stable statistics of reaching, grasping, folding, rotating, and tool use instead of memorizing one robot table. Appendix C makes the diversity concrete: retail and consumer goods account for 20.1% of categorized hours, fashion for 11.8%, repair services for 11.5%, food and beverage for 11.5%, home environments for 9.5%, construction and hardware for 7.7%, food processing for 6.7%, and printing and design for 4.4%, followed by a long tail.

The environment and task distributions are also long-tailed. Frequent environments include homes, flower shops, electronics repair shops, furniture repair shops, woodworking shops, and clothing stores. Frequent tasks include folding clothes, cleaning shoes, potting plants, ironing, assembling boxes, arranging flowers, sanding wood, and food preparation. Common objects include cardboard boxes, plastic bags, bowls, trays, tape, clothes, dough, batteries, and circuit boards. This is not a tidy simulation vocabulary; it is a broad set of contact-rich, real-world behaviors.

Figure 3: Category, environment, task, and object distributions are long-tailed rather than concentrated in a small synthetic task family.

Stage II is deliberately small: 344 tabletop manipulation tasks, approximately 30 human trajectories and 5 robot trajectories per task, about 50 hours of human data, and only 4 hours of robot data. The collection hardware aligns the two domains. Vive trackers capture wrist pose, Manus gloves record 25 hand joint transforms, and a head camera plus two wrist cameras provide consistent visual streams with calibrated intrinsics. This set does not add large task diversity. Its purpose is precise human-robot correspondence.

The division is clear. Stage I decouples manipulation knowledge from a particular robot, while Stage II binds it to robot sensing and actuation. Without Stage I, Stage II has too little diversity. Without Stage II, Stage I has no bridge to proprioception and robot control. The two stages make scale useful instead of leaving it as an ungrounded visual representation.

Architecture: Shared Representation, Embodiment-Specific Interfaces

EgoScale follows a flow-based VLA design similar to GR00T N1. At time $t$, the observation is:

$$o_{t}=(I_{t},l_{t})$$

where $I_t$ is an image and $l_t$ is a language instruction. The vision-language component encodes the observation into $\phi_t$. A DiT action expert then predicts a future action chunk with a flow-matching objective. This formulation lets the policy represent multimodal contact strategies rather than regressing to the mean of all valid actions.

Robot data includes proprioceptive state $q_t$; human data does not. Replacing $q_t$ with a learnable placeholder token keeps the model architecture unified. The network does not need a human-only branch, and the robot does not need to discard a useful state channel. The missing channel is learned as a domain-specific token rather than pretending the two domains have identical sensing.

Different robots also have different state and hand spaces. Input and output interfaces use lightweight embodiment-conditioned MLP adapters. Input adapters encode proprioception, and output adapters decode hand actions. Relative wrist-motion prediction, the vision-language backbone, and the DiT action expert remain shared. This is why a policy pretrained in a 22-DoF hand space can be adapted to a 7-DoF tri-finger hand without rebuilding the entire policy.

Figure 4: Hand-pose extraction and retargeting feed the action encoder, while the pretrained VLM and DiT action expert remain shared across embodiments.

Training Schedule

The three stages control where parameters are allowed to move. Stage I has broad but noisy supervision, so it fully unfreezes the VLA and trains for 100,000 steps on 256 GB200 GPUs with a global batch of 8,192 and learning rate $5\times 10^{-5}$. Stage II is small and precise, so the vision-language backbone is frozen; training runs for 50,000 steps with batch size 2,048 and learning rate $3\times 10^{-5}$, updating the vision encoder and DiT action expert. Stage III is task-specific and runs for 10,000 steps with batch size 512 and learning rate $3\times 10^{-5}$.

Stage	Data	Steps	Global batch	Learning rate	Key plasticity
Stage I pretraining	20,854 hours of egocentric human data, including 829 hours of EgoDex	100K	8,192	$5\times 10^{-5}$	256 GB200 GPUs; full VLA update
Stage II mid-training	344 aligned play tasks: about 50 hours human and 4 hours robot	50K	2,048	$3\times 10^{-5}$	Freeze VL backbone; update vision encoder and DiT
Stage III post-training	Task-specific robot demonstrations	10K	512	$3\times 10^{-5}$	Freeze vision encoder when mid-training was used

The schedule encodes a useful engineering lesson. Large noisy data needs enough parameter mobility to shape representations. Small aligned data should not be allowed to rewrite the whole representation, because it would overfit to a few tabletop scenes. Task post-training should adapt the last control pathway while preserving the broad prior. This is a direct answer to the common failure where a small fine-tuning set erases a large pretrained model's generality.

Robot Systems And Control

Galaxea R1Pro is a dual-arm wheeled humanoid. During experiments the base and torso are fixed, and control focuses on two 7-DoF arms in relative end-effector space. Actions specify incremental position and orientation changes, matching the human wrist representation. Each arm drives a Sharpa Wave hand with 22 degrees of freedom under joint-space control.

Perception uses three RGB cameras. The head camera provides a first-person view consistent with human video. Two wrist cameras face the palms and provide close-range feedback for finger-object contact. Human pretraining relies primarily on egocentric imagery; the wrist cameras compensate for the fine visual detail needed at the contact point during robot execution.

Figure 5: R1Pro uses OAK-D-Wide head and OAK-1-Wide wrist cameras with 22-DoF Sharpa hands; G1 uses 7-DoF tri-finger hands.

End-To-End Flow

flowchart LR
  A["First person RGB
30 FPS"] --> B["SLAM
camera pose"]
  A --> C["Hand pose estimation
21 keypoints"]
  B --> D["SE(3) transforms
wrist in world"]
  C --> D
  D --> E["Relative wrist motion
delta W"]
  C --> F["Optimization retargeting
22 DoF joints"]
  E --> G["Human pretraining
20,854 hours"]
  F --> G
  H["Aligned play data
50h human + 4h robot"] --> I["Mid-training
robot sensing and control"]
  G --> I
  I --> J["Task post-training
100 demos per task"]
  J --> K["R1Pro evaluation"]
  I --> L["G1 adapters
tri-finger hand"]
  L --> M["Cross embodiment evaluation"]
  

Experimental Design

The evaluation has five tasks: Shirt Rolling, Card Sorting, Dexterous Tool Use with Tongs, Unscrewing a Bottle Cap, and Syringe Liquid Transfer. Each task has 100 teleoperated robot demonstrations except Shirt Rolling, a deformable task that uses 20. Each method is trained with two random seeds. Most checkpoints are evaluated over 10 trials; Tongs and Bottle add object-instance trials. An image-overlay initialization procedure gives the evaluator a visual overlay of the intended initial scene to reduce placement variability.

The scoring design matters as much as the success rate. Tasks with independent subskills use an additive rubric. Tasks with tightly coupled stages or deformable objects use a progress-based milestone rubric. Scores lie in $[0,1]$. Thus, a syringe trial is not simply 0 or 1: picking up the syringe, aiming at tube A, drawing liquid, re-aiming at tube B, injecting, transferring or releasing, and disposing are separate observable milestones.

Task	Required behavior	Training demonstrations	Scoring strategy
Shirt Rolling	Coordinate both hands to fold and roll a T-shirt, then place it in a basket	20	Milestone: initial fold 0.3; multiple folds or partial rolling 0.5; continuous compact rolling 0.8; basket placement 1.0
Card Sorting	Separate one card from a deck and insert it into the correct holder	100	Milestone: single card pickup 0.5; clean correct placement 1.0
Tongs Fruit Transfer	Grasp tongs, pick a fruit, place it in a basket, return tongs	100	Additive: grasp tongs 0.4; pick fruit 0.2; place fruit 0.2; return tongs 0.2
Bottle Cap Unscrewing	Grasp bottle, continuously rotate and remove the cap, place it down	100 total, four bottles with 25 each	Additive: grasp bottle 0.1; at least three rotations 0.5; fully remove cap 0.2; place cap 0.2
Syringe Liquid Transfer	Pick syringe, draw from tube A, inject into tube B, dispose syringe	100	Additive: pickup 0.1; aim A 0.1; draw 0.2; aim B 0.1; inject 0.2; transfer or release 0.2; dispose 0.1

Figure 6: From top to bottom: Shirt Rolling, Card Sorting, Tongs Fruit Transfer, Bottle Cap Unscrewing, and Syringe Liquid Transfer.

Result One: The Combination Beats Either Data Type Alone

The authors compare four checkpoints: training from scratch, pretraining only on the aligned human-robot play data, pretraining only on large-scale human data, and combining human pretraining with aligned mid-training. Across tasks, human pretraining improves average task completion by over 55% versus scratch. The full combination gives the best overall result and improves average success rate by 54% over the no-pretraining baseline.

The ordering is informative. Large human pretraining is noisy, not task-aligned, and not sensor-aligned, yet it already outperforms the mid-training-only baseline across most tasks. That means broad human motion contains an inductive bias strong enough to help even before precise embodiment alignment. It is not merely a visual pretraining signal.

The full model is still clearly better. Human pretraining supplies reusable manipulation structure, while mid-training grounds that structure in robot sensing and control. This is why 4 hours of robot data can have an outsized effect. It does not need to teach all manipulation from scratch; it only has to connect an existing action prior to the current embodiment.

For practitioners, this challenges a common shortcut: using human video only for visual representation learning. If human motion is converted into wrist and hand actions, the model can learn when fingers close, how tools rotate, and how two hands coordinate. Those are control-relevant patterns, not just appearance patterns.

Result Two: A Measured Scaling Trend

To test scale, the authors pretrain on 1k, 2k, 4k, 10k, and 20k hours of human data and then directly post-train each checkpoint on downstream tasks, isolating the effect of Stage II. Average task completion rises monotonically from 0.30 at 1k hours to 0.71 at 20k hours. No saturation appears in the explored regime.

The offline metric is not a casual validation loss. It uses 2,000 held-out egocentric episodes, samples 20 timesteps per trajectory, draws 16 samples from the flow-matching policy at each timestep, averages predicted action chunks, and computes mean squared error against ground-truth wrist and hand actions. At convergence, optimal validation loss is fit by:

$$\mathbf{L}=0.024-0.003\cdot\ln(D)$$

where $D$ is the number of human pretraining hours. The fit has $R^2=0.9983$. Models trained on 1k or 2k hours can initially reduce validation loss but later plateau or degrade, showing limited behavioral diversity. Models trained on 10k or 20k hours improve stably and monotonically.

The authors are careful not to extrapolate beyond the measured range. Still, within 1k to 20k hours, the offline loss closely tracks real robot performance. That correlation is the paper's most consequential result: the validation metric becomes a way to reason about embodied capability before running every expensive robot evaluation.

Setting	Human data	Observed metric	Result
Small-scale endpoint	1k hours	Average task completion after post-training	0.30
Large-scale endpoint	20k hours	Average task completion after post-training	0.71
Scaling fit	1k to 20k hours	Validation loss versus $\ln(D)$	$R^2=0.9983$
Full policy	20,854-hour pretraining plus aligned mid-training	Average success rate versus no pretraining	+54%

Result Three: One-Shot Adaptation Emerges From Mid-Training

The one-shot setting is stricter than normal post-training. Fold Shirt and Unscrewing Water Bottles do not appear in the mid-training data. Each target task receives only one robot demonstration, supplemented by 100 aligned human demonstrations. This tests whether the model can instantiate a new task from a minimal robot example after learning shared motion primitives.

Only the full combination succeeds. For Fold Shirt, mid-training alone reaches 0.25 task completion and 0.15 success; human pretraining alone reaches 0.65 completion and 0.35 success; human pretraining plus mid-training reaches 0.92 completion and 0.88 success. For Water Bottle, the two partial models score 0.10 completion and zero success, while the full model reaches 0.64 completion and 0.556 success.

Figure 7: Aligned mid-training exposes reusable motion structure, enabling one-shot transfer to unseen shirt folding and bottle unscrewing.

Task	Metric	Mid-training only	Human pretraining only	Human pretraining + mid-training
Fold Shirt	Task completion	0.25	0.65	0.92
	Success rate	0.15	0.35	0.88
Water Bottle	Task completion	0.10	0.10	0.64
	Success rate	0.00	0.00	0.556

Failures are usually partial. Shirt trials can complete an incomplete fold; bottle trials may remove the cap but fail to maintain a stable grasp afterward. The milestone scoring makes those differences visible. It also explains the mechanism: mid-training does not magically create zero-shot robot control, but it connects a broad prior to the robot's observation and action interface well enough that a single demonstration can specify a new task instance.

This is one of the most practical results in the paper. A policy with strong human pretraining but no aligned mid-training knows something about folding but cannot reliably execute it on the robot. A policy with mid-training but no broad human prior can align with the robot but lacks the motion structure to generalize. The combination reduces task data collection from roughly 100 demonstrations to one.

Result Four: Transfer To A Different Embodiment

The Unitree G1 test asks whether the prior is tied to the Sharpa hand. G1 has a shorter arm, a smaller reachable workspace, and a 7-DoF tri-finger hand. The human pretraining action space was still instantiated by the Sharpa hand, but mid-training includes a small amount of G1 play data, and the architecture uses embodiment adapters.

Two G1 tasks are evaluated. In Pen in Bin, the robot opens or picks up a canister, places it stably, picks up a marker, and puts the marker into the canister. In Dish Handover in Rack, three plates must be picked up, potentially transferred between hands, and placed upright in a dish rack. Each method uses two random seeds and 10 trials per seed. The lower body is handled by a separately trained Homie policy while EgoScale predicts upper-body commands.

Figure 8: On G1, human pretraining plus G1-aware mid-training clearly outperforms comparable data exposure without the full recipe.

G1 task	Metric	Mid-training only	Human pretraining only	Human pretraining + mid-training
Pen in Bin	Task completion	0.45	0.68	0.83
	Success rate	0.30	0.30	0.67
Dish in Rack	Task completion	0.54	0.58	0.88
	Success rate	0.20	0.20	0.50

The improvement cannot be explained by more G1 trajectories alone. A checkpoint pretrained and fine-tuned directly on the same G1 dataset, without prior human pretraining, does not reach comparable success. With the full recipe, success rate rises by 37 percentage points on Pen in Bin and 30 percentage points on Dish in Rack relative to the mid-training-only baseline. The authors also observe qualitatively smoother behavior.

The interpretation is careful and credible. Human pretraining learns reusable grasping and manipulation primitives: hand opening, closing, coordinated finger articulation, and object handover. G1 mid-training adapts those primitives to a different kinematic and control interface. This is not zero-shot embodiment transfer, but it is strong evidence that the learned prior is more portable than a task-specific Sharpa policy.

Result Five: Action Representation Controls What Scale Can Buy

The hand-action ablation is easy to underestimate. Wrist-only pretraining removes all finger-level supervision. It performs poorly on tasks that need precise articulation, such as Tongs and Card Sorting. Policies grasp tools too weakly, close prematurely, or contact objects at unstable heights.

Fingertip-based pretraining predicts $\mathbb{SE}(3)$ trajectories for the wrist and fingertips, followed by an MLP mapping to robot joint commands. It gives richer geometric supervision than wrist-only, but small fingertip errors can map to implausible joint configurations. That hurts Card Sorting and Bottle Cap Unscrewing, where contact timing and joint validity are decisive.

Retargeted joint-space supervision is the most consistent. This does not mean fingertip information is useless; it means the supervision should terminate in a controllable space. At 20,000 hours of data, the representation choice compounds with scale. A less executable action space can waste the benefit of more video, while a well-aligned joint space lets additional data translate into more stable behavior.

Limitations

The paper states several constraints directly. SLAM and hand-pose estimates are noisy because the large human corpus is collected without controlled instrumentation. The authors explicitly do not extrapolate the scaling law beyond 1k to 20k hours. They also identify future work in jointly scaling human data and model capacity and in incorporating weaker or unlabeled video with self-supervised objectives.

There are additional engineering boundaries. No public code or data pipeline is provided, so reproduction requires rebuilding SLAM, hand-pose estimation, CasADi/IPOPT retargeting, the flow-based VLA, and a 256-GPU training setup. The scaling relationship is between human data volume and validation loss on this action distribution; it is not yet a universal law across model sizes, robot platforms, or task families.

The one-shot result still uses one target robot demonstration, so it is not zero-shot. Cross-embodiment transfer is demonstrated on two G1 tasks, and the mid-training mixture includes G1 play data. The evidence supports efficient adaptation with embodiment-aware mid-training rather than immediate execution on an unseen robot.

Evaluation realism has tradeoffs. Two seeds, ten trials per seed for most tasks, and image-overlay initialization improve comparability, but dexterous manipulation is sensitive to small initial-condition changes. The overlay procedure is a reasonable experimental control, yet it does not remove the broader deployment problem of perception, placement uncertainty, and recovery from unexpected contact.

Finally, the compute footprint is high. Stage I uses 256 GB200 GPUs for 100,000 steps with batch size 8,192. The central finding is scalable in principle, but access to that much data and compute is itself a gate for reproduction.

Conclusion And Outlook

EgoScale's contribution is a coherent recipe rather than a single trick. Convert unconstrained human video into relative wrist motion and retargeted hand joints. Pretrain broadly so the model learns reusable manipulation structure. Add a small aligned human-robot corpus to ground that structure in robot sensing and control. Adapt only the embodiment-specific interfaces when hardware changes. The result is 54% higher average success rate than no pretraining, a clean scaling fit with $R^2=0.9983$, 0.88 one-shot success on shirt folding, and substantial gains on G1 with a tri-finger hand.

For robot developers, the actionable lesson is to stop treating human video as generic visual pretraining. If the downstream task is manipulation, the supervision should contain relative end-effector motion and finger articulation. Conversely, a small robot dataset should be reserved for grounding rather than forced to cover all task diversity. Human scale and robot precision are complementary, not competing.

The next decisive experiments would extend the curve across model capacity, additional embodiments, and more long-horizon tasks. Public release of the retargeting and data-processing pipeline would matter as much as another benchmark result, because the field needs to test whether human video can become a reusable foundation rather than a one-paper dataset.

Golden Lines

"Human data is valuable not because it looks human, but because it can be translated into executable robot motion."

"EgoScale puts the small robot dataset on the bridge between human scale and machine control, not at the center of every generalization problem."