Pre-training Visual Dexterity in Simulation

Authors: Sarthak Kamat*, Adam Rashid* (co-first), Satvik Sharma, Aseem Doriwala, Chelsea Finn, Phillip Isola, C. Karen Liu · Stanford University / MIT / Scale AI · arXiv:2608.15917v1 · Project page: spd.bot · Code status: the paper pledges to release the spd-75h dataset, the spd-vr VR teleoperation software, and six tuned scenes; as of v1 no public code repository is linked.

One-Sentence Summary

SPD lets human operators manipulate virtual objects inside a VR headset, amassing 75 hours of on-embodiment dexterous data in a single week; a diffusion transformer pre-trained on this simulation data and fine-tuned with only one to two hours of real demonstrations beats from-scratch behavior cloning on all five real bimanual dexterous tasks.

Background and Motivation

The pre-train-then-fine-tune recipe has become the standard route to data-efficient robot policies, but the large multi-task datasets behind it are overwhelmingly built around parallel-jaw grippers. Multi-fingered dexterous hands face a harsher data famine: the hardware is expensive and fragile, real teleoperation throughput is low, and every hour of physical demonstration carries wear-and-tear cost.

Off-embodiment data sources do not solve the problem cleanly. Egocentric human hand video is abundant, but contact-rich hand-object interaction is heavily self-occluded, so pose reconstruction is noisy; even motion-capture gloves reproduce neither the contact points nor the actuation of a robot hand, yielding demonstrations the robot cannot faithfully execute. Handheld interfaces such as UMI remove the robot from the loop, yet covering every degree of freedom of a dexterous hand without an embodiment gap remains an open hardware problem.

Physics simulation offers a third route: data collected by teleoperating the target embodiment inside a simulator is action-labeled by construction, resets are instantaneous, and collection can be parallelized and decentralized. The open question SPD poses is whether human-in-the-loop simulation data is realistic enough to pre-train policies that transfer to real contact-rich dexterous manipulation.

SPD answers with two aligned teleoperation systems. spd-vr streams a MuJoCo simulation of the target bimanual dexterous robot to a Meta Quest 3 headset; five operators collect roughly 2,000 episodes (75 hours) across six scenes and hundreds of object instances in one week. spd-teleop is the real-world mirror used for post-training data, sharing the same top and wrist camera placements, the same pair of dexterous hands on 6-DoF arms, and a similar object collection, so the sim-real gap is attacked at the data level rather than left to the policy.

The result is a clean experimental claim: pre-training on simulation teleoperation data, then fine-tuning on one to two hours of physical demonstrations per task, outperforms training the same architecture from scratch on the real data alone across five bimanual tasks, and the gains concentrate in the reactive policy configuration the ablations identify.

Preliminaries

Behavior cloning (BC) trains a policy to imitate demonstrations directly; its performance scales with demonstration quantity and coverage, which is precisely what dexterous hands lack. Pre-training amortizes the cost of learning visuomotor representations on broad data so that downstream fine-tuning needs far fewer target-task demonstrations.

SPD's policy generates actions with flow matching. A noise sample $x_0 \sim \mathcal{N}(0,I)$ is interpolated toward the clean action chunk $x_1$ along $x_t = (1-t)\,x_0 + t\,x_1$ with $t \sim \mathcal{U}[0,1]$, and the network is trained to predict the constant velocity field $v = x_1 - x_0$. At inference, integrating the learned velocity from pure noise synthesizes actions, capturing the multimodal action distributions that dexterous manipulation demands.

Architecturally the policy follows the $\pi_0$-style diffusion transformer: a shared transformer trunk processes observation tokens while a separate action expert denoises action chunks. SPD drops language conditioning (its data has no dense language annotations) and instead conditions on visuomotor history.

Method in Detail

The full pipeline is shown in the Mermaid diagram below: VR teleoperation produces the pre-training corpus, batch rendering and augmentation turn raw trajectories into training-ready video, flow-matching pre-training produces the checkpoint, and a small real-world corpus collected with spd-teleop fine-tunes it for deployment on the 56-DoF bimanual platform.

flowchart TD
  A[Operator in Quest 3 headset] --> B[MuJoCo sim 480 Hz
IK-driven arms and hands]
  B --> C[spd-75h dataset
1930 episodes 75 h 6 scenes]
  C --> D[Madrona batch rendering
mask-based tint and texture swaps
left-right symmetry augmentation]
  D --> E[Diffusion transformer pre-training
flow matching 170k steps Muon]
  E --> F[Fine-tune on 1-2 h real demos
collected with spd-teleop]
  F --> G[56-DoF bimanual platform
YAM Pro arms + Sharpa Wave hands]
  G --> H[Five real tasks
plates mugs jenga cups bottles]
  E --> I[Best config w=32 c=8
history conditioning + short chunks]

Figure: the SPD pipeline, from VR teleoperation in MuJoCo to real-world fine-tuning on the bimanual dexterous platform.

In spd-vr, all computation runs on a host workstation; the headset is only a WebXR client that renders scene meshes streamed at 60 Hz and returns hand poses over a USB tether. The simulation steps at 480 Hz with the implicitfast integrator, elliptic friction cones, and one no-slip iteration, while control, streaming, and recording run at 60 Hz. The detected wrist pose and fingertip positions drive the simulated arms and hands through inverse kinematics, and the robot arms are rendered translucent to minimize occlusion. Hand-object contacts are physically simulated, so every trajectory carries exact robot actions with zero pose-estimation loss.

Data diversity comes from open-ended task design. The six scenes (spelling blocks, dishes, mugs, bottles, cups, Jenga bricks) are paired with a task registry in which each task defines a natural-language prompt, a target duration, and a reset function that randomizes asset selection, object placement, and physical properties; every sampled value is logged for exact episode reconstruction. Tasks specify outcomes, not strategies, so the same prompt elicits visibly different manipulation plans (Figure 2). A three-button foot pedal gives operators checkpoint, pause, and revert control, and checkpoints are rejected while a hand is in contact, so reverting always restores a contact-free state.

Figure 2: distinct strategies produced by the operator pool on the same open-ended tasks (spelling, Jenga stacking, domino chains).

Post-processing is where simulation shows its rendering advantage. Spans with more than ten seconds of no hand-object contact are cut, and all clips are rendered in parallel at 224x168 with a MuJoCo Warp adaptation of the Madrona batch renderer, preserving instance segmentation masks. The masks drive cheap but powerful visual randomization: random object tinting and background/table texture swaps applied as GPU transforms after data loading. A symmetry augmentation swaps the two arms and reflects images, proprioception, and actions, effectively doubling the bimanual coverage of the corpus.

The real platform is a 56-DoF bimanual system: two upgraded YAM Pro arms, each ending in a 22-DoF Sharpa Wave hand, with the arm's J3/J4 motors replaced by 40:1 gearings to cope with the hands' weight. Three RealSense D405 cameras (one top, two ulnar wrist) mirror the VR viewpoints. Real teleoperation mirrors the VR retargeting but replaces headset hand tracking with Manus gloves (25 keypoints per hand) for fingers and a Quest controller for the wrist. Each finger's fingertip position in a palm-centric frame is mapped into the robot palm frame by a per-operator affine map $p_r = A\,p_h + b$ fit by least squares in a short calibration routine; the five fingertip targets then drive mocap-based IK in a fixed-base hand simulation to produce the 22 joint commands. Wrist targets use mink-based differential IK with four QP iterations per tick under posture, joint-limit, and velocity tasks.

The policy is a 222M-parameter diffusion transformer over an interleaved token stream: at every 30 Hz timestep one 56-D proprioception token and one 56-D previous-action token; every eighth timestep, four pooled visual tokens per camera plus an 8-step noised action chunk. Each training sequence spans 256 timesteps, and all chunks in the sequence are denoised in parallel under a causal mask with independent per-chunk flow times, amortizing the long context over 32 chunk predictions.

Visually, each camera frame is encoded by a frozen DINOv3 ViT-B/16 (86M parameters); four learnable queries pool the patch bank into four tokens via cross-attention, and these pooled tokens re-attend to the raw patches through camera-specific cross-attention blocks every two trunk blocks, making the pooling context-dependent. The trunk is an 8-block transformer with hidden size 768, 12 heads, and MLP expansion 4; all attention is causal with rotary temporal positions, and every layer uses sliding-window attention over a 32-timestep window so deployment can run a fixed-length rolling KV cache. Following $\pi_0$, the action-denoising expert keeps its own 58M parameters.

Training supervises the flow-matching velocity objective $\mathcal{L} = \mathbb{E}\left[\lVert v_\theta(x_t, t, c) - (x_1 - x_0) \rVert^2\right]$ where $c$ denotes the visuomotor history context. Chunk tokens receive a Gaussian Fourier embedding of the flow time $t$ plus a sinusoidal embedding of the position within the chunk, each through a two-layer MLP. To reduce the distribution shift of conditioning on its own history, proprioception and action inputs are perturbed with i.i.d. Gaussian noise $\epsilon \sim \mathcal{N}(0, \sigma^2 I)$, $\sigma = 0.03$, during training. Optimization uses Muon for matrices (AdamW for the rest) at a constant learning rate of $10^{-3}$, weight decay 0.1, batch size 64, an EMA with 20-step half-life, and 170k pre-training steps.

At deployment the transformer runs as an incremental engine over a rolling KV cache matched to the 32-timestep training window. Each control tick appends the current observation tokens; at chunk boundaries the engine integrates the flow ODE with 10 Euler steps, $x_{t+\Delta} = x_t + \Delta\, v_\theta(x_t, t)$, and emits the next 8 actions. The ablated design choice is the pair $(w, c)$: sliding-window size $w \in \{1, 32\}$ and chunk size $c \in \{8, 32\}$, with $w=32, c=8$ emerging as the configuration that combines temporal coherence from history with reactivity from short chunks.

Experimental Results

Five bimanual tasks are evaluated with objects similar but not identical to those seen in pre-training: plate racking, mug hanging (after a bimanual handover), Jenga playing, cup stacking, and bottle tossing (Figure 4). Each task is fine-tuned on 44-121 minutes of real demonstrations (Table 2), and each checkpoint runs 20 trials scored against per-task rubrics; progress is the achieved score normalized by the task maximum.

Figure 4: autonomous real-robot rollouts of the five evaluation tasks.

Training regime	w	c	plates	mugs	jenga	cups	bottles
SPD pre-trained	1	8	31.9	35.0	0.0	0.0	0.0
SPD pre-trained	1	32	55.6	58.3	5.0	15.0	36.2
SPD pre-trained	32	8	80.6	93.3	85.0	55.6	68.8
SPD pre-trained	32	32	44.4	78.3	16.7	36.9	70.0
BC from scratch	1	8	16.2	38.3	0.0	0.0	0.0
BC from scratch	1	32	36.9	60.0	18.3	17.5	33.8
BC from scratch	32	8	66.9	80.0	65.0	35.0	47.5
BC from scratch	32	32	41.9	70.0	30.0	41.2	51.2

Table 1: average task progress (%) for every sliding-window and chunk-size variant (paper Table 1).

In the chosen configuration (w=32, c=8), the SPD pre-trained policy leads the from-scratch baseline on every task: 80.6 vs 66.9 on plates, 93.3 vs 80.0 on mugs, 85.0 vs 65.0 on jenga, 55.6 vs 35.0 on cups, and 68.8 vs 47.5 on bottles. The stage-by-stage curves of Figure 5 show the pre-trained checkpoint reaching later manipulation stages more often, and its training loss starts and converges lower, consistent with prior evidence that BC training loss, unlike validation loss, tracks downstream performance.

The ablations are the most instructive part of the paper. With a single frame of context (w=1), shrinking the chunk to c=8, the $\pi_0$-style default of a one-second chunk notwithstanding, collapses performance and produces visibly shaky behavior; history conditioning removes this trade-off, and with w=32 the 8-step chunk becomes the strongest variant in both training regimes. Crucially, this reactive configuration is also the one that extracts the most from pre-training: its average progress improves by 18 points over its from-scratch counterpart, versus 3 points or less for the other three variants. Pre-training, in other words, pays off precisely when the policy must re-plan frequently while staying temporally coherent.

Scene	Episodes	Minutes
Jenga (tower, dominoes, handovers)	557	2,178
Spelling blocks	275	976
Mugs	406	491
Dishes	208	394
Cups	120	223
Bottles	350	253
Total (listed tasks)	1,916	4,516

Table 2: spd-75h composition per scene, aggregated from paper Table 2 (durations at 30 Hz).

Task	Fine-tune steps	Real data (min)	Episodes
bottles	6k	72	270
plates	6k	70	161
cups	10k	121	217
jenga	6k	48	193
mugs	6k	44	238

Table 3: real-world fine-tuning data per task (paper Table 5).

Figure 1: top, the 75-hour simulation corpus across six scenes; bottom, real-world tasks adapted with one hour of demonstrations.

Limitations

The authors state that SPD inherits the fidelity of its simulation scenes: if object masses, friction, or contact responses deviate from real-world expectations, operators will demonstrate strategies that transfer poorly, and the current corpus is limited in scene and object diversity, with real evaluation objects chosen to be similar to simulated ones.

Beyond the authors' own list, the evaluation measures normalized progress over short trials rather than long-horizon autonomous throughput, so gains in stage completion are not yet evidence of sustained autonomous operation. The findings are also tied to a single embodiment pair (YAM Pro arms with Sharpa Wave hands); whether the same simulation corpus pre-trains policies for other dexterous hands is untested. Finally, the deliberate omission of language conditioning means the pre-trained policy cannot be instructed at task level, limiting its role to a fine-tuning prior.

Conclusion and Outlook

SPD provides the first clean evidence that simulation teleoperation is a viable pre-training source for real-world dexterous manipulation: on-embodiment, action-labeled, robot-free, and scalable to 75 hours in a week with five operators. The released spd-75h corpus, spd-vr software, and six tuned scenes lower the barrier for follow-up work.

The authors sketch three directions: mixing simulation teleoperation with real teleoperation and egocentric human video in a single pre-training corpus; scaling scenes, objects, and hours to characterize the generalization that simulation data itself provides; and using the pre-trained policy as an initialization for reinforcement learning, which scales with compute rather than operator time. For the dexterous-manipulation community, the more immediate message is methodological: before collecting more real hand data, align your simulator with your robot and let humans play in it.

Golden Quote

"The most scalable teacher for dexterous manipulation may not be the real world, but a simulator that resets instantly and a headset that never fatigues."