PAPER DEEP DIVE
HiFi-UMI: Learning Deployable Manipulation Policies from High-Fidelity UMI Data Alone
Learning deployable manipulation policies is bottlenecked by the scarcity of data that is both high-fidelity and scalable. Real-robot teleoperation is accurate but costly to scale; robot-free UMI capture scales readily, and current practice uses the resulting data mainly for pre-training, adding a small real-robot "anchor" at post-training. We ask whether raising the fidelity of robot-free UMI data, rather than shrinking the real-robot fraction, can remove that anchor. We present HiFi-UMI, a portable UMI data-production system co-designed for trajectory accuracy, inter-gripper relative pose, synchronization, and field of view: head-mounted offline stereo-inertial SLAM, native rather than reconstructed relative pose, a shared microsecond GPIO trigger, and two wide-angle cameras per hand covering ~200 degrees. It reaches 3 mm workspace-local end-effector accuracy without external tracking infrastructure. Using this corpus, we demonstrate zero-robot post-training: a policy post-trained solely on HiFi-UMI demonstrations deploys directly on a real robot and matches in-domain teleoperation across three backbones spanning the vision-language-action and world-action-model families, with success-rate differences of -2.5, +3.1, and -0.6 percentage points on StarVLA-QwenPI, OpenPI-pi_0.5, and LingBot-VA; the strongest policy reaches 85% on a precision insertion task, even though the teleoperation baseline is collected in the evaluation scene and no HiFi-UMI trajectory is. Pre-training on 4,000 hours from the same corpus lowers action error on ten unseen tasks by 41% and, on StarVLA-QwenPI, raises real-robot success by a further 18.1 percentage points. We open-source HiFi-UMI-2K, 2,000 hours of microsecond-synchronized, ultra-wide-FoV demonstrations, each automatically reconstructed and validated through simulation replay, as a large-scale, high-fidelity resource for the robot-learning community.
1. Overview: Learning Deployable Manipulation Policies from High-Fidelity UMI Data Alone
HiFi-UMI presents an end-to-end high-fidelity portable data-collection system and validates a bold hypothesis — "zero-robot post-training": using only high-fidelity robot-free UMI data for post-training yields manipulation policies that deploy directly on a real robot without any teleoperated data. The system remedies four fidelity deficiencies of handheld UMI through hardware-software co-design: trajectory accuracy (head-mounted stereo offline SLAM, 3mm end-effector accuracy), inter-gripper relative pose (natively accurate), synchronization (GPIO hardware trigger, <40μs), and field-of-view coverage (two non-parallel stereo cameras covering ~200°). The pipeline has generated over 20,000 hours of data, retaining 96% of raw captures as robot-executable trajectories. Across three VLA/WAM backbones, UMI post-training matches in-domain teleoperation (differences of -2.5, +3.1, and -0.6 percentage points).
2. Core Problem: The Data Bottleneck for Deployable Manipulation Policies
The bottleneck for deployable manipulation policies is data, not model capacity. The dominant paradigm is real-robot teleoperation — producing perfectly embodied, directly trainable trajectories but at prohibitive scale cost: each hour requires the target robot, teleoperation rig, and a skilled operator. UMI (Universal Manipulation Interface) is the prominent low-cost alternative: a handheld instrumented gripper collecting in-the-wild demonstrations without a robot. But current handheld capture inherits fidelity deficiencies: visual SLAM drift causing inaccurate trajectories, single-camera limited FoV hindering depth perception, software sync causing cross-sensor timing offsets, and gripper morphology mismatch. These deficiencies are the fundamental reason robot-free data is confined to a pre-training role throughout the field — real-robot teleoperation is assumed necessary for the post-training that grounds a policy for deployment. HiFi-UMI asks: can sufficiently high-fidelity UMI data break this division?
3. Method: High-Fidelity Data Collection and Processing Pipeline
3.1 HiFi-UMI Capture Device
A head-mounted stereo camera enables offline SLAM for low-drift long-horizon trajectories, while natively accurate inter-gripper relative pose is obtained jointly with world-frame poses of both hands. A GPIO hardware trigger achieves microsecond-level cross-sensor synchronization. Two non-parallel stereo cameras cover approximately 200° ultra-wide FoV. Gripper morphology matches the target robot. End-effector poses are recovered in a shared world frame and converted to robot end-effector frames. Pipeline fidelity metrics: 3mm end-effector accuracy, <40μs synchronization, <2 dropped frames/hour, 98% trajectory reconstruction success, gripper-state error <0.1°.
3.2 Action Representation and Policy Backbones
Actions are represented in a robot-centric end-effector frame, predicting relative pose increments and absolute gripper opening. Each arm $j$'s future pose is expressed relative to the current observation pose, with translation in the anchor EE frame, orientation as Rotation6D, and absolute gripper target. Each arm contributes $3+6+1=10$ physical channels, 20 for bimanual. The future pose increment:
$$\Delta\mathbf{T}_{t_0,h}^{j} = (\mathbf{T}_{t_0}^{j})^{-1}\mathbf{T}_{t_0+\delta_h}^{j} \tag{1}$$where $\delta_h^{(m)}$ is backbone $m$'s native future offset, $h$ is the chunk index. Chunk rows share one measured anchor, not recursive. The end-effector error metric:
$$E_{\text{XYZ}} = 10^3\sqrt{\frac{1}{6H}\sum_{t=1}^{H}\sum_{b \in \{L,R\}}\|\hat{\mathbf{p}}_{t,b} - \mathbf{p}_{t,b}\|_2^2} \tag{2}$$Three backbones: StarVLA-QwenPI (Qwen3-VL-4B + π-style conditional flow-matching DiT action head, $H=20$ action steps, executing first $H_{\text{exec}}=10$), OpenPI-$\pi_{0.5}$ (PaliGemma + Gemma action expert, continuous flow matching), LingBot-VA (causal WAM predicting future video latents then inverse-dynamics action decoding). StarVLA-QwenPI flow-matching training — given ground-truth action chunk $a$ and Gaussian noise $\epsilon \sim \mathcal{N}(0,I)$, sample $u \sim \text{Beta}(1.5, 1.0)$, set $\tau = (s-u)/s$, $s=0.999$, train velocity field:
$$\mathcal{L}_{\text{FM}} = \mathbb{E}\left[\left\|v_\theta(a_\tau, \tau, z_t) - (a - \epsilon)\right\|_2^2\right], \quad a_\tau = (1-\tau)\epsilon + \tau a \tag{3}$$OpenPI-$\pi_{0.5}$ continuous flow-matching path, given context $c=(o,q,\ell)$, flow time $t$, construct $x_t = (1-t)a + t\epsilon$:
$$\mathcal{L}_{\text{FM}} = \mathbb{E}\left[\left\|v_\theta(x_t, t \mid c) - (\epsilon - a)\right\|_F^2\right] \tag{4}$$LingBot-VA decomposes joint prediction into future video latent prediction and inverse-dynamics action decoding:
$$\mathcal{L}_{\text{LingBot}} = \mathcal{L}_{\text{video}} + \mathcal{L}_{\text{action}}, \quad p_\theta(a_{t:t+H-1}, z_{t+1:t+K} \mid h_t, \ell) \tag{5}$$where $z_t = E_{\text{VAE}}(o_t)$ is the multi-view observation VAE latent, $h_t = (z_{\le t}, a_{<t})$ is video-action history. Pre-training scaling follows a power law:
$$\mathcal{L}_{\text{heldout}}(S) = \mathcal{L}_\infty + AS^{-\alpha} \tag{6}$$with pre-training distribution $\alpha=0.268$ ($R^2=0.993$), OOD $\alpha=0.095$.
4. Experiments
4.1 Zero-Robot Post-Training: UMI vs Teleoperation
Four tabletop bimanual tasks compare UMI-only post-training vs real-robot teleoperation. Three backbones (two VLAs + one WAM), 40 rollouts per task. VLA backbones show approximate parity, all within sampling noise. LingBot-VA: UMI aggregate 56.9% (91/160) vs teleoperation 57.5% (92/160), difference -0.6 pp. UMI-post-trained policies produce more natural, continuous large-amplitude motions, while teleoperation policies execute more incremental corrections.
| Backbone | UMI Post-Train | Teleop Post-Train | Diff (pp) |
|---|---|---|---|
| StarVLA-QwenPI | ~parity | ~parity | -2.5 |
| OpenPI-π0.5 | ~parity | ~parity | +3.1 |
| LingBot-VA | 56.9% | 57.5% | -0.6 |
4.2 UMI Data Scaling and Pre-training Effects
On Remote Insertion, UMI data scaling: 400 demos → 37.5% success, 800 → 65.0%, 3200 → 85.0%, 6400 → 82.5% (saturated). 4,000 hours of pre-training reduces offline action error on ten unseen tasks by 41% and raises real-robot success by 18.1 pp at matched post-training data. Task-family analysis shows rigid utensil interactions improve fastest, cloth folding slowest, reflecting pre-training corpus composition.
| UMI Demos | Success Rate | Phase |
|---|---|---|
| 400 | 37.5% | Low-data regime |
| 800 | 65.0% | Rapid improvement |
| 3,200 | 85.0% | Near saturation |
| 6,400 | 82.5% | Saturated |
5. Limitations
- Task scope: Zero-robot post-training evidence covers only four tabletop bimanual tasks and three backbones under scene-level distribution shift; generalization to other tasks, embodiments, and shifts remains untested.
- Not sample-matched: Without pre-training, UMI post-training uses roughly 10× more demonstrations than teleoperation, comparing practical data pipelines rather than per-trajectory efficiency.
- Fidelity not ablated: Fidelity is realized jointly as a design principle; individual factors (trajectory accuracy, synchronization, FoV) are not isolated through controlled degradation to quantify marginal contributions.
6. Conclusion
HiFi-UMI achieves end-to-end high-fidelity portable data collection through hardware-software co-design: head-mounted stereo offline SLAM provides 3mm end-effector accuracy, GPIO hardware trigger achieves <40μs cross-sensor synchronization, and dual non-parallel stereo cameras cover 200° ultra-wide FoV. The pipeline has generated over 20,000 hours, retaining 96% as robot-executable trajectories. The core experiment validates "zero-robot post-training": across three VLA/WAM backbones, UMI-only post-training matches real-robot teleoperation within ±3 percentage points, all within sampling noise. 4,000 hours of UMI pre-training reduces unseen-task action error by 41% and raises real-robot success by 18.1 pp. The core insight is that the limitation of robot-free data is fidelity, not the robot-free setting — when trajectories are accurate enough, synchronization precise enough, and FoV wide enough, UMI data can upgrade from a pre-training role to the sole directly deployable post-training source, without a teleoperation anchor.
flowchart TD
A["Operator holds HiFi-UMI device"] --> B["Head-mounted stereo: offline SLAM"]
B --> C["3mm end-effector accuracy"]
A --> D["GPIO hardware trigger"]
D --> E["<40μs cross-sensor sync"]
A --> F["Dual non-parallel stereo cameras"]
F --> G["~200° ultra-wide FoV"]
C --> H["High-fidelity trajectories"]
E --> H
G --> H
H --> I["Data quality validation"]
I --> J["96% retained as executable"]
J --> K{"Post-training mode"}
K -->|UMI-only post-train| L["VLA/WAM policy"]
K -->|UMI pre-train + post-train| M["Pre-train 4000h"]
M --> N["Action error -41%"]
N --> L
L --> O["Real-robot deployment"]
O --> P{"vs teleop post-training"}
P -->|StarVLA| Q["-2.5pp ≈ parity"]
P -->|OpenPI-π0.5| R["+3.1pp ≈ parity"]
P -->|LingBot-VA| S["56.9% vs 57.5% ≈ parity"]