Open-AoE: An Open Egocentric Manipulation Dataset and Toolchain for Embodied Learning

Figure 1 — Open-AoE overview. Independent modules for smartphone capture, structured data processing, and downstream robot learning, supporting 2,000 hours, 500+ contributors, 400+ device models, 400+ scenes, 8,000+ tasks.

1. Background & Motivation

Embodied foundation models are increasingly constrained by real-world interaction data scale and quality. Unlike language models learning from internet text, robot learning requires physical demonstrations capturing how humans perceive scenes, move hands, contact objects, and complete temporally extended tasks. Such data must be large, diverse, and structured in geometry, time, and semantics.

Egocentric video is a natural scalable modality: recording from the actor's viewpoint captures hands, objects, scenes, and action progress in one visual stream. It's closer to robot execution's perceptual structure than third-person video, and collectable more naturally and cheaply than teleoperation.

Existing datasets each have shortcomings:

Open-AoE is the only dataset combining low-cost continuous capture, manipulation-level structured annotations, cross-embodiment retargeting, and downstream training recipes.

2. Core Method

Figure 2 — Data processing pipeline: from raw smartphone recordings to structured samples through temporal action segmentation, semantic annotation, hand reconstruction, and camera trajectory reconstruction.

2.1 Smartphone Capture

500+ contributors, 400+ smartphone models, natural environments. Kitchens (24.8%), tabletop/indoor (23.3%), offices (17.8%), bedrooms (9.1%), plus living rooms, workshops, outdoors. The 100-hour random sample contains 32,407 action descriptions, 175 verbs, 8,030 object strings, 135 scene labels. Long-tailed contributor distribution: median 2.6 min/ID, top-10 only 13.7%.

Figure 3 — Collection, sensor, and semantic diversity in a 100-hour random sample. Nested donut: phone brand/model composition. Center: collection context and contributor coverage. Word clouds: action verbs, objects, task language.

2.2 Structured Data Processing

Figure 4 — Structured data processing modules: transforming raw recordings into aligned annotated samples.

Four-step pipeline:

• Temporal action segmentation: 99.99% temporal coverage, mean segment 9.64s, density 13.97 segments/min.
• Semantic annotation: natural-language action descriptions, object/scene labels.
• Hand reconstruction: MANO-based dual-hand 55D params, concatenated as 110D: $2\times[\mathrm{valid}(1)+\mathrm{wrist\ trans}(3)+\mathrm{wrist\ axis\text{-}angle}(3)+\mathrm{velocity}(3)+\mathrm{MANO\ pose}(45)]$.
• Camera trajectory reconstruction: SLAM-estimated camera poses synchronized with hand motion.

2.3 Downstream Toolchain: AoE-Training-Ready

Figure 5 — AoE-Training-Ready: projecting synchronized segments into action representations with different supervision densities, connecting to VLA policies, World Action Models, and World Models.

Routes: policy learning & cross-embodiment (VITRA, H-RDT, GR00T, SmolVLA, $\pi_{0.5}$, ACT, DP) and world-action & action-conditioned video (DreamZero, Ctrl-World, iVideoGPT).

flowchart TB
    subgraph Capture["Smartphone Capture"]
        A[500+ contributors
400+ phone models] --> B[2000h raw video]
    end
    subgraph Process["Structured Processing"]
        B --> C[Temporal Action Segmentation
99.99% coverage]
        B --> D[Semantic Annotation
32K action desc]
        B --> E[Hand Reconstruction
MANO 110D]
        B --> F[Camera Trajectory
SLAM]
        C & D & E & F --> G[Aligned Structured Samples]
    end
    subgraph Downstream["AoE-Training-Ready"]
        G --> H1[VLA Policies
GR00T, SmolVLA, pi0.5]
        G --> H2[World Action Models
DreamZero, Ctrl-World]
        G --> H3[World Models
iVideoGPT, LAOM]
    end
    style G fill:#e0e7ff,stroke:#2563eb
    style E fill:#fef3c7,stroke:#d97706
    style H1 fill:#dcfce7,stroke:#16a34a

3. Experimental Results

3.1 Annotation Consistency

Visual-language evaluator scores each sampled frame $a_{q,t} \in [1,5]$. Sequence-macro score:

$$S_{\mathrm{macro}} = \frac{1}{Q}\sum_{q=1}^{Q}\left(\frac{1}{T_q}\sum_{t=1}^{T_q} a_{q,t}\right)$$

3.2 Training-Window Retention & Multimodal Completeness

Figure 9 — Training-sample yield and multimodal supervision availability across datasets: (a) candidate windows/hour and retention; (b) native hand-signal availability; (c) sequence-level availability of action annotations, bounding boxes, confidence, hand pose, camera pose.

Candidate windows with $h=2$s, $f=2$s, $\Delta=2$s:

$$N_q = \max\!\left(0,\, \left\lfloor\frac{L_q - h - f}{\Delta}\right\rfloor + 1\right), \quad \eta_{\mathrm{window}} = \frac{Y_{\mathrm{window}}}{3600/\Delta}$$

Open-AoE yields ~1,760 windows/hour, 97.8% retention. Hand signal: 98.93% at least one hand valid, 98.02% both valid. Only release with all five supervision modalities at near-universal coverage.

4. Main Contributions

• Low-cost large-scale community collection: 2000h, 500+ contributors, 400+ phone models — broadest consumer-phone device coverage for egocentric manipulation.
• Complete structured annotations: MANO hand pose + text + camera trajectory + temporal atomic actions, 99.99% temporal coverage, 4.583/5 consistency far exceeding peers.
• Cross-embodiment retargeting toolchain: Multi-interface action representations (110D/48D/62D/20D/22D) adapting to VLA/WAM/World Model full spectrum.
• Open ecosystem positioning: Not a static package but a community co-developed infrastructure for data, tools, and models.

5.

类别概率归一化

$$ p_{s,j}=\frac{\lambda_{s,j}}{\sum_{r}\lambda_{s,r}} $$

有效响应度

$$ R_{\mathrm{eff}}^{(s)}=\exp\!\left(-\sum_{j}p_{s,j}\log p_{s,j}\right) $$

kNN纯度度量

$$ M_{\mathrm{kNN}}^{(s)}=1-\frac{1}{N_{s}}\sum_{i:y_{i}=s}\frac{1}{k}\sum_{j\in\mathcal{N}_{k}(i)}\mathbf{1}\!\left[y_{j}=s\right] $$

Limitations & Future Work

Analysis:

Open-AoE's core value is connecting low-cost smartphone capture to downstream model training. Its 2000 hours, while less than EgoScale's 20,854, is unique in annotation completeness (MANO + camera trajectory + atomic actions) and toolchain usability (retargeting + training recipes). Camera-domain diversity (400+ models, multimodal FOV) provides natural sensor-domain randomization that may improve robustness to unseen cameras — but the paper honestly notes downstream benefit remains a training-side hypothesis, not a verified conclusion.

Limitations: no robot joint labels (cross-embodiment transfer relies on retargeting); evaluation is data-quality audit not downstream task benchmarks; 2000h is large but smaller than EgoScale. Future: community scaling to 10K+ hours, direct robot joint annotations, systematic downstream benchmarks.

6. Conclusion

Open-AoE is an open data infrastructure coupling ~2,000 hours of real-world egocentric manipulation data with a complete path from data to models. On the production side, the capture and processing pipeline turns raw smartphone video into structured samples aligned across vision, language, hand motion, camera trajectories, and action boundaries. On the consumption side, the open-source toolchain supports visualization, 4D reconstruction, cross-embodiment retargeting, and training representations for VLA policies, World Action Models, and World Models. Its broad coverage of scenes, actions, objects, participants, device models, and fields of view connects low-cost real-world collection with multiple embodied-learning paradigms.

Open-AoE aims to be the lowest-barrier egocentric data infrastructure for embodied intelligence — making egocentric data easy for anyone to collect and use, so that data is no longer the bottleneck for embodied foundation models.