Skip to content
RobotWorld
Back to Papers

PAPER DEEP DIVE

具身智能PaperEmbodied AI

Data Pyramid for Embodied Manipulation

Multimodal foundation models learned to see and to speak by consuming the whole internet. Embodied agents admit no such shortcut, since they require data that couple observations with physical states and actions. These signals can be provided, to varying degrees, by multiple data sources. In this work, we organize the embodied data ecosystem as a "pyramid" spanning five complementary sources: real-robot data, UMI-style data, egocentric and exocentric data, simulation data, and general vision-language data. We organize the pyramid around the tension between scalability and robot alignment, and further characterize each source in terms of data quality, diversity, reusability, and physical fidelity. We then analyze recent embodied foundation models through the lens of their data recipes, examining how different sources are selected, aligned, and mixed during pretraining. For embodied brain models, vision-language-action models, and world-action models alike, we relate data composition to capabilities in perception, reasoning, planning, action generation, and world prediction. We close by discussing six open challenges: building large-scale tactile datasets, collecting failure and recovery data, developing scalable data-collection pipelines, aligning actions across embodiments, leveraging egocentric data for dexterous manipulation, and designing principled data recipes for robot learning. We hope this work paves the foundation for the design of next-generation embodied systems.

Yifan Ye, Yankai Fu, Yaoxu Lv, Bohan Hou, Jun Cen, Lingdong Kong, Duo Zheng, Tianxing Chen, Jiaming Liu, Ziang Cao, Yunfan Lou, Wei Chow, Xian Sun, Yingshuo Wang, Kuangzhi Ge, Xiaowei Chi, Xidong Zhang, Zhibo Pang, Yiwu Zhong, Sirui Han, Zhihe Lu, Weihao Yuan, Qifeng Chen, Michael Yu Wang, Yao Mu, Ziwei Liu, Jianfei Yang, Ping Luo, Shanghang ZhangJuly 27, 20266 min read
中文

1. Overview: Data Pyramid for Embodied Manipulation

This is a data-centric survey of embodied intelligence that defines an embodied data pyramid organized primarily by scalability and robot alignment, with quality, diversity, reusability, and physical fidelity as complementary dimensions. From apex to base, the pyramid contains five data categories with distinct strengths and limitations: real-robot data provides the most direct supervision for executable behavior and physical interaction, but is costly and hard to scale; UMI-style data supports portable real-world collection with explicit end-effector and gripper supervision, but requires calibration, retargeting, and embodiment-specific control; egocentric and exocentric data captures diverse human activities, hand-object interactions, and long-horizon task structures, limited by the human-robot embodiment gap; simulation data enables controllable and parallel generation of robot-oriented experience with privileged supervision, constrained by environment coverage and sim-to-real gap; general multimodal data offers broad semantic, perceptual, and reasoning priors at large scale, but provides limited grounding in robot actions, contacts, and physical outcomes.

Data pyramid organization overview

2. Core Problem: The Data Dilemma of Embodied Foundation Models

Multimodal foundation models have acquired broad perceptual, linguistic, and reasoning capabilities by scaling pretraining over abundant visual and linguistic data. Extending this paradigm to embodied foundation models requires more than processing visual and linguistic inputs: embodied agents must understand physical states and dynamics, reason about how actions transform the environment, and execute appropriate behaviors in the physical world. These requirements fundamentally change the nature of supervision needed for pretraining, raising a central question: what data should be used to train embodied foundation models? The community has explored complementary supervision sources: observation-action trajectories from physical robots and simulation, egocentric/exocentric recordings of human interaction, general image/video/language/VL data, and UMI-style demonstrations. Each source provides different forms of semantic, temporal, physical, and action-related supervision, but their respective roles, trade-offs, relationships, and integration strategies remain insufficiently systematized. Existing surveys tend to focus on model architecture rather than data.

Evolution of data scale

3. Method: Five-Layer Data Pyramid

3.1 Real-Robot Data (Apex)

Real-robot trajectories are collected via teleoperation, scripting, and human-in-the-loop enhancement, providing the most direct physical supervision. Teleoperation methods include hand retargeting, kinesthetic teaching, and vision-based feedback. Data is formalized as state-action pairs:

$$\mathcal{D}_{\text{real}} = \{(o_t, a_t, o_{t+1})\}_{t=1}^{T} \tag{1}$$

where $o_t$ is the observation (multimodal perception + proprioception) and $a_t$ is the action (joint positions/velocities/torques or end-effector poses). Strengths: physically grounded, directly executable; limitations: high cost, hardware dependency, limited scale.

3.2 UMI-Style Data

UMI (Universal Manipulation Interface) collects real-world manipulation demonstrations via handheld gripper devices without requiring a robot. The key innovation is the relative trajectory action representation:

$$a_t = T_{\text{rel}}(p_{t+1}, p_t) = p_{t+1} \ominus p_t \tag{2}$$

where $p_t$ is the end-effector pose and $\ominus$ is the relative pose difference on the Lie group. This representation enables cross-embodiment deployment—the same demonstration can be mapped to different robots. Limitations: requires calibration, retargeting, and embodiment-specific control adaptation.

Evolution of data utilization

3.3 Egocentric and Exocentric Data

Captures natural human demonstrations including hand-object interactions and long-horizon task structures. Supervision construction is categorized into semantic (activity recognition, object detection), geometric (hand pose, object pose), multimodal (vision-language-tactile fusion), and robot-oriented (action proxy extraction) types. Action proxy extraction is formalized as:

$$a_t = \text{Proxy}(o_t, \mathcal{M}_{\text{track}}) \tag{3}$$

where $\mathcal{M}_{\text{track}}$ is the motion and pose tracking model. Strengths: large scale, high diversity; limitations: human-robot embodiment gap (kinematic differences between human hands and robot grippers).

3.4 Simulation and General Data

Simulation data provides controllable robot-oriented experience via parallel generation with privileged supervision (perfect states, depth, segmentation). Simulation data generation efficiency is proportional to parallel environment count:

$$N_{\text{sim}} = N_{\text{env}} \times R_{\text{steps}} \times T_{\text{sim}} \tag{5}$$

where $N_{\text{env}}$ is the parallel environment count, $R_{\text{steps}}$ is steps per environment, and $T_{\text{sim}}$ is simulation time. Simulation data is efficient but constrained by the sim-to-real gap. General data (images, video, language, VL) provides large-scale semantic and perceptual priors. General data supervision quality is quantified as:

$$Q_{\text{general}} = \alpha \cdot S_{\text{semantic}} + \beta \cdot D_{\text{diversity}} + \gamma \cdot A_{\text{accessibility}} \tag{4}$$

where $S_{\text{semantic}}$ is semantic richness, $D_{\text{diversity}}$ is diversity, $A_{\text{accessibility}}$ is accessibility, and $\alpha, \beta, \gamma$ are weights. Limitation: weak action grounding—general data captures neither proprioception nor contact physics.

Action-trajectory diversity visualization

4. Data Applications in Embodied Foundation Models

4.1 Action Representations and Cross-Embodiment Transfer

Action representations determine data compatibility and cross-embodiment transfer capability. Discrete action tokens are compatible with language models but lose continuous control precision; continuous action heads (e.g., diffusion policies, flow matching) preserve precision but have weaker cross-data compatibility. The relative trajectory representation $\Delta p_t = p_{t+1} \ominus p_t$ is key to cross-embodiment transfer—it decouples absolute coordinate frame dependency.

Data TypeScalabilityRobot AlignmentPhysical FidelityAction Grounding
Real-robotlowhighesthighestdirect
UMI-stylemidhighhighrelative trajectory
Ego/exocentricmid-highmidmidproxy extraction
Simulationhighhighcontrollabledirect
General multimodalhighestlowlowweak
UMI and real-robot data overview

4.2 Data Recipes and Model Analysis

Recent embodied foundation models (PaLM-E, RT-2, GR00T, Motus, InternVLA-A1) are analyzed from the perspective of data recipes. VLA models evolve from discrete tokens to continuous action heads, with diffusion policies and flow matching becoming mainstream. Embodied brain models, VLA models, and world models each have different data requirements.

ModelYearTypeData Strategy
PaLM-E2023VLAMultimodal pretraining
RT-22023VLAVision-language + robot
GR00T2026Embodied foundationPyramid data recipe
Motus2026Embodied foundationHierarchical data
InternVLA-A12026VLAMulti-source fusion
Real-robot dataset statistics

5. Limitations

  • Insufficient data evaluation standardization: current lack of unified data quality evaluation benchmarks makes it difficult to quantitatively compare contributions across data types.
  • Incomplete cross-embodiment transfer theory: while relative trajectories partially decouple embodiment dependency, a complete cross-embodiment transfer theory framework remains immature.
  • Sim-to-real gap: simulation data is scalable but physically fidelity-limited; sim-to-real transfer remains an unsolved challenge.

6. Conclusion

This work provides a data-centric synthesis of embodied intelligence, defining an embodied data pyramid organized by scalability and robot alignment. From the apex real-robot data (most direct supervision but hard to scale) to the base general multimodal data (most scalable but weak action grounding), five layers each have distinct strengths and limitations. UMI-style data's relative trajectory representation is a key innovation for cross-embodiment transfer; simulation data provides scalable privileged supervision but is constrained by sim-to-real; general data provides semantic and reasoning priors. The evolution of action representations from discrete tokens to continuous action heads determines data compatibility. The core insight: the data strategy for embodied foundation models is not "more is better" but "hierarchical complementarity"—pyramid layers are irreplaceable, real-robot data anchors physical feasibility, general data provides semantic breadth, intermediate layers bridge the gap, and the real challenge is effective cross-layer integration rather than single-layer scaling.

flowchart TD
    A["General multimodal data: most scalable, weak action grounding"] --> B["Simulation data: scalable, sim-to-real gap"]
    B --> C["Ego/exocentric data: mid scale, embodiment gap"]
    C --> D["UMI-style data: portable collection, needs calibration"]
    D --> E["Real-robot data: most direct supervision, hard to scale"]
    E --> F["Embodied foundation model"]
    A --> F
    G["Action representation: discrete token -> continuous diffusion/flow matching"] --> F
    H["Relative trajectory: delta_p = p_{t+1} minus p_t"] --> D
    F --> I["Cross-embodiment + physical feasibility + semantic breadth"]
The data strategy for embodied foundation models is not "more is better" but "hierarchical complementarity"—pyramid layers are irreplaceable, real-robot data anchors physical feasibility, general data provides semantic breadth, intermediate layers bridge the gap, and the real challenge is effective cross-layer integration rather than single-layer scaling.

Related Papers

Humanoid Seated Locomotion on Passive Mobile Chair

Humanoid Seated Locomotion on Passive Mobile Chair

Research on omnidirectional seated locomotion for humanoids on passive chairs. Policies use proprioception only, outperforming standing in tracking. Analyzes regularization effects on convergence.

人形机器人Humanoid坐姿移动Aug 28, 2026
GigaBrain-0.7: Scaling Embodied Foundation Models to Emergent Capabilities with a Three-System Architecture

GigaBrain-0.7: Scaling Embodied Foundation Models to Emergent Capabilities with a Three-System Architecture

Vision-language-action (VLA) models have become a dominant paradigm for generalist embodied agents, demonstrating strong complex and long-horizon task completion in structured settings. Yet it remains an open question whether current VLA systems can benefit from more effective architectural design, scale to substantially larger and more heterogeneous data regimes, and achieve broader generalization across tasks and embodiments. To this end, we present GigaBrain-0.7, an embodied foundation model with substantially improved generalization across diverse robot embodiments. Specifically, GigaBrain-0.7 unifies understanding, prediction, and action through a three-system architecture, scales pretraining to over 37,000 hours of heterogeneous embodied data, and introduces one-stage alignment training that jointly optimizes vision-language understanding and multi-embodiment action generation. Compared with the preceding GigaBrain-0 series and prior state-of-the-art models including $π_{0.5}$, GigaBrain-0.7 achieves substantial improvements in foundation zero-shot capabilities, language-conditioned instruction following, and post-training task success rates. In particular, on our in-house Maker H01 platform and mainstream robot embodiments, GigaBrain-0.7 demonstrates strong task adaptability and completion ability across both home and industrial scenarios. All training code and pretrained model weights will be released.

VLA具身智能世界模型Aug 16, 2026
ABot-N1: Toward a General Visual Language Navigation Foundation Model

ABot-N1: Toward a General Visual Language Navigation Foundation Model

ABot-N1 is a general visual-language navigation foundation model built on a slow-fast dual-system architecture: a 4B slow VLM reasoner emits explicit chain-of-thought plus dual pixel goals (Target Pixel + Affordance Pixel), while a 2B fast action expert decodes continuous SE(2) waypoints via QFormer action queries. The unified pixel-goal interface covers five tasks — point-goal, object-goal, POI-goal, instruction-following and person-following — in a single 30M-sample multi-task checkpoint, further aligned by GRPO post-training with format/target/safety rewards. Two closed-loop benchmarks are released (ABotN-PointBench and ABotN-POIBench). ABot-N1 sets new SOTA on all five benchmarks, boosting POI entrance arrival to 77.3% (+35.0 pp) and reaching 92.9%/95.4% outdoor/indoor point-goal SR, with full deployment on the TuTu quadruped running on a Jetson AGX Orin.

视觉语言导航VLN导航基础模型Jul 11, 2026
Zetta ζ: An Efficient Closed-Loop Embodied Harness for Self-Evolving Physical Intelligence

Zetta ζ: An Efficient Closed-Loop Embodied Harness for Self-Evolving Physical Intelligence

Zetta is a closed-loop embodied harness that keeps the base VLA frozen and evolves code-based runtime critics and recovery skills through three timescale-separated loops: action-frequency governance, rollout-batch failure diagnosis, and validation-gated skill updates. With the Z-Infra rollout infrastructure (20.6x throughput), it reaches 90.8% on LIBERO-Pro and 93.6% on RoboCasa, with zero-shot skill transfer and robotic Aha moments.

具身智能Embodied AIVLAAug 17, 2026