RoboInter1.5: A Holistic Intermediate Representation Suite for Embodied World Modeling and Robotic Manipulation

1. Overview

With rapid advances in foundation models, both Vision-Language-Action (VLA) policies and embodied world models have progressed significantly. Yet existing robot datasets remain expensive to curate, embodiment-specific, and lack fine-grained intermediate representation annotations needed for generalizable reasoning, execution, and long-horizon dynamics simulation.

RoboInter1.5 extends the prior RoboInter1.0, providing a holistic suite of intermediate representations spanning data, benchmarks, and models. The core idea treats intermediate representations as a bidirectional interface connecting semantic understanding, physical execution, and future simulation — both regularizing low-level action spaces and constraining latent rollouts of open-world physical simulators.

2. Dataset: RoboInter-Data

RoboInter-Data contains over 230k manipulation episodes across 571 scenes with dense per-frame annotations covering 10+ types of intermediate representations.

Data sources include In-the-Wild (diverse indoor scenes from Droid, RH20T, OXE) and Table-Top (tabletop interactions from RH20T). The RoboInter-Tool semi-automatic annotation tool performs:

• Task decomposition & keyframes: 15 predefined primitive skills, ChatGPT preliminary labels, human verification
• Object recognition: SAM2 segmentation and tracking with async review
• End-effector localization: calibration matrix + gripper detection + point tracking for 2D trace reconstruction
• Grasp annotation: affordance box and contact points from contact frame
• Placement annotation: object position at subtask end as target placement
• Gripper annotation: anchor points projected 3D→2D forming gripper bounding box

Statistics: ~61M-frame object annotations, ~70M-frame gripper traces, 190k affordance/placement proposals, 760k language clip annotations.

3. RoboInter-VQA

Annotations are converted into embodied VQA tasks organized along two axes:

• Spatial VQA: understanding (3 selection + 1 judgment) and generation (5 prediction tasks: object box, grasp pose, placement, keypoints, gripper box)
• Temporal VQA: understanding (5 selection + 4 judgment) and generation (trace generation, multi-step planning under varying context completeness)

Statistics: ~1M spatial generation, 172k spatial understanding, 131k temporal generation, 935k temporal understanding entries. 7,246 videos as evaluation pool.

4. Method

4.1 RoboInter-VLM Planner

Base VLMs (Qwen2.5-VL-3B/7B, LLaVA-OneVision-7B) fine-tuned on RoboInter-VQA to generate Functional Chain-of-Thought (F-CoT) content, flexibly combining subtasks, skills, object boxes, affordance boxes, motion traces, etc.

4.2 RoboInter-VLA: Plan-Then-Execute

Three paradigms supported:

• RoboInter-IC-E2E: VLM generates F-CoT, directly consumed by DiT head (implicit integration)
• RoboInter-EC-E2E: Joint optimization of action prediction and CoT generation (explicit joint)
• RoboInter-Modular: Planner generates F-CoT, Executor conditions on it (modular), supporting textual (Te-Modular) and visual-prompted (Im-Modular) F-CoT

4.3 RoboInter-World: Controlled Imagination

Uses intermediate representations as structured conditioning signals for controllable future world state prediction. Denoising model:

$$f_{\theta}(x_{t},t\mid z^{\mathrm{hist}},c,a,u)$$

where $z^{\mathrm{hist}}$ = observed history latents, $x_{t}$ = noisy future latent at diffusion timestep $t$, $c$ = language instruction, $a$ = robot action sequence, $u$ = control video from intermediate representations.

Optimization objective:

$$\mathcal{L}_{\mathrm{world}}=\mathbb{E}_{y,\epsilon,t}\left[\left\|v_{t}-f_{\theta}(x_{t},t\mid z^{\mathrm{hist}},c,a,u)\right\|_{2}^{2}\right]$$

where $v_{t}$ is the flow-matching target from noisy latent $x_{t}$ to clean latent $y$.

Key design: control video constructed by rendering object sample points and gripper traces onto a blank canvas, leveraging pretrained vision model spatial priors. Mask latent distinguishes history/future content. Stochastic perturbations on intermediate representations during training enhance robustness to imperfect planner-generated controls at inference.

5. Architecture

flowchart TB
    subgraph Data Layer
        RD[RoboInter-Data
230k episodes, 10+ types]
        RVQA[RoboInter-VQA
Spatial + Temporal]
        RCV[RoboInter-CV
Control Video Dataset]
    end
    RD --> RVQA
    RD --> RCV
    subgraph Model Layer
        VLM[RoboInter-VLM
Planner generates F-CoT]
        VLA[RoboInter-VLA
IC-E2E / EC-E2E / Modular]
        WM[RoboInter-World
Controlled World Model]
    end
    RVQA --> VLM
    VLM -->|F-CoT| VLA
    RCV --> WM
    WM -->|Predicted future latents| VLA
    style RD fill:#4f9eff,color:#fff
    style VLM fill:#ff6b6b,color:#fff
    style WM fill:#51cf66,color:#fff

6. Experimental Results

6.1 Planner Benchmarking

On Where2Place, RoboRefIt, and RoboVQA benchmarks, RoboInter-VLM substantially outperforms base models. At 3B scale, RoboInter-Qwen-3B improves 49.1% over RoboBrain2.0 on RoboRefIt and 12.7% on RoboVQA. At 7B scale: 76.8% and 42.8% respectively.

6.2 Intermediate Representation Ablation

Coarse representations (subtask, primitive skill) yield only marginal improvements; spatially grounded representations (object box, gripper box, affordance) yield larger gains; Trace introduces dense temporal information with the most significant improvement.

6.3 Closed-Loop Real-World Evaluation

On Franka Research-3 arm, 4 tasks (Object Collecting, Cup Stacking, Towel Folding, Clutter Cleaning), 15 ID + 15 OOD trials each. RoboInter-IC-E2E: ID success 77.3% (vs Vanilla 65.0%), OOD 58.3% (vs 38.3%). EC-E2E shows only 8.3% ID→OOD degradation (vs IC-E2E's 19%), demonstrating stronger generalization.

6.4 World Model Benchmarking

Replacing raw action control with intermediate representations (Seg+Trace): 14B model PSNR 18.26→21.05, LPIPS 0.171→0.102. Gains are larger with limited history context (H1P3: +1.50 PSNR) and most significant for long prediction horizons (H4P16: +2.79 PSNR).

6.5 World Model Guides VLA

Feeding RoboInter-World's predicted latent features to the VLA executor yields consistent and substantial gains across all thresholds, approaching the Oracle upper bound. Standard world model baseline (I2V-Baseline) predictions can be harmful (degrading performance under strict thresholds), showing unstructured predictive features introduce noise.

7.

世界模型训练损失

$$ \mathcal{L}_{\mathrm{world}}=\mathbb{E}_{y,\epsilon,t}\left[\left\|v_{t}-f_{\theta}(x_{t},t\mid z^{\mathrm{hist}},c,a,u)\right\|_{2}^{2}\right] $$

上下文特征拼接

$$ h^{\mathrm{ctx}}=\left[\phi_{\mathrm{text}}(c);\phi_{\mathrm{act}}(a)\right] $$

策略动作输出

$$ \mathbf{a}_{t}=f_{a}\bigl(f_{r}\bigl(f_{v}(\mathbf{o}_{t}),\ell\bigr)\bigr) $$

Conclusion

RoboInter1.5 provides a unified intermediate representation platform: RoboInter-Data sets a new standard with 230k densely annotated episodes; RoboInter-VQA systematically evaluates VLM embodied understanding and generation; RoboInter-VLA integrates geometric priors into modular and end-to-end control frameworks; RoboInter-World leverages these representations for accurate future dynamics prediction. Experiments demonstrate that intermediate representations — especially dense temporal traces — bring significant gains across reasoning, execution, and world modeling, with the three mutually reinforcing.