Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models

Figure 1: Embodied-R1.5 at a glance. Top: performance across 24 embodied VLM benchmarks; bottom: the capability evolution path from foundation model to VLA.

Background: Three Bottlenecks for a Unified EFM

Large language models have transformed the digital world, but grounding intelligence in the physical world remains a central challenge. The authors argue that general physical intelligence requires an Embodied Foundation Model (EFM): all capabilities grounded in physics, with perception, reasoning, and execution unified in a single architecture. The required capabilities organize into three dimensions:

• Embodied Cognition & Spatial Reasoning — understanding the semantic and spatial structure of the physical world: spatial relations, metric reasoning (distances, sizes, whether free space accommodates an object), monocular depth and cross-view geometry, and robot-centric object/scene cognition;
• Embodied Planning & Correction — the full execution lifecycle: long-horizon task decomposition, next-step planning, plus process monitoring (completion detection), error localization, and error correction;
• Embodied Pointing & Location — grounding high-level reasoning into coordinates and trajectories: referring expression grounding (REG), region referring grounding (RRG), object functional grounding (OFG), and visual trace generation (VTG) for object flow and end-effector flow, extended to 3D traces.

Three bottlenecks stand in the way. (1) Fragmented capabilities — existing models each cover part of the spectrum; cross-dimension attempts (e.g., RynnBrain) still rely on separate models of different scales rather than true unification. (2) Multi-task conflict — long-form reasoning, trajectory prediction, and diverse grounding tasks differ drastically in output format; joint training suffers convergence difficulties where capabilities erode one another. (3) Lack of closed-loop autonomy validation — most work stays at Embodied QA, never verifying whether reasoning is truly grounded under long-horizon complex decision-making.

Method

Unified Capabilities and Architecture

Figure 2: Capability taxonomy and architecture. The three dimensions form a progressive reasoning chain from perception to decision to execution.

Built on Qwen3-VL-8B-Instruct, the 8B scale balances reasoning capability with deployment cost. All outputs — coordinates, trajectories, free-form reasoning — are expressed as plain-text token sequences, with coordinates normalized and trajectories emitted as ordered coordinate sequences. Compared with approaches using additional special tokens for coordinates, generating numeric coordinates as text yields more stable predictions at negligible vocabulary overhead; coordinates also persist as referenceable symbols across reasoning steps, maintaining visual anchoring.

It further extends to Embodied-R1.5-VLA: a lightweight DiT-based flow-matching action expert attached to the VLM backbone forms a dual-system architecture — System 2 for reasoning, System 1 for action generation. The action expert extracts vision-language features from intermediate VLM layers and generates continuous action sequences via action chunking. The core hypothesis: when the VLM has sufficiently internalized embodied reasoning, large-scale action pretraining becomes less critical — a stronger embodied backbone substantially reduces downstream action data requirements.

Three Automated Pipelines: 15B Tokens

The corpus integrates 34 datasets exceeding 15B tokens. Three automated pipelines fill critical gaps that open-source data does not cover:

Pipeline 1: 3D scene annotation (spatial reasoning). After integrating multi-view spatiotemporal, metric depth, and robot-view cognition data, fine-grained tabletop manipulation reasoning is still missing. ER1.5-Spatial (20K samples) is fully automated: 3D semantic scene graphs are reconstructed from real robot-scene RGB images by chaining dual-backend semantic understanding, metric-scale monocular geometry from MoGe-2, Grounded-SAM open-vocabulary instance segmentation, and RANSAC horizontal plane alignment — with quality control embedded at every stage — then spatial reasoning QA pairs are programmatically generated from the graphs.

Pipeline 2: Failure-aware annotation (planning and correction). Existing planning data only provides successful demonstrations. ER1.5-Correction (800K samples) is organized along two orthogonal dimensions: by stage (planning vs. execution failures) and by cognitive level (detection, localization, correction) — the three levels mirroring the full correction chain in closed-loop autonomy. Planning failures use five structured perturbation operators (step omission, redundancy, swap, object error, action replacement) applied to correct sub-task plans, each generating all three QA levels; execution failures combine video truncation, description replacement, and physics-engine perturbation injection. Sources span real (BridgeData V2, RoboFail, RoboFAC) and simulated (ManiSkill, GEMBench) settings, all subjected to perturbation validity verification, positive/negative balancing, and human sampling review.

Pipeline 3: Affordance and trajectory data (pointing). Real-world part-level annotation is prohibitively expensive, so scaling comes from augmentation and restructuring: functional part grounding data is synthesized in simulation (ManiSkill-PartNet, PRISM) while heterogeneous affordance annotations are reorganized into a unified OFG format. RegularRearrangement data presents regular patterns (stars, squares) with 1-2 elements missing, requiring the model to infer placement — fusing spatial reasoning with pointing. For trajectories, datasets with 3D end-effector poses are projected to the image plane; datasets without metadata use a fine-tuned Detectron2 end-effector detector to track gripper motion; object traces use Co-Tracker3. Notably, trace data does not require high-fidelity simulation — low-fidelity simulated data still generalizes strongly (semantic generalization), because VTG targets topological structure and directional semantics rather than rendering fidelity. The final ER1.5-Pointing dataset has 400K samples.

Two-Stage Training: SFT + Multi-Task Balanced RFT

Stage 1 (SFT): full-parameter fine-tuning from Qwen3-VL-8B, with all heterogeneous outputs represented as token sequences and jointly optimized under one causal LM objective for one epoch. The vision encoder is jointly trained with the LLM backbone without freezing — embodied visual distributions differ significantly from general pretraining data and require deep adaptation.

Stage 2 (RFT): applying GRPO naively to heterogeneous multi-task training introduces two imbalances — intra-task, group-level std normalization biases updates toward low-variance samples, under-optimizing medium-difficulty samples with the strongest learning signal; inter-task, vastly different reward scales let high-density tasks dominate gradients. The balanced recipe attacks both:

1. Difficulty-aware data filtering: filter the SFT corpus by rollout pass rates, retaining 200K medium-difficulty samples;
2. Dynamic filtering: degenerate groups where all rollouts receive identical rewards are automatically detected and masked, so gradients come only from discriminative samples;
3. Global batch reward normalization: advantages use the within-group mean (preserving relative ordering) with the std over the entire mixed batch (unifying gradient magnitudes) — no task labels or historical state required, more stable than EMA-GRPO's per-task moving averages when data volumes differ sharply;
4. Five reward families: exact match (0/1), IoU (spatial grounding), point distance (piecewise-linear decay), trajectory RMSE, and semantic similarity (reward-model scoring), with all continuous rewards using partial credit to avoid binary-reward sparsity;
5. Advantage and global-batch normalization: advantages are computed as $A_i = \frac{r_i - \bar{r}_{\text{group}}}{\sigma_{\text{batch}}}$ — the group mean preserves relative ordering while the batch std unifies cross-task gradient magnitudes;
6. Piecewise-linear decay rewards: given a distance metric $d$ and threshold pair $(\underline{d}, \bar{d})$, $$R(d) = \begin{cases} 1 & d \le \underline{d} \\ \frac{\bar{d} - d}{\bar{d} - \underline{d}} & \underline{d} < d < \bar{d} \\ 0 & d \ge \bar{d} \end{cases}$$ giving dense gradient signal;
7. IoU reward (spatial grounding): $$R_{\text{IoU}} = \frac{|B_{\text{pred}} \cap B_{\text{gt}}|}{|B_{\text{pred}} \cup B_{\text{gt}}|}$$
8. Trajectory RMSE reward (VTG): after interpolating predicted and ground-truth trajectories to equal sample counts, $$R_{\text{traj}} = f\!\left(\sqrt{\frac{1}{N}\sum_{k=1}^{N}\lVert \hat{\tau}_k - \tau_k \rVert^2}\right)$$ with the depth dimension of 3D traces evaluated by MAE and averaged equally;
9. Adaptive thinking: only the output format is constrained, no explicit reasoning chain is forced. Since the reward evaluates only the final answer, the model naturally learns to allocate computation on demand — emitting coordinates with near-zero reasoning overhead for perception-style pointing, while generating structured thinking for complex planning. This emergent adaptive allocation matches embodied requirements: high-frequency localization needs real-time response, lower-frequency planning benefits from thorough reasoning.

PGC Closed-Loop Autonomy

Figure 3: The Planner-Grounder-Corrector closed loop. A single Embodied-R1.5 instance serves as a unified inference service driving the full autonomy stack — no multi-model cascading, no multi-agent orchestration.

Demonstrated on "make a cup of milk tea": the Planner (high-level) decomposes the instruction into a structured sub-task sequence and performs next-step planning from the latest observation as execution progresses. The Grounder (low-level) adaptively orchestrates pointing capabilities per sub-task — "press the tea dispenser switch" first localizes the switch as a functional part via OFG, then plans a pressing trajectory via VTG — producing precise spatial commands for the low-level skill executor. The Corrector runs as an asynchronous query, assessing execution state (SUCCESS / PROCESS / FAIL) from the current observation and Memory history. Memory is a simple FIFO buffer with fixed-rate image sampling and per-sub-task status annotations. When a failure is detected, the error is written to Memory and fed back to the Planner, triggering retry or adaptive replanning.

The elegance is that all three roles are served by the same model instance: the Corrector directly leverages the reasoning context established during planning and the spatial understanding from grounding, avoiding cascade information loss and yielding more accurate error attribution. PGC itself is merely a lightweight stateless harness with no reasoning logic or heuristic rules — the system's ceiling is set solely by model capability and scales naturally as the model improves. Long-chain real tasks (making milk tea, sweeping garbage, stacking cups) complete autonomously without human intervention.

Experiments

Embodied VLM Benchmarks: 16 of 24 SOTA

Figure 5: Experimental evaluation overview. All results run through the unified EmbodiedEvalKit pipeline for reproducibility.

On planning and correction, RoboFAC 77.2% directly validates the failure data pipeline while EgoPlan-2 53.8% confirms egocentric planning. Pointing shows the widest margin: Part-Afford 82.9% and RoboAfford 80.0% confirm the OFG functional-part pipeline's gains, while visual trace leads on both object-centric flow (VABench-V RMSE 7.00) and robot-centric flow (ShareRobot-V RMSE 15.79). The cognition dimension is closer to top general-purpose VLMs but has no single weak benchmark; on 7 standard vision benchmarks, degradation from the Qwen3-VL-8B base is minimal, with MMMU (54.8 vs 52.0) and ScienceQA (93.1 vs 92.3) actually improving — embodied training does not compromise general understanding.

Simulation Manipulation: Small Action Data Beats Large-Scale Pretraining

Attaching a DiT-B action head (2048-dim VLM hidden states as cross-attention context plus 32 learnable future query tokens; flow matching generates 7-D continuous actions via 4-step Euler integration at inference), without any large-scale action pretraining, fine-tuned directly on benchmark data:

It leads on all four suites — over 20 points above π0.5 on SimplerEnv Google Robot, and 11 points above the specialized ManipLLM on PartNet-Mobility. In LIBERO-Plus per-perturbation breakdowns the advantage is largest under language and background perturbations — exactly the robustness to semantic and appearance variation that internalized embodied reasoning should confer. This is direct evidence for the paper's central claim: internalized embodied reasoning effectively substitutes for action data scale — once intent is understood upstream, the action head only needs to learn a simple mapping from intent to continuous actions.

PGC Real-Robot Closed Loop

Zero-shot real-robot experiments cover instruction following, affordance grounding, articulated object manipulation, and long-horizon complex tasks. Under PGC, long-horizon tasks such as making milk tea, sweeping garbage, and stacking cups complete without human intervention; a failed sub-task (e.g., missing the dispenser switch on the first try) is localized by the Corrector, replanned by the Planner, and succeeds on the second attempt — closed-loop correction is not decoration but a direct contributor to real-world long-horizon success.

Significance and Limitations

Three layers of significance. First, it shows an 8B model can surpass Gemini-Robotics-ER-1.5 and GPT-5.4 on embodied reasoning, with the entire stack open-sourced — the barrier to embodied foundation models is dropping fast. Second, the hypothesis that "internalized reasoning substitutes for action data scale" is empirically supported across four suites, offering a route around the expensive robot-data arms race for VLA training. Third, PGC demonstrates a minimal viable form of single-model closed-loop autonomy: with capabilities unified in one model, the agent's ceiling is bounded by model capability rather than system-engineering complexity.

On limitations: the main text does not dwell on failure modes, but the experimental design leaves open questions — VLA action generation is still validated primarily in simulation, and Sim-to-Real long tails, the simplified FIFO Memory design, and the boundaries of semantic generalization from low-fidelity simulated trajectories remain to be answered by future work.