OmniBrain-0: A Unified Agentic Model for Embodied Intelligence

Paper Information

Title: OmniBrain-0: A Unified Agentic Model for Embodied Intelligence

Authors and institutions: Xiaomi Embodied Intelligence Team, Tsinghua University, and University of Macau.

Version: arXiv 2610.36013v1, submitted October 26, 2026. The technical report contains 40 pages, 27 figures, and 8 tables.

Resources: paper · code repository · project page

Code status: The repository README states that code is not yet available. Its current public commit contains only the README and an asset preparation script. No training implementation, model weights, or checkpoints are released, so the reported experiments cannot yet be reproduced independently.

Executive Summary

OmniBrain-0 trains a compact 4B or 9B high-level planner to coordinate manipulation and navigation tools. It combines embodied pretraining, specialized teacher distillation, and multi-teacher on-policy distillation, then uses verifier-backed rubrics and reminders to grow additional experience.

Why the Problem Is Hard

A robot can possess a strong grasping policy and still fail a household task. The missing capability may be deciding that a drawer must be opened before an object can be reached, noticing that a failed grasp changed the object pose, selecting a different tool, or recognizing that the task is not actually complete. These are long-horizon decision problems rather than isolated perception or control problems.

One conventional answer is to train a larger vision-language-action policy end to end. Such a policy can be efficient inside a narrow distribution, but it often lacks explicit task decomposition and recovery behavior outside its training data. Another answer is to use a general multimodal agent as a high-level controller. This approach offers broad language and commonsense reasoning, but its token cost, latency, and dependence on external experience are poorly matched to repeated robot execution.

OmniBrain-0 starts from a different premise: keep specialized low-level tools, but move the unified intelligence into a learned high-level decision model. Perception, VLA, VLN, and motion tools retain their native control interfaces. The compact planner learns when to invoke them, what arguments to provide, how to interpret responses, and when to recover. The paper therefore places unification inside the decision distribution, not merely inside a tool API.

This decomposition has practical consequences. The planner does not need to output continuous actions, and the low-level tools do not need to understand the full task. The hard research question becomes how to transfer the task decomposition and recovery behavior of strong teachers into a small model without asking that model to imitate every detail of the teachers' language.

Formal Interface and Closed-Loop Execution

At step $t$, the controller receives visual observations, proprioceptive state, environmental task status, and interaction history:

$$o_t=(I_t,q_t,p_t),\qquad H_t=(h_t,M).\tag{1}$$

Here $I_t$ is the image observation, $q_t$ is the robot state, $p_t$ is task status exposed by the environment, $h_t$ is the within-episode history, and $M$ is an optional experience library. The formulation separates online evidence from reusable prior knowledge.

Conditioned on instruction $g$, the policy selects a semantic tool $k_t$ and invocation parameters $\xi_t$:

$$x_t=(g,o_t,H_t),\qquad u_t=(k_t,\xi_t)\sim\pi_\theta(\cdot\mid x_t),\qquad k_t\in P_e.\tag{2}$$

The selected tool executes and returns new observation and feedback, which condition the next decision. A single high-level action space can therefore coordinate tools with different observation formats and control rates. Importantly, the low-level skills remain responsible for motor behavior; OmniBrain-0 learns task-level orchestration.

flowchart LR
    G["Instruction"] --> P["4B or 9B planner"]
    O["Current observation"] --> P
    H["Interaction history"] --> P
    E["Optional experience"] --> P
    P --> S{"Choose semantic tool"}
    S --> P1["Perception"]
    S --> P2["VLA manipulation"]
    S --> P3["VLN navigation"]
    S --> P4["Motion control"]
    P1 --> F["Feedback and new state"]
    P2 --> F
    P3 --> F
    P4 --> F
    F --> P
    F --> C{"Task complete"}
    C -->|No| P
    C -->|Yes| T["Stop"]
  

Figure 1: A paper-specific representation of the closed-loop semantic tool interface in Section 3.

Turning Expert Logs into Student Supervision

Raw expert logs are not directly suitable for compact-model training. They can contain hidden simulator context, infrastructure failures, repetitive actions without new evidence, and verbose reasoning tied to the teacher's linguistic style. The paper processes them through normalization, quality control, reasoning reconstruction, and consistency validation.

A retained episode is represented as a temporally ordered multimodal trajectory:

$$\tau=(g,x_0,y_0,u_0,f_1,x_1,\ldots,y_{T-1},u_{T-1},f_T,x_T),\tag{3}$$

where $y_t$ is decision text, $u_t$ is the executed tool call, and $f_{t+1}$ is execution feedback. This representation preserves the causal chain from observation to decision to environmental response. It is richer than a sequence of successful actions because it exposes what the teacher did after failed contact or changed object state.

The pipeline keeps both canonical successes and successful recoveries. Canonical trajectories show the intended procedure. Recovery trajectories show re-localization, gripper adjustment, re-grasping, and continuation after failure. Repeated actions that produce no new information are classified as noisy and discarded. This distinction is central: recovery is a task-level capability, not an accidental exception.

Teacher reasoning is rewritten into four fields: phase, state, evidence, and decision. Phase identifies localization, grasping, transport, placement, verification, or recovery. State records the held object, gripper status, target, and destination. Evidence comes from visual observations and tool returns. Decision is aligned with the actual next tool call. The rewrite reduces stylistic imitation and prevents future information from appearing in supervision.

The resulting training material contains approximately 820,453 embodied VQA samples, 3,846 initial agent demonstrations, and 1,142 demonstrations added through recursive improvement. A final consistency check removes evidence-deficient trajectories and cases where state, action, or visual evidence disagree.

Figure 2: Paper Figure 4. Logs pass through normalization, quality control, structured reasoning reconstruction, and consistency validation.

Progressive Capability Transfer

Stage One: Establish a Shared Embodied Base

Qwen3.5-4B and Qwen3.5-9B are pretrained on a mixture of embodied VQA and long-horizon interaction data. VQA supplies object, spatial, affordance, and planning knowledge. Interaction trajectories teach the model how to organize multi-turn tool use and respond to feedback. The output, OmniBrain-Base, initializes both the specialist teachers and the eventual unified student.

Pretraining uses 16 NVIDIA H20 GPUs with bfloat16, two epochs, a batch size of one, four gradient accumulation steps, and a learning rate of $5\times10^{-6}$. The goal is not to solve either domain in isolation, but to create a common representation for later specialization and consolidation.

Stage Two: Distill Manipulation and Navigation Teachers

Two copies of OmniBrain-Base are trained as high-level specialist teachers. The manipulation teacher sees manipulation trajectories; the navigation teacher sees navigation trajectories. For teacher parameters $\phi_k$ and target sequence $z$, the objective is autoregressive negative log-likelihood:

$$L_{\mathrm{expert}}^{(k)}=-\mathbb{E}_{(x,z)\sim D_k}\left[\frac{1}{|z|}\sum_{j=1}^{|z|}\log\pi_{\phi_k}(z_j\mid x,z_{<j})\right],\qquad \phi_k^{(0)}=\theta_{\mathrm{base}}.\tag{4}$$

Only decision text and tool calls are supervised; images and feedback remain conditioning context. By learning from standard successes and recoveries, each teacher acquires domain-specific task ordering and correction behavior without replacing the low-level control policy.

Teacher training runs for three epochs with batch size two, learning rate $1\times10^{-5}$, and two gradient accumulation steps. The teachers are then frozen and used as distributional supervisors for a single student.

Stage Three: Consolidate Expertise with MOPD

The student starts from the shared base and is trained against the frozen teacher set

$$\theta^{(0)}=\theta_{\mathrm{base}},\qquad \mathcal{T}=\{\pi_{\phi_k}\}_{k=1}^{K}.\tag{5}$$

The important distinction from ordinary offline distillation is where supervision is evaluated. The student first samples a decision sequence from its own policy. A domain teacher then scores each next token on the same context and the student-generated prefix:

$$p_{\theta,j}(v)=\pi_\theta(v\mid x,\hat z_{<j}),\qquad q_{k,j}(v)=\pi_{\phi_k}(v\mid x,\hat z_{<j}),\qquad v\in V.\tag{6}$$

Here $\hat z$ is the student rollout and $V$ is a renormalized shared top-$k$ vocabulary subset. Teacher feedback therefore follows the actual state induced by the student. This is especially relevant for recovery: a teacher can correct a mistake after the student has made it, rather than only demonstrating the canonical path.

The joint loss aggregates domain-specific discrepancies with weights $\rho_k$ and masks non-decision tokens:

$$L_{\mathrm{MOPD}}(\theta)=\sum_{k=1}^{K}\rho_k\mathbb{E}_{x\sim\mu_k,\hat z\sim\pi_{\bar\theta}}\left[\frac{1}{M}\sum_{j=1}^{|\hat z|}m_jD_{\mathrm{KL}}(p_{\theta,j}\Vert q_{k,j})\right],\qquad M=\sum_{j=1}^{|\hat z|}m_j>0.\tag{7}$$

The divergence is reverse KL:

$$D_{\mathrm{KL}}(p_{\theta,j}\Vert q_{k,j})=\sum_{v\in V}p_{\theta,j}(v)\log\frac{p_{\theta,j}(v)}{q_{k,j}(v)}.\tag{8}$$

The student places more weight on modes that it already assigns probability to, while the teacher constrains those modes. Training uses two epochs, batch size two, learning rate $2\times10^{-6}$, temperature 1.0, top-$k$ equal to 64, and equal weights of 0.5 for the manipulation and navigation teachers.

Figure 3: Paper Figure 5. Specialist teachers are trained after embodied pretraining, and MOPD merges their decision behavior into one student.

Verifier-Backed Recursive Self-Improvement

The self-improvement loop does not simply train on model-generated text. It first runs tasks, verifies outcomes and efficiency, and passes execution evidence to a Meta-Agent. The Meta-Agent extracts reusable criteria from successful, failed, and inefficient trajectories, then turns those criteria into a state-dependent reminder policy.

The rubric and reminder updates are written as

$$U_{k+1}=A(D_k,U_k),\qquad R_{k+1}=G(U_{k+1}),\tag{9}$$

where $U_k$ is the current rubric set, $D_k$ is the newly collected evidence, $A$ analyzes and revises the rubrics, and $G$ constructs reminders. Rubrics describe what should be checked; reminders select the check most relevant to the current state.

At each decision step, the harness may insert guidance before a tool call:

$$r_t=R_{k+1}(U_{k+1},x_t),\qquad u_t\sim\pi_\theta(\cdot\mid x_t,r_t).\tag{10}$$

If no issue is detected, $r_t$ is empty. Otherwise it may remind the model about prerequisites, ordering, grasp verification, or recovery evidence. The paper freezes model parameters during reminder validation, which separates the effect of guidance from the effect of later weight updates.

Verified successful and recovered episodes are then added to the specialist corpora:

$$D_{\mathrm{SFT},k+1}^{(d)}=D_{\mathrm{SFT},k}^{(d)}\cup Q(T_k^{(d)}).\tag{11}$$

The domain experts pass through supervised training again, MOPD consolidates them, and the updated unified model is redeployed for another interaction round. The paper uses GPT-6.1-Sol as the Meta-Agent and GPT-5.6-Sol on the verifier side.

Figure 4: Paper Figure 7. Challenging cases feed rubric guidance, verified data collection, expert updates, and MOPD reconsolidation.

flowchart TD
    A["Execute on difficult tasks"] --> B["Verifier checks outcome and efficiency"]
    B --> C["Meta-Agent diagnoses evidence"]
    C --> D["Add or revise rubrics"]
    D --> E["Build conditional reminders"]
    E --> F["Collect targeted rollouts"]
    F --> G["Curate successful recoveries"]
    G --> H["Manipulation expert update"]
    G --> I["Navigation expert update"]
    H --> J["MOPD reconsolidation"]
    I --> J
    J --> K["Redeploy unified student"]
    K --> A
  

Figure 5: The recursive improvement process from Section 5 expressed as a paper-specific data flywheel.

Main Experimental Findings

The 9B model reaches 65.5% on LIBERO-Pro, 83.8% on LIBERO-Plus, 39.6% on RoboCasa365, 88.2% on RoboTwin 2.0, and 65.0% on the R2R-CE rand100 Val-Unseen subset. The recursive-improvement version improves these figures to 69.3%, 84.5%, 43.2%, 89.4%, and 66.0%, respectively.

The tradeoff against the closed-source teacher is consistent. OmniBrain-0 is below GPT-5.6-Sol on success, but the gap varies by benchmark and is often smaller than the reduction in execution cost. On LIBERO-Pro, the 9B model finishes in 1.6 minutes versus 13.8 minutes for the teacher. On LIBERO-Plus, both the 9B and recursive versions require 2.2 minutes versus 10.5 minutes.

The strongest result is RoboTwin 2.0. The 9B model reaches 88.2% and the recursive version reaches 89.4%, only 3.0 and 1.8 points below the teacher. The average time remains 3.7 minutes versus 10.5. This is evidence that the compact planner can absorb a large part of a teacher's task-level behavior when the tools and task structure are stable.

Navigation also transfers well but not perfectly. R2R-CE success rises from 34.0% for the few-shot Qwen3.5-9B backbone to 65.0% for OmniBrain-0 and 66.0% with recursive improvement. The teacher reaches 71.0%. Navigation error falls to 3.70 to 3.78 meters, and the model completes tasks in less than one minute on average.

RoboCasa365 reveals the opposite side of the result. Atomic-Seen performance is 71.1%, but Composite-Unseen is only 11.3%. The recursive version improves the unseen split to 15.0%, yet the gap to the teacher remains large. The model has learned recognizable task composition more effectively than wholly new combinations of skills.

Benchmark	OmniBrain-0 9B	With recursive improvement	GPT-5.6-Sol	9B time
LIBERO-Pro	65.5%	69.3%	78.3%	1.6 min
LIBERO-Plus	83.8%	84.5%	86.2%	2.2 min
RoboCasa365	39.6%	43.2%	52.4%	13.3 min
RoboTwin 2.0	88.2%	89.4%	91.2%	3.7 min
R2R-CE Val-Unseen	65.0%	66.0%	71.0%	0.86 min

Table 1: Main success rates compiled from Tables 1 through 5 of the report.

Figure 6: Paper Figure 2. The overview connects the training stages, tool execution loop, recursive guidance, and cross-domain results.

Spatial Reasoning and Physical Manipulation

OmniBrain-0 9B scores 81.6% overall on EmbSpatial-Bench, a 4.4-point gain over its Qwen3.5-9B backbone. It leads across MP3D, AI2-THOR, and ScanNet, which suggests that the improvement is not confined to one visual source.

The RefSpatial-Bench result is mixed. Placement accuracy improves from 41.0% to 48.0%, and unseen combinations improve from 36.4% to 41.6%. Direct object localization, however, falls from 65.0% to 52.0%, leaving overall accuracy at 47.7% versus the backbone's 48.4%. The training pipeline appears to favor relational placement and executable spatial decisions over raw point localization.

The real-world study includes block stacking, apple placement, and drawer opening. OmniBrain-0 obtains 75%, 60%, and 85%, for a macro-average of 73.3%. The standalone π0.5 policy obtains 50%, 40%, and 85%, for 58.3%. The teacher reaches 90%, 75%, and 95%, for 86.7%. The unified planner improves over the low-level standalone policy but still needs better contact handling and long-horizon correction.

Figure 7: Paper Figure 8. Three physical manipulation tasks compare OmniBrain-0, the standalone π0.5 policy, and the GPT-5.6-Sol teacher.

Efficiency, Ablations, and Evidence of What Matters

Token efficiency is one of the paper's strongest practical results. Relative to the teacher, the 9B model uses 4.5 times fewer tokens on LIBERO-Pro, 2.6 times fewer on LIBERO-Plus, 3.5 times fewer on RoboCasa365, 12.7 times fewer on RoboTwin 2.0, and 3.7 times fewer on R2R-CE. The savings are especially large on RoboTwin because success is close to the teacher while token use drops from 1.4M to about 0.11M per task.

The reminder-policy ablation freezes weights and changes only inference guidance. Success increases by 6.1 points on LIBERO-Pro, 1.3 on LIBERO-Plus, 7.0 on RoboCasa365, 1.0 on RoboTwin 2.0, and 2.0 on R2R-CE. The largest gain on RoboCasa365 matches the task's need for dependency tracking and progress verification.

Removing recovery trajectories from the data pipeline causes roughly a nine-point LIBERO-Pro drop. Using unconstrained teacher reasoning instead of structured reconstruction also degrades performance. These ablations connect the measured result to the paper's data design: removed failures reduce recovery supervision, while unstructured language encourages style imitation rather than state-grounded decisions.

The experience-library experiment is separate from the compact model result. Supplying the library to GPT-5.6-Sol improves LIBERO-Pro, RoboTwin 2.0, and RoboCasa365 by 27.7, 6.8, and 26.4 points. This shows that reusable cross-episode priors remain valuable even for a strong teacher, but the main OmniBrain-0 comparisons are reported without the external library at evaluation time.

Diagnostic	Measured change	Interpretation
Reminder on LIBERO-Pro	65.5 to 71.6	Frozen-weight guidance has a direct execution effect
Reminder on RoboCasa365	39.6 to 46.6	Long dependent tasks benefit most
No recovery trajectories	About 9-point drop	Recovery supervision is necessary
Unstructured teacher reasoning	Substantial LIBERO-Pro drop	Structured state and evidence matter
Direct RefSpatial localization	65.0 to 52.0	Capability transfer is uneven

Table 2: Ablation and diagnostic results assembled from Section 6.3 and the spatial referring table.

Training stage	Settings	Reported values
Embodied pretraining	Epochs, batch, learning rate, accumulation	2, 1, 5×10⁻⁶, 4
Specialist teachers	Epochs, batch, learning rate, accumulation	3, 2, 1×10⁻⁵, 2
MOPD	Epochs, batch, learning rate, top-k	2, 2, 2×10⁻⁶, 64
MOPD sampling	Temperature, teacher weights	1.0, 0.5 manipulation and 0.5 navigation
Hardware	GPUs and precision	16 NVIDIA H20, bfloat16

Table 3: Reproducible training settings listed in Appendix C.

What the Method Contributes

The first contribution is a stronger interpretation of unification. A tool-using system can connect many skills while still depending on a large external controller. OmniBrain-0 instead trains one compact student to absorb domain teachers' decision behavior. The unified policy is learned rather than inferred at runtime from an ensemble or prompt.

The second contribution is the separation of capability acquisition from consolidation. Pretraining creates shared embodied priors, specialist training creates domain depth, and MOPD integrates those depths. This decomposition gives each stage a clear target and avoids asking one training objective to satisfy scene understanding, manipulation ordering, navigation, and cross-domain balance at once.

The third contribution is the feedback structure around self-improvement. Verification is not optional. A task outcome is checked, a rule is proposed from evidence, reminders are evaluated with fixed weights, and only verified trajectories return to training. This design is more defensible than unconstrained self-training because every generated sample has a path back to an execution signal.

The fourth contribution is cost measurement on both success and execution. Reporting tokens and wall-clock time matters for robotics. A model that matches success but takes several times longer may not fit a deployed control loop. The paper does not claim a universal win over the teacher, but it demonstrates a different point on the capability-cost frontier.

Limitations and Open Risks

Author-stated limitations: The system depends on the coverage and reliability of its low-level skills. It still struggles with unseen task compositions and spatial localization. The real-world evaluation covers only a limited set of tasks. These are supported directly by the RoboCasa unseen split, RefSpatial localization drop, and three-task physical study.

Reproducibility: The public repository has no implementation, checkpoints, or released data. The README's code status is explicitly unavailable, and the current commit does not allow an external team to reproduce the data processing, MOPD, reminder generation, or physical experiments. Claims about training details therefore remain paper-only evidence.

Uneven transfer: The model improves relational placement and some unseen spatial combinations but loses direct localization accuracy. It nearly matches the teacher on RoboTwin while retaining a large gap on RoboCasa unseen compositions. A single aggregate score hides these capability-specific failures and could encourage inappropriate deployment.

Evaluation comparability: Some baselines are trained low-level policies, while others are few-shot agents with different tool stacks. The paper reports matched task settings, tools, observation interfaces, and budgets, but the missing code prevents independent verification. Wall-clock results also include simulator and low-level execution, so they are not pure planner latency.

Deployment safety: The Meta-Agent can revise reminders, but reminders are not a substitute for low-level safety constraints. Collision checks, force limits, workspace monitoring, and human oversight remain necessary. The reported task success does not establish safe behavior under distribution shift.

Conclusion

OmniBrain-0 demonstrates a credible route from large, expensive embodied teachers to compact high-level robot agents. Its key mechanism is not a new low-level controller. It is a training pipeline that converts expert interactions into grounded supervision, learns manipulation and navigation specialists, and consolidates their decision distributions through on-policy distillation.

The recursive component adds a practical way to turn failures into rules and verified recoveries, while the reminder ablation shows that some gains are available without changing model weights. The remaining failures are equally important: novel compositions, direct spatial reference, and physical contact still place clear boundaries around the method. The next step is not only more data, but auditable code, stronger uncertainty-aware state representation, and evaluation that reports which capabilities transfer rather than only an aggregate average.