ABot-AgentOS: A General Robotic Agent OS with Lifelong Multi-modal Memory

Paper: ABot-AgentOS: A General Robotic Agent OS with Lifelong Multi-modal Memory
Affiliation: AMAP CV Lab (Alibaba / AMap) · Published: 2026-07-10 (arXiv v3)
Links: arXiv:2607.10350 · Project page · Code: the paper advertises github.com/amap-cvlab/ABot-AgentOS, but the repository is not public as of this writing (404)

TL;DR

ABot-AgentOS is a general-purpose robotic agent operating system that sits above low-level controllers and couples three ideas: a deliberative agent harness in which a main LLM, an isolated Skill Runner, and a multi-stage Verifier form a closed reasoning-execution-verification loop; a universal multi-modal graph memory that stores embodied experience as typed, source-grounded nodes and edges; and a failure-driven lifelong self-evolution protocol that converts diagnosed memory failures into gated JSON-DSL assets that may only be promoted to later evaluation splits. The paper pairs the system with EmbodiedWorldBench, an executable benchmark of 16 indoor/outdoor/hybrid scenes, four difficulty levels, and 200+ tasks scored from execution traces.

Background and Motivation

The paper frames its contribution around a gap that VLM and VLA progress has not closed: translating high-level semantic reasoning into reliable multi-step physical execution. It poses three foundational questions. How does semantic reasoning become dependable multi-step physical behavior? How do capabilities transfer across heterogeneous robot morphologies without extensive retraining? And how does a robot build persistent, long-term memory for continual interaction? These are the gates between lab prototypes and deployable systems.

Three research lines point toward answers but remain only loosely connected. First, dual-system-inspired robotic foundation-model stacks such as Galaxea G0, Hi Robot, and the RoboBrain lineage pair fast perception-action policies with slower deliberative reasoning, while RoboMemory and Agentic Robot explore memory architectures and cognitive decomposition; yet these systems tend to be bound to a particular model stack, embodiment, or control interface. Second, general agent research (ReAct, Toolformer, SWE-agent, and interactive systems like Operator and Claude computer use) shows that language models can interleave reasoning with tool use, and benchmarks such as OSWorld expose how badly even strong multimodal agents struggle with open-ended long-horizon execution. Physical execution adds constraints absent from software agents: partial observability, actuation uncertainty, ambiguous completion signals, and the need to verify whether a planned action actually changed the world. Third, long-term memory has evolved from memory streams and reflection in generative agents to persistent user memory in MemoryBank, MemGPT, and Mem0. Those systems establish that agents should not live off the current prompt and parametric knowledge alone, but embodied agents need a stronger substrate: memory must bind dialogue, egocentric visual evidence, object states, identities, places, temporal and spatial relations, provenance, and robot task traces in a form that can be retrieved and audited during future physical interaction.

From this survey the paper distills three gaps. The reasoning-execution gap: many foundation-model robotic systems map model outputs directly to actions or run monolithic pipelines, with no intermediate agent layer for task decomposition, tool invocation, skill delegation, verification, and recovery. The embodiment-generalization gap: agent systems are tightly coupled to specific hardware, control APIs, and environment assumptions, making cross-body transfer expensive. The persistent embodied-memory gap: short-term buffers, text-only caches, and task-specific memory modules cannot reliably preserve multi-modal, relational, source-grounded experience over long horizons. An evaluation gap follows directly: long-horizon embodied agents need benchmarks that test executable multi-scene scenarios with dynamic events, interaction, and trace-grounded scoring, not isolated navigation, manipulation, or VQA tasks.

Figure 1: System architecture. Multi-source inputs (microphone, app, camera) feed a dual-LLM edge-cloud core; the Agent Harness manages a verification-aware ReAct loop, context, and skill evolvement over an extensible skill library; a hierarchical memory system synchronizes private edge memory with shared cloud memory.

System Architecture: An OS Layer, Not Another Controller

ABot-AgentOS is deliberately positioned as middleware. It does not replace existing control stacks; it provides a unified cognitive layer above robot hardware and low-level controllers, connecting perception, memory, reasoning, planning, and skill execution across humanoids, quadruped robot dogs, mobile manipulators, and robot arms. Inputs from microphones, mobile apps, and cameras are mapped into a runtime loop of perception, reasoning, and action.

The runtime adopts an edge-cloud collaborative LLM design. A lightweight Tiny LLM runs on the edge at every interaction turn, handling low-latency perception, instruction understanding, state tracking, and routine decisions; when a task demands complex reasoning, long-horizon planning, or ambiguity resolution, the system escalates on demand to a cloud-based Large LLM. The Agent Harness layer organizes prompts, tools, APIs, and execution protocols into a reliable loop through three components: Verification-aware ReAct (ReAct plus a verifier module), Context Management (selecting, compressing, and retrieving information from observations, dialogue history, robot state, and memory), and Skill Evolvement (refining, reusing, and expanding the skill set through continued physical interaction). Above the runtime, a unified Skills and Tools layer abstracts manipulation, navigation, motion control, and vision into pluggable interfaces; embodiment-specific mobility and manipulation interfaces are plugged in per platform, so the same high-level reasoning and planning logic is reused across bodies rather than baked into one model.

The Agent Harness: Reasoning, Execution, Verification

Figure 2: The Agent Harness. The main LLM performs scene-conditioned planning with memory and context, delegates procedural subtasks to the Skill Runner, and receives corrective feedback from the Verifier.

The framework starts from an observation that separates embodied agents from code agents and ordinary tool-use agents: embodied tasks rarely provide explicit completion signals for intermediate steps. An agent can issue a navigation command without leaving its current location, or rotate and collide repeatedly while believing, at the language level, that it is making progress. So instead of letting the LLM call more tools, ABot-AgentOS organizes execution as a closed loop of reasoning, execution, and verification, and separates three roles that are usually entangled in a single controller.

Main LLM — scene-conditioned planning. The main LLM is the semantic planner. Before acting, it determines whether the task requires navigation, search, human interaction, reporting, manipulation, or additional observation, and forms a revisable high-level plan with explicit completion conditions. Planning is scene-conditioned rather than purely linguistic: the same instruction yields different strategies depending on current location, available visual evidence, known map structure, reachable regions, nearby objects, and interaction history. The plan is updated as observations, tool results, skill summaries, and verifier feedback enter the context, while local execution detail is kept out of the main reasoning thread so that global task state stays coherent.

Skill Runner — procedural execution in an isolated context. A skill is not a one-shot tool call or an action macro. It is executed by a skill-level subagent with an isolated local context containing the subgoal, recent observations, skill state, failed attempts, and recovery strategy. Context isolation matters precisely because long-horizon execution involves repeated movement, relocalization, view adjustment, and recovery; appending every collision and retry to the main LLM context would drown the global objective in procedural noise. The Skill Runner absorbs this local complexity and returns a compact semantic summary: whether the subgoal was achieved, why it failed, what scene information was discovered, what recovery was attempted, and how the main LLM should use the outcome.

Verifier — multi-stage supervision. Unlike digital tasks where tests, API responses, or page states settle success, embodied success requires consistency between the agent's declared progress, the execution trajectory, and the observed scene state. Verification therefore operates at three stages. Runtime verification watches whether recent behavior indicates effective progress or reveals stagnation, repeated collisions, local loops, or plan-inconsistent actions. Skill-level verification checks whether a delegated subtask actually satisfied its semantic objective rather than accepting success because a tool returned normally. Finish-time verification is triggered when the main LLM attempts to terminate: it weighs the original instruction, the current plan, execution history, skill summaries, observations, and any requirements introduced during interaction before allowing termination. The verifier is thus a supervisory signal inside the harness, not merely a final evaluator.

Edge-cloud routing. Routine requests are handled by the on-device model and local tools; long-horizon tasks escalate to the cloud. The routing policy is learned from training samples and execution feedback rather than fixed rules, capturing which requests local tools can reliably solve, which need cloud-level planning, and which need more observation before routing — keeping latency and cost bounded while retaining cloud-scale capability where it matters.

flowchart TB
    U["User Instruction"] --> M["Main LLM
Scene-Conditioned Planning"]
    MEM[("Multi-modal Graph Memory")] --> M
    CTX["Context Management"] --> M
    M -->|direct tool call| TOOL["Skills and Tools Layer"]
    M -->|delegate procedural subtask| SR["Skill Runner
Isolated Local Context"]
    SR -->|compressed semantic summary| M
    TOOL --> ROB["Robot Execution"]
    SR --> ROB
    ROB --> V{"Verifier"}
    V -->|runtime stagnation feedback| M
    V -->|skill objective not met| SR
    V -->|finish-time approval| FIN["Task Terminated"]
    V -->|missing conditions| M

The reasoning-execution-verification loop of the Agent Harness, drawn from Section 2.2 of the paper.

Universal Multi-modal Graph Memory

Figure 3: Memory architecture. Online, observations and interactions are written into a source-grounded memory graph, task-relevant evidence is retrieved, and retrieval/answer traces are recorded. Offline, failure traces are diagnosed and converted into gated runtime evo-assets for later deployments.

The memory system is positioned beneath the online runtime and complements context management: context management decides what goes into the current prompt, while memory decides what persists beyond the episode and how past evidence is retrieved back into context. The paper states three requirements for a practical robotic memory. It must be multi-modal, because useful experience is distributed across language, egocentric vision, object states, spatial layouts, identities, and task traces. It must be relational, because embodied recall depends on who participated in an event, where an object was last seen, what changed over time, and which frame supports a claim. It must be auditable and improvable: every answer exposes its evidence so failures can be attributed to memory writing, evidence selection, temporal grounding, visual matching, or answer composition.

Long-term experience is represented as a typed graph

$$\mathcal{G}=(\mathcal{V},\mathcal{E}), \tag{1}$$

where each node $v\in\mathcal{V}$ denotes an entity or evidence unit and each edge $e\in\mathcal{E}$ denotes a temporal, semantic, spatial, identity, interaction, or provenance relation. Every node stores a compact JSON-style field with schema version, dataset/source reference, time reference (time_ref, created_at, or adapter-specific fields), evidence summary, confidence, adapter-specific fields, and provenance; provenance records the source id, adapter version, extractor model, and extraction metadata, so any memory item can be traced back to the observation, video segment, frame, image, or session that produced it. The schema covers source containers, evidence units, entities, places, sessions, and semantic events as node types, with edges encoding temporal order, containment, observation, participation, location, identity continuity, spatial relations, and interaction. It is intentionally platform-agnostic — semantic experience and evidence rather than hardware-specific control details — so one memory interface serves multiple embodiments.

The contrast with text-chunk RAG is substantive. Text retrieval returns semantically similar snippets; the memory graph first identifies relevant seed nodes and then expands a local evidence subgraph, letting the answerer reason over identity, time, location, participation, provenance, and spatial relations with structured evidence instead of parametric memory or isolated snippets.

Writing. Memory updating happens during interaction and at reflection checkpoints. A memory-writing service, instantiated as graph-construction adapters per data source, normalizes heterogeneous inputs into the same source-grounded schema: video and egocentric streams yield semantic records of visible entities, object states, places, actions, frames, and events; dialogue and multi-modal sessions yield session-level and event-level records with attached images as evidence nodes. The writer favors semantic compression over raw storage — long-term memory should increase information density without sacrificing traceability. The paper's running example is telling: the utterance "I adopted a Maltese dog yesterday" becomes a time-grounded semantic event linking the utterance, resolved temporal context, identity hypothesis, confidence estimate, and source evidence, rather than a raw transcript line, with any attached image connected as supporting evidence. Writing is selective: identities, object locations, state changes, social commitments, user preferences, abnormal events, temporal facts, and evidence needed for later verification are prioritized. Post-insertion maintenance keeps the graph compact: near-duplicate nodes with compatible provenance, temporal context, and identity evidence are merged; frequently observed entities keep compact state summaries with a bounded set of representative evidence nodes; stale state facts are superseded by newer observations through temporal edges rather than deleted, so retrieval can distinguish current state from historical evidence.

Retrieval and grounded answering. Retrieval is invoked when a task needs information beyond immediate observation and short-term context. The query is converted into semantic signals plus lightweight structural cues (entity mentions, temporal expressions, modality, place references, expected evidence type). Seed nodes are selected by hybrid scoring:

$$s(q,v)=\lambda_{\mathrm{sem}}s_{\mathrm{sem}}(q,v)+\lambda_{\mathrm{lex}}s_{\mathrm{lex}}(q,v)+\lambda_{\mathrm{meta}}s_{\mathrm{meta}}(q,v)+\lambda_{\mathrm{type}}s_{\mathrm{type}}(q,v), \tag{2}$$

where $s_{\mathrm{sem}}$ is embedding similarity, $s_{\mathrm{lex}}$ lexical overlap, $s_{\mathrm{meta}}$ metadata compatibility (time, source, modality, place), and $s_{\mathrm{type}}$ favors node types expected by the query. Top-ranked seeds are expanded along task-relevant typed edges under a fixed depth and evidence-token budget, producing a local evidence subgraph serialized into a compact evidence context paired with a retrieval trace. The answerer is instructed to ground responses in retrieved evidence; when evidence is insufficient or contradictory it reports uncertainty instead of hallucinating, and in forced-choice settings it still records the evidence limitation in the trace. Every memory-augmented QA run records a retrieval trace (retrieved nodes and edges, source references, evidence summaries, retrieval stages, ranking signals). Online, the trace makes answers inspectable; offline, it is the diagnostic signal for failure analysis — a wrong answer can be attributed to missing memory writing, failed retrieval, evidence misuse, or unresolved temporal, spatial, relation-direction, or entity-grounding operations.

Failure-Driven Lifelong Self-Evolution

Figure 4: Failure-to-evolution examples. Left: visual memory QA retrieves image-grounded identity evidence but exposes missing breed-specific cues. Right: temporal text memory QA resolves relative dates via session metadata but reveals temporal-normalization errors. Both failure traces become targeted improvements.

Long-term memory should evolve not only by accumulating content but by improving the pipeline that writes, retrieves, selects, and uses evidence — otherwise the system repeats the same extraction, retrieval, temporal-grounding, visual-selection, and answer-composition errors even as the graph grows. The paper formalizes this as a split-wise protocol over an ordered sequence of disjoint splits $\mathcal{D}_{1},\ldots,\mathcal{D}_{T}$. At split $t$ the system maintains two persistent states: the accumulated memory graph $G_{t-1}$ and the promoted evo-asset set $A_{<t}$. The online run, asset proposal, and promotion steps are:

$$\mathcal{T}_{t},G_{t}=\operatorname{Run}(\mathcal{D}_{t};G_{t-1},A_{<t}), \tag{3}$$

$$\Delta A_{t}=\operatorname{Gate}\!\left(\operatorname{Compile}\!\left(\operatorname{Propose}\!\left(\operatorname{Diagnose}(\mathcal{T}_{t})\right)\right)\right), \tag{4}$$

$$A_{\leq t}=A_{<t}\cup\Delta A_{t}. \tag{5}$$

Here $\mathcal{T}_{t}$ denotes the answer and failure traces collected on split $t$, and only $A_{\leq t}$ is available for later splits. The protocol's core invariant is that no evo-asset generated from split $t$ is used during inference on split $t$: in benchmarks, failures are identified with ground-truth answers only after the split is complete; in deployment, analogous feedback comes from environmental signals, task success or failure, human correction, or low-confidence traces. This turns self-evolution into a cumulative lifelong process rather than one-shot post-hoc repair and blocks current-split ground-truth leakage by construction.

The failure-to-asset loop is constrained end to end. A failed QA attempt is represented by the question, retrieved evidence, prediction, expected answer, retrieval trace, and dataset context. The loop diagnoses the failure source, proposes a candidate repair, compiles it into a constrained JSON DSL evo-asset, and evaluates it with both target and regression checks. Accepted assets are lifecycle-managed JSON DSL records that never execute generated Python code; each declares its target layer, triggering condition, permitted action, safety constraints, provenance, validation result, and version id. Assets may target memory writing, evidence selection, answer composition, visual frame selection, temporal normalization, or adapter-level normalization. Loading is tiered: writer-side and frame-policy assets require fresh graph construction, while evidence-selection and answerer assets can be loaded at runtime. The gate accepts an asset $a$ only if

$$\operatorname{Accept}(a)=\mathbb{I}\left[\Delta S_{\mathrm{target}}(a)\geq\tau_{\mathrm{gain}}\land\Delta S_{\mathrm{reg}}(a)\geq-\tau_{\mathrm{reg}}\right]. \tag{6}$$

Promotion is therefore conservative: an asset must improve the failure pattern it targets while preserving behavior on previously reliable cases. The system accumulates two forms of lifelong knowledge — content-level experience in the memory graph and pipeline-level improvements in the evo-asset set.

The appendix documents a real gate record from an OpenEQA self-evolution split (split_00). Three candidates were proposed: a retriever room-anchor/last-seen focus asset was accepted (target delta +0.800, global delta +0.044, low-score count reduced by 10); a writer materialization asset was rejected because protected object-state recognition regressed beyond tolerance despite target and global improvements; and a writer directed-support-relations asset was rejected with a negative target delta and the same protected regression. The stack containing the accepted asset passed confirmation: normalized mean rose from 0.6053 to 0.6545 (+0.0492), low-score count fell from 91 to 77, protected object-state recognition held, and protected object localization improved by roughly +0.073. The resulting runtime policy is generic — "when a query contains an object anchor, room/place cue, or spatial phrase, rank observation-grounded object memories ahead of broad scene summaries" — and contains no gold answers or question-specific shortcuts.

flowchart LR
    subgraph SPLIT["Split t Evaluation"]
        RUN["Run online with
G(t-1) and prior assets"] --> TRACE["Answer and Failure Traces"]
    end
    TRACE --> DIAG["Diagnose
writing / retrieval / temporal / frame / answering"]
    DIAG --> PROP["Propose Candidate Repair"]
    PROP --> COMP["Compile
JSON DSL Evo-Asset"]
    COMP --> GATE{"Gate
target gain AND regression bound"}
    GATE -->|accept| PROM["Promote to Asset Set"]
    GATE -->|reject| DISC["Discard with Recorded Reason"]
    PROM --> LATER["Available only to later splits t+1 ..."]

The split-wise failure-driven self-evolution protocol, drawn from Equations 3-6 in Section 2.3.4.

Edge-Cloud Memory Management and Privacy Gating

Robots continuously acquire heterogeneous memories — maps, semantic landmarks, obstacles, objects, and person-related observations. Public environmental memories (roadblocks, construction zones, navigational landmarks) help multi-agent collaboration when shared, while privacy-sensitive memories (faces, personal belongings, person-associated objects) must stay local. The design partitions memory accordingly: each robot keeps private edge memory (maps, semantic knowledge, multi-modal interaction history, user preferences, local environmental experience), while the cloud maintains common memory restricted to public, low-sensitivity knowledge. The governing rule is private-by-default: a memory item is never uploaded unless explicitly classified as shareable. Before synchronization, each new item passes a privacy judgment: anything containing or associated with personally identifiable information (persons, faces, names, personal objects, ownership relations to a specific individual) is non-shareable, while public environmental information (maps, traffic cones, barriers, road damage, static landmarks) is shareable. On a dedicated privacy-classification and upload-decision dataset, the mechanism identifies whether a memory item is suitable for cloud synchronization with over 99% accuracy.

EmbodiedWorldBench: Executable Scenarios, Not Prompt Collections

Figure 5: EmbodiedWorldBench. Compound tasks span indoor and outdoor spaces with tightly coupled navigation, NPC interaction, and environment perception: 16 scenes, four difficulty levels, 200+ tasks.

Existing embodied benchmarks suffer three shared limitations: environmental fragmentation (indoor and outdoor scenes evaluated in isolation), task homogeneity (a single capability dimension per benchmark), and static evaluation (no dynamic events, hence no test of adaptive replanning). EmbodiedWorldBench answers with 16 executable scenes across indoor, outdoor, and hybrid settings (hospitals, museums, supermarkets, suburban neighborhoods), four difficulty levels, and over 200 tasks. Every case is a complete runnable world, formalized as:

$$\text{Scenario}=\langle\mathcal{M},\mathcal{S}_{0},\mathcal{O},\mathcal{N},\mathcal{C}\rangle, \tag{7}$$

where $\mathcal{M}$ is the spatial map, $\mathcal{S}_{0}$ the initial state, $\mathcal{O}$ the observation rules, $\mathcal{N}$ the NPC behaviors, and $\mathcal{C}$ the success criteria. The paper's suburban example makes the point: a single compound instruction sends the agent to check a street, inspect a backyard pool, verify an indoor appliance state, and return to report, with the task path traversing outdoor streets, a residential backyard, and an indoor living room in one episode — cross-scene navigation, NPC interaction, perception, and reporting exercised jointly rather than in isolation.

Construction runs on UnrealZoo, a collection of photo-realistic UE5 environments with NavMesh-based navigation and programmable object spawning. Annotators navigate scenes in first person, marking and naming key objects, points of interest, interaction targets, and typical paths to build structured semantic maps; a normalization pipeline converts raw waypoints into three layers — point (coordinates, semantic type, floor/room association, visual attributes), room (semantic topology aggregating points), and polygon (enclosed regions for membership tests) — covering 300+ annotated waypoints. Task generation extracts a scene capability summary (available regions, NPC spawn points, key object inventories), takes annotator-specified feasible task types per scene, and has an LLM generate simple and compound multi-stage tasks with natural-language descriptions and formal success criteria, followed by human review for executability, semantic clarity, evaluator consistency, and information isolation. Difficulty is calibrated human-in-the-loop as a property of the required execution process — spatial exploration scope, procedural length, interaction complexity, evidence requirements, dynamic adaptation — rather than a count of scenes, NPCs, or objects. Every task enforces a visibility boundary: the agent receives only a filtered semantic map and instructions, while NPC positions, evaluation signals, and expected trajectories stay isolated; automated validation checks waypoint reference validity, isolation integrity, and evaluation coverage.

At evaluation time, validated queries run as closed-loop episodes. NavMesh navigation abstracts low-level path planning so the LLM agent focuses on high-level decisions (where to navigate, when to observe, what evidence to collect, when to terminate); visual information flows through a VLM-based observation tool that converts first-person views into textual evidence. Episodes produce structured traces (navigation commands, visited regions, observation records, intermediate outputs, final responses, optional video). Metrics are Task Success Rate (TSR), requiring all subtask objectives and terminal conditions to be satisfied, and Goal Completion Rate (GCR), the proportion of satisfied subtask objectives. Scoring is hybrid: geometric conditions (location arrival, waypoint visitation) are checked programmatically by comparing recorded poses to reference points within a success radius; semantic conditions are judged from run-derived evidence by an LLM judge that never observes the environment directly or touches hidden evaluator state, emitting structured verdicts to keep semantic scoring auditable.

Training Pipeline for a Deployable Student Policy

Figure 6: Training pipeline. Controllable text-based environments are constructed, teacher trajectories are distilled for SFT initialization, and the policy improves through online RL with LLM-as-a-Judge rewards and GiGPO advantages.

The paper also describes a deployment-oriented pipeline transferring ABot-AgentOS-style long-horizon planning and tool use from large teacher models to a smaller deployable agent model. Because this targets business deployment, private data and production results are not released; the contribution is the method. Four stages form a closed loop.

Text-based semantic sandbox. Each natural-language task instruction becomes a stateful sandbox instance. An LLM generates Easy, Medium, and Hard variants, each with three structured components: env_state (locations, reachability, object locations and states), failure_triggers (physical or semantic obstacles), and human_persona (the NPC's role, capabilities, and possible assistance). During execution the agent receives feedback only through tool calls: an EnvController validates skill calls against current state and failure triggers, returning observations or failures, while a HumanAgent handles valid help requests and may act to update env_state. Obstacle handling, help seeking, replanning, and state updates thus all become training signal. The difficulty expansion doubles as a curriculum: Easy scenarios emphasize decomposition and basic skill selection, Medium introduces recoverable obstacles, Hard requires multi-turn interaction, persistent goal tracking, and recovery from multiple or nested failures.

Teacher distillation and SFT. A strong teacher controls an ABot-AgentOS-style agent in the sandbox, reusing the system prompt and tool definitions, producing ReAct-style traces interleaving reasoning, tool calls, observations, human responses, and final answers. Crucially, the collected trajectories include failed attempts, help-seeking, retries, and recovery steps. An LLM-as-a-Judge filter scores task completion, obstacle handling, loop avoidance, tool feasibility, and consistency with embodiment-specific capability constraints, retaining only logically consistent trajectories. Retained traces become tool-call SFT examples that preserve interaction structure rather than flattening episodes into single responses, training the student as an executable policy over ABot-AgentOS skills.

Online RL with GiGPO. The SFT-initialized student samples actions online in the same sandbox and updates from reward-guided policy optimization. The loop separates four roles: the policy is the only trainable component; the EnvController executes tool calls against sandbox state; the NPC module provides human-assistance responses; and a frozen LLM-as-a-Judge reward provider reads rollout evidence and assigns rewards. Advantages come from GiGPO: multiple rollouts sampled from the same initial state are compared on merged returns to form an episode relative advantage (which complete rollout solves the task better), while turn-level rewards at aligned or comparable decision steps form a step relative advantage (reinforcing better local recovery choices in similar situations, such as the same blocked passage or nearby NPC). The final advantage combines both levels of relative comparison.

Self-evolving reward engine. The engine converts each trajectory into turn-level and episode-level signals. Turn-level rewards are skill-aware — navigation, manipulation, visual QA, and tool-use actions route to different rubrics — combining LLM semantic judgment with verifiable rule checks (invalid tools, illegal commands, malformed arguments, missing temporal preconditions) plus action-omission penalties for locally plausible actions that skip necessary causal steps (reporting a destination before verifying the target; asking for help without checking the relevant environmental constraint). Episode-level rewards decompose along efficiency, consistency, and completeness:

$$r^{\mathrm{episode}}(\tau)=\lambda_{\mathrm{eff}}R_{\mathrm{eff}}(\tau)+\lambda_{\mathrm{cons}}R_{\mathrm{cons}}(\tau)+\lambda_{\mathrm{comp}}R_{\mathrm{comp}}(\tau), \tag{8}$$

and merge with effective turn-level rewards into

$$R(\tau)=\sum_{t=1}^{T}\hat{r}_{t}+r^{\mathrm{episode}}(\tau), \tag{9}$$

where $\hat{r}_{t}$ is produced by skill-specific rubric routing, semantic judgment, rule checks, and omission penalties. A Meta-Judge validates reward reliability: rather than judging the agent directly, it receives the first-order judge output, the highlighted turn, and full trajectory context, and checks the judge's score and rationale along five dimensions — accuracy, logical soundness, completeness, clarity, and feedback value — aggregated into a quality score

$$Q(x_{t})=\sum_{k=1}^{5}w_{k}q_{k}. \tag{10}$$

Samples below threshold become low-quality judge cases:

$$\mathrm{LowQualityJudgeCase}(x_{t})=\mathbb{I}[Q(x_{t})\<\tault;\tau], \tag{11}$$

which are more useful than binary error labels because they indicate whether the reward prompt is underspecified, ambiguous, or misaligned with embodied constraints. A multi-agent workflow converts these cases into localized prompt updates: Cluster groups cases by skill and failure type; Analyzer locates the defective rubric component and produces a revision plan; Refiner applies localized edits (the prompt is treated as editable structured components, since a global rewrite may fix one failure while damaging a stable skill); Validator evaluates the revised prompt on the full validation set and accepts only if overall quality improves, otherwise the system rolls back to the previous snapshot. Under the paper's validation setting, the initial judge achieves roughly 60% human alignment, and Meta-Judge-driven self-evolution lifts alignment above 90%.

Experiments

Agent Evaluation on an EmbodiedWorldBench Subset

The agent evaluation uses a benchmark subset covering the main scene types and task forms (semantic navigation, regional person/object search, object inspection, NPC information query, status reporting, multi-stage instruction following), and is framed as initial system validation rather than a leaderboard. Three settings are compared: a single-LLM ReAct controller as baseline, ABot-AgentOS with the same base model, and ABot-AgentOS with a stronger main LLM. Across all settings, visual observation is handled by the same Qwen3-VL-Plus observation tool; only the main LLM controller differs.

Agent	Model	TSR	GCR
ReAct	Qwen3.6-Plus	49.97%	57.95%
ABot-AgentOS	Qwen3.6-Plus	61.96%	68.79%
ABot-AgentOS	DeepSeek-V4-Pro	68.18%	74.62%

Table 1 from the paper: agent evaluation on the EmbodiedWorldBench subset.

Under the same base model, the hierarchical architecture improves TSR by 11.99 points and GCR by 10.84 points over the single-controller baseline, suggesting that hierarchical execution, task memory, skill-level feedback, and finish-time verification help long-horizon embodied execution under an identical base model and tool interface. Because GCR and TSR rise together, the gain is not limited to partial progress — more trajectories convert into complete task success. Swapping the main LLM to DeepSeek-V4-Pro adds another 6.22 points of TSR, reflecting the backbone's impact on instruction understanding, stage planning, and failure recovery. The authors are candid about remaining failures: the agent lacks fine-grained active observation mechanisms to confirm target states; the VLM observation tool confuses people and objects, hurting NPC localization and interaction decisions; and in scenes with indoor-outdoor connections the agent can confuse visible regions with its own location, e.g. interpreting an outdoor area seen from indoors as evidence that it has already moved outside.

Memory Evaluation Across Five Benchmarks

The memory module is evaluated independently of agent control on five benchmarks stressing complementary capabilities: LoCoMo (very long-term multi-session conversational recall), OpenEQA EM-EQA (open-vocabulary embodied QA over indoor environments), Mem-Gallery (multi-modal long-term conversational memory with visual-textual dependencies), NExT-QA (temporal, causal, and descriptive video QA), and EgoLife (long-context egocentric daily-life QA). All experiments share the same base hybrid graph retriever, so retrieval is not an experimental variable; writer/answerer models are instantiated per benchmark protocol (GPT-5.4 for OpenEQA as writer, answerer, and LLM-Match judge; Qwen3.6-Plus for LoCoMo, Mem-Gallery, and NExT-QA writing; Qwen3.5-Flash for EgoLife egocentric writing). Information boundaries are strict: non-oracle memory graphs are built only from inference-time information, gold answers/rationales/evidence never enter the graph or the answerer context, and supervision used for self-evolution is applied only after the corresponding split has been evaluated.

Method	Setting	Single-hop	Temporal	Multi-hop	Open-domain	Adversarial	Overall
A-MEM	Memory	35.9	31.8	23.1	23.9	7.4	26.4
MIRIX	Memory	82.6	81.3	82.9	64.6	36.3	71.2
MemGPT	Memory	86.3	85.6	86.8	57.3	65.9	80.3
MemInsight	Memory	82.1	76.9	80.5	64.6	90.3	82.0
Mem0	Memory	94.1	96.6	90.8	68.8	62.3	85.6
ABot-AgentOS Static	Graph memory	92.9	87.5	90.9	70.8	80.9	87.5
Human	Human	95.1	92.6	85.8	75.4	89.4	87.9

Table 2 from the paper: LoCoMo results under the Mem0 judge protocol (full-context GPT-5.4 reference scores 84.4 overall).

On LoCoMo, ABot-AgentOS Static scores 87.5 overall, beating the strongest reproduced memory baseline Mem0 (85.6) by 1.9 points and approaching the human score of 87.9; its 80.9 on adversarial questions stands well above most memory methods. The result suggests graph memory pays off precisely when a benchmark requires combining long-horizon conversational facts with relation- and provenance-aware retrieval. On OpenEQA EM-EQA, the static system reaches 59.2 overall with 8 frames (62.8 on ScanNet) and 59.9 with 24 frames, outperforming every listed memory baseline — scene-graph memory (GPT-4 + ConceptGraphs, 36.5), caption memory (43.6), retrieved captions (R-EQA, 46.0), dynamic scene memory (GraphPad, 55.3), 3D snapshot memory (3D-Mem/SnapMem, 57.2), and 3DGS memory (GaussExplorer, 57.8) — while direct VQA rows such as GPT-5.4 with 24 frames (74.1) serve as non-memory upper-bound references rather than memory systems.

Method	Memory / Input	Frames	ScanNet	HM3D	Overall
GPT-4 + ConceptGraphs	Scene graph	10	37.8	34.0	36.5
R-EQA	Retrieved captions	3	49.1	42.8	46.0
GraphPad	Dynamic scene memory	5-20	-	-	55.3
3D-Mem / SnapMem	3D snapshot memory	3.1	-	-	57.2
GaussExplorer	3DGS memory	n/a	-	-	57.8
ABot-AgentOS Static	Graph memory	8	62.8	52.3	59.2
ABot-AgentOS Static	Graph memory	24	61.9	55.7	59.9
Human	Human	-	87.7	85.1	86.8

Table 3 from the paper: OpenEQA EM-EQA by data source (LLM-Match).

On Mem-Gallery, ABot-AgentOS Static scores 88.6 overall, above the listed textual and multi-modal memory baselines (MemGPT 87.6, MuRAG 84.4, UniversalRAG 84.7, MemoryOS 81.7, AUGUSTUS 80.6) and below the full-context upper bound (92.6). Its largest advantages sit exactly where source-grounded records should help: visual-centric reasoning (81.0), conflict detection (97.5), and answer refusal (100.0), where provenance tracking lets the answerer preserve consistency and refuse unsupported recall. On NExT-QA validation, the system reaches 76.5 Acc@All (causal 77.9, temporal 73.4, descriptive 83.4), improving over the best memory-based baseline GraphVideoAgent (73.3) by 3.2 points, with gains concentrated on causal and temporal questions where the graph preserves event order, entity relations, and cross-segment interactions; the Qwen3.6-Plus direct-QA row (81.9) is reported as a strong non-memory model reference, not a memory baseline. On EgoLifeQA, ABot-AgentOS with Qwen3.5-Flash reaches 65.4% average accuracy while retrieving only a single frame from 1FPS-sampled video, beating EGAgent-Gemini2.5 Pro (57.5%), WorldMM (56.0%), and conventional VQA; the only category where another method leads is TaskMaster, where EGAgent scores 74.6% against 66.7%.

Lifelong Self-Evolution Gains

The self-evolving variant shares the static run's graph schema, base retriever, and writer-answerer configuration; the only difference is that later splits may load evo-assets promoted from earlier splits (writer rules, evidence-selection preferences, answerer calibration rules, frame-selection policies, temporal-normalization policies). Across all five benchmarks the primary score improves over the static system: the largest absolute gain is on NExT-QA (+4.1 Acc@All); LoCoMo rises from 87.5 to 88.7; OpenEQA from 59.9 to 60.4; Mem-Gallery gains 0.4 overall (88.6 to 89.0) with category-level gains concentrated in knowledge resolution, conflict detection, and multi-entity reasoning; EgoLife improves from 65.4 to 66.2, with the largest category gain on TaskMaster. Because assets are promoted only after gated validation and used only on later splits, these gains correspond to reusable memory-pipeline improvements rather than post-hoc repair of the current split. The same mechanism is directly compatible with deployment: instead of benchmark ground truth, environmental outcomes and human interaction feedback can serve as the correctness signal for future evolution.

Limitations

The authors themselves list four limitations. Large-scale real-world validation remains to be done under noisy perception, imperfect actuation, network latency, safety constraints, and heterogeneous embodiments. EmbodiedWorldBench's scene diversity, social-interaction depth, and current agent-evaluation coverage are still limited; the complete benchmark evaluation and public release are deferred to future work. The memory and self-evolution pipeline depends on structured traces and post-hoc feedback, and privacy-aware edge-cloud sharing needs stronger auditing, user control, access management, and failure analysis. Finally, the small-model training pipeline should move beyond text-state semantic sandboxes by incorporating visual observations, multi-modal feedback, and additional simulation platforms.

A few additional points deserve scrutiny. The agent evaluation covers only a benchmark subset and compares against a single ReAct baseline, with no other hierarchical agent frameworks or memory-augmented agent baselines in the table, so the 11.99-point TSR gain cannot be fully attributed to individual components. In benchmark settings the self-evolution loop uses post-hoc ground truth as its failure signal; the split protocol prevents current-split leakage, but the equivalence between this controlled substitute and the sparse, noisy feedback of real deployment is still an assumption. The training pipeline releases neither data nor production results for commercial reasons, limiting reproducibility of that part. And several external baseline numbers come from different protocols in source papers — the authors acknowledge that the most controlled comparison is between their own Static and Self-evo runs.

Summary and Outlook

ABot-AgentOS makes the case that what sits between foundation models and robot controllers matters as much as the models themselves. The edge-cloud dual-LLM runtime addresses real latency and cost constraints; the main-LLM / Skill-Runner / Verifier triad attacks the missing-completion-signal problem of embodied execution head on; the typed, source-grounded graph memory offers a relational recall substrate stronger than text-chunk RAG; and the gated, split-wise self-evolution protocol gives the memory pipeline a second kind of lifelong knowledge — improvements to itself. EmbodiedWorldBench fills a real evaluation gap with executable scenarios, visibility isolation, and trace-grounded scoring. On the evidence side, the memory system beats its corresponding memory baselines on all five heterogeneous benchmarks and approaches human level on LoCoMo (87.5 vs 87.9); self-evolution pushes later-split scores higher without touching the retriever; and the agent architecture delivers double-digit TSR gains under a fixed base model. If the promised full benchmark release, real-robot validation, and public code materialize, this combination of reasoning-execution-verification, auditable memory, and constrained self-evolution will be a serious reference point for embodied agent system design.

Quotes

"An embodied agent must not only interpret a natural-language instruction, but also understand the scene in which the instruction is grounded, the current execution stage, and the conditions under which the task can be considered complete."

"Long-term memory should increase information density without sacrificing traceability."

"An asset must improve the failure pattern it targets while preserving behavior on previously reliable cases."