AtlasVLA: Persistent World-Ego State Modeling for Vision-Language-Action Models

AtlasVLA: Persistent World-Ego State Modeling for Vision-Language-Action Models

Authors: Guiyu Zhao, Longteng Guo, Yanghong Mei, Zilin Zhu, Yu Zhang, Bin Cao, Mingming Yu, Xingjian He, Jie Jiang, Jing Liu

Paper: arxiv.org/abs/2608.06729

Code: Not yet released

One-Line Summary

AtlasVLA introduces a dual-memory architecture—4D Persistent World State Memory and Ego-Working State Memory—that shifts VLA from reactive observation to proactive reasoning, achieving 97.6% on LIBERO and 70.8% on RLBench with only a wrist camera, outperforming π₀ by 17.5% on long-horizon real-world tasks.

Background and Motivation

Vision-Language-Action (VLA) models map multimodal sensory inputs to low-level robotic control commands, achieving significant progress in embodied AI. However, current VLAs operate under a fundamentally reactive paradigm: they directly map immediate observations to actions, heavily relying on instantaneous visual inputs. This exposes two critical bottlenecks in unstructured, partially observable environments.

The first bottleneck is perception forgetting. The wrist camera's field-of-view moves dynamically with the end-effector, and task-relevant objects are immediately forgotten once out of view. Without persistent internal state, the agent loses track of its spatial surroundings—"where is the box?" becomes unanswerable, causing catastrophic execution failures outside multi-view setups.

The second bottleneck is task-progress forgetting. Complex tasks like sequential object organization require the agent to execute multiple sub-steps while maintaining clear cognitive awareness of progress. Standard VLAs lack the historical context to remember completed steps, easily losing their place in the overarching execution sequence, causing compounding errors—repeating or skipping critical sub-steps.

Figure 1: Dual bottlenecks of current reactive VLAs versus AtlasVLA's advantage. (A) Partial observation: wrist-only VLAs suffer limited FoV. (B) Task-progress forgetting: reactive baselines lack temporal context. (C) AtlasVLA integrates 4D Persistent World State and Ego-Working State for robust execution.

Human cognition does not rely on an always-visible, omniscient camera. The human brain navigates partial observability by maintaining an internal world model—a continuous cognitive map tracking objects, structures, and dynamics even outside the line of sight. To achieve true autonomy, embodied agents must transcend reactive behaviors and develop the capacity to construct and maintain a persistent world-ego state.

Preliminaries

The Diffusion Transformer (DiT) is the action generator. Unlike standard diffusion models with global conditioning, AtlasVLA's DiT adopts a decoupled step-wise conditioning mechanism: at each diffusion step, it first injects ego-working context then world state context. A 7B LLM serves as the semantic backbone, with a ~300M parameter DiT as the action expert.

Depth Anything v3 provides streaming depth estimation, leveraging historical frames for temporal depth consistency. The TSDF (Truncated Signed Distance Function) map integration strategy inspires the voxel-hashing world state update mechanism—aggregating latent features of tokens corresponding to the same physical region for efficient persistent spatial memory.

Method

Figure 2: Overall architecture of AtlasVLA. Wrist-camera images extract 2D tokens lifted to 3D via depth and extrinsics to form instantaneous world state, updated into persistent world memory via voxel-hashing fusion; intent queries condense ego tokens to update working memory; dual-path retrieval conditions DiT for robust action generation.

1. Problem Setting

At time step $t$, the agent receives solely the wrist-mounted observation $O_t^w$, proprioceptive state $S_t$, and language instruction $L$. The objective is to learn a mapping $\pi_\theta(\cdot)$ generating future actions:

$$A_t = [a_t, a_{t+1}, \dots, a_{t+k-1}] \sim \pi_\theta(\cdot \mid O_t^w, S_t, L)$$

where $A_t$ is an action chunk of size $k$, and each action $a_t \in \mathbb{R}^7$ is a 6-DoF end-effector pose plus binary gripper state. The key constraint is strict wrist-only observation—no third-person views.

2. Persistent World State Memory

Given wrist observation $I_t^w$, a frozen vision encoder extracts 2D tokens. Depth is estimated via Depth Anything v3, and extrinsics derived from robot state: $\mathbf{T}^{ex} = \psi(S_t) \cdot \mathbf{T}^{h2e}$, where $\psi(\cdot)$ transforms end-effector state to pose matrix and $\mathbf{T}^{h2e}$ is the hand-eye calibration matrix.

The spatial back-projection module lifts 2D tokens into 3D latent space:

$$m_t, P_t = \text{Back-Projection}(X_w^t, D_t^w, \mathbf{T}^{in}, \mathbf{T}_t^{ex})$$

where $m_t$ is the instantaneous world state comprising 2D latent tokens with 3D positions, and $P_t$ the corresponding 3D coordinates.

To endow the world state with precise spatio-temporal awareness, a dual positional embedding mechanism injects 3D coordinates and timestamps:

$$\widehat{m}_t = m_t + \mathcal{E}_{spatial}(P_t) + \mathcal{E}_{temporal}(t)$$

where $\mathcal{E}_{spatial}(\cdot)$ encodes 3D position to preserve geometry and $\mathcal{E}_{temporal}(\cdot)$ injects temporal encoding. Both are MLP-parameterized. By entangling spatial and temporal cues, the instantaneous world state transitions from an isolated observation pool to a unified 4D representation.

The world state update adopts a voxel-hashing strategy. 3D space is partitioned into uniform voxels; for incoming tokens corresponding to identical physical regions, their latent features are aggregated with weighted averaging within each voxel. When a new world state $m_t(v)$ with confidence weight $w_t$ is projected globally, the memory $\mathcal{M}_t(v)$ updates:

$$\mathcal{M}_t(v) = \frac{\mathcal{W}_{t-1}(v)\mathcal{M}_{t-1}(v) + w_t m_t(v)}{\mathcal{W}_{t-1}(v) + w_t}$$

where the confidence weight $w_t(v) = c_t(v)$ derives from depth estimation confidence, and cumulative weight $\mathcal{W}_t(v) = \lambda \mathcal{W}_{t-1}(v) + w_t(v)$. Higher-confidence depth observations contribute more to global memory while uncertain measurements are suppressed.

Temporally, the global memory maintains a sliding window of maximum size $W$. Critically, the first frame's spatio-temporal memory is permanently anchored—since the first frame typically provides the optimal field-of-view and accurately reflects the initial state, providing persistent global context.

3. Ego-Working State Memory

The intent-aware query mechanism defines learnable queries $Q^{ego} \in \mathbb{R}^{N \times d}$ that are fed into the VLM at each decision step to aggregate goal-oriented information:

$$Z^{ego} = \text{Softmax}\left(\frac{Q^{ego} K^T}{\sqrt{d}}\right) V$$

where $K$ and $V$ are key and value projections of all tokens. By condensing extensive spatio-temporal context into focused intent tokens, the model maintains a coherent global receptive field, explicitly preventing intention forgetting.

The ego-working memory bank updates through redundancy-aware latent consolidation:

$$\mathcal{M}_t^{ego} = \text{Cons}\left(\mathcal{M}_{t-1}^{ego} \cup \{Z_t^{ego} + \mathcal{E}_{temporal}(t)\}\right)$$

where $\text{Cons}(\cdot)$ merges temporally adjacent and semantically similar intent tokens, avoiding unbounded memory growth and semantic redundancy.

4. World-Ego-Guided Action Generation

The current ego token $Z_t^{ego}$ retrieves historical context from the consolidated memory:

$$C_t^{ego} = \text{CrossAttn}(Z_t^{ego}, \mathcal{M}_t^{ego}, \mathcal{M}_t^{ego})$$

Then the ego context $C_t^{ego}$ serves as query to retrieve task-relevant world states:

$$C_t^{world} = \text{AddNorm}\left(\text{FFN}\left(\text{IntentAttn}(C_t^{ego}, \mathcal{M}_t, \mathcal{M}_t)\right)\right)$$

This ego-guided retrieval selectively extracts relevant world states while suppressing spatial redundancy. Finally, the DiT at each diffusion step $k$ first applies ego-working attention then world state attention, followed by Add & Norm layers, with the action decoder outputting final actions $A_t$.

graph TD
    A["Wrist Camera Observation"] --> B["2D Visual Tokens + Depth Estimation"]
    B --> C["Spatial Back-Projection to 3D"]
    C --> D["Spatio-Temporal Positional Embedding to 4D"]
    D --> E["Voxel-Hashing Fusion + Sliding Window"]
    E --> F["Persistent World State Memory M"]
    A --> G["Intent-Aware Queries Q_ego"]
    G --> H["Ego-Working Tokens Z_ego"]
    H --> I["Redundancy-Aware Consolidation"]
    I --> J["Ego-Working Memory Bank"]
    J --> K["Ego Retrieval C_ego"]
    K --> L["World Retrieval C_world"]
    F --> L
    L --> M["DiT Step-Wise Conditioned Action Generation"]
    K --> M
    M --> N["7-DoF Actions A_t"]

Experimental Results

LIBERO Benchmark

AtlasVLA with only a wrist camera achieves 97.6% average success rate, outperforming baselines equipped with additional third-person views—3.4% higher than π₀. On the challenging LIBERO-Long, it reaches 94.6%, outperforming MemoryVLA by 7.0%. MemoryVLA drops 5.8% when stripped of third-person views, while AtlasVLA's explicit persistent world state modeling eliminates reliance on external spatial priors.

Method	Cameras	Spatial	Object	Goal	Long	90	Avg
π₀	3rd	90.8	91.8	89.6	80.2	—	88.1
MemoryVLA	3rd	98.4	98.4	96.4	93.4	95.6	96.5
π₀	3rd+wrist	96.8	98.8	95.8	85.2	—	94.2
MemoryVLA	wrist	96.2	99.2	96.4	87.6	90.7	94.0
AtlasVLA	wrist	99.4	99.8	98.2	94.6	95.8	97.6

RLBench Benchmark

On RLBench with only a wrist camera, AtlasVLA achieves 70.8% average success rate, surpassing all VLA baselines—7.5% and 15.8% above MemoryVLA under third-person and wrist-only settings respectively. Advantages are particularly pronounced on tasks with high spatial variation like sweep-to-dustpan and umbrella-out-frame-off.

Method	Cameras	Avg
OpenVLA	3rd	23.3
MemoryVLA	3rd	63.3
CogACT	wrist	34.2
MemoryVLA	wrist	55.0
AtlasVLA	wrist	70.8

Figure 3: AtlasVLA qualitative results on real-world long-horizon tasks. Shows complete execution of multi-stage operations, validating persistent world state and ego-working memory effectiveness.

Real-World Evaluation

On a Franka robot with strict wrist-only camera, AtlasVLA achieves 78.7% on general tasks—8.0% above MemoryVLA's third-person setting and 16.4% above its wrist-only setting. On long-horizon tasks, 69.5% average—17.5% above π₀ and 9.0% above MemoryVLA.

Method	General	Long-Horizon
π₀ (3rd+wrist)	66.7	52.0
MemoryVLA (3rd)	70.7	60.5
MemoryVLA (wrist)	62.3	—
AtlasVLA (wrist)	78.7	69.5

Ablation Studies

Variant	LIBERO	Real-World Long
w/o World State Memory	93.5	54.0
w/o Ego-Working Memory	95.0	56.5
w/o World State Update	94.6	58.0
w/o Spatial PE	96.4	67.5
w/o Temporal PE	96.8	65.0
w/o World State Conditioning	95.2	61.5
Full AtlasVLA	97.6	69.5

Removing world state memory causes catastrophic collapse on real-world tasks (69.5%→54.0%, a 15.5% drop), demonstrating that spatial map memory is indispensable for wrist-only manipulation. Removing ego-working memory causes a 13.0% drop, validating latent ego memory for preventing intention drift. Replacing spatio-temporal update with naive accumulation drops 11.5%, confirming TSDF voxel aggregation and sliding window effectiveness. Removing world state attention drops 8.0%, confirming that injecting world state is essential for precise action generation.

Limitations

The paper does not include a dedicated limitations section, but several boundaries can be identified from the method design and experimental setup. First, world state memory depends on depth estimation quality—Depth Anything v3's confidence directly affects voxel fusion weights, and depth estimation on reflective, transparent, or low-texture surfaces may be unreliable, injecting noise into the world state memory. Second, the permanent first-frame anchor assumes the initial frame provides the optimal global FoV; if the initial pose is suboptimal, the global context may be limited. Third, voxel resolution is a fixed parameter—too coarse loses spatial detail, too fine increases computation—the paper does not discuss adaptive voxel resolution. Fourth, real-world evaluation is conducted only on a Franka platform, without validating transfer to humanoid robots or mobile platforms.

Conclusion and Outlook

AtlasVLA demonstrates a profound insight: the key to upgrading VLA from reactive observation to proactive reasoning lies not in more cameras or larger models, but in constructing persistent world-ego state. The 4D voxel-hashed world state memory lets the robot "remember" object positions beyond the field of view, while the ego-working memory lets it "know" which step of the task it has reached—the synergy of these two memories enables a single wrist camera to surpass multi-view systems.

The 97.6% LIBERO score and 17.5% real-world long-horizon improvement show that this paradigm shift is more leveraged than adding observation views. The catastrophic 15.5% drop from removing world state memory further proves that persistent spatial memory is not a luxury but a necessity under partial observability. This suggests future VLA development should move from "more sensors" toward "smarter internal state construction."

Future work may explore adaptive voxel resolution, learning-policy-trajectory-driven world state updates, and extension to humanoid whole-body manipulation. Fusing world state memory with tactile feedback or force sensing could further enhance manipulation robustness in severely occluded scenarios.

When a robot not only sees the current frame but remembers objects that left its view and steps it has completed, one wrist camera suffices to surpass three—because the true perception bottleneck is never the number of viewpoints, but the persistence of internal state.