PAPER DEEP DIVE
Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers
Multi-agent interactive world models should not only generate consistent observations, but also maintain world states that persist across agents and evolve across views. Existing autoregressive video diffusion pipelines carry forward observation history as conditioning context, which makes shared state difficult to maintain in multi-agent and multi-view settings. We present WorldWeaver (W^2), a streaming multi-agent video diffusion model that augments rollout with cross-agent world state registers: learnable tokens that store shared world information, track individual agent status, and are dynamically updated after each generated chunk. We ground these registers with supervision signals spanning individual agent status, global state views including bird's-eye views, and scene text. We further improve the architecture with a Mixture-of-Transformers design that uses separate weights for world state modeling and visual frame modeling. Extensive experiments in two-agent Minecraft video generation show that explicit world-state modeling improves logical consistency and generation quality.
In One Sentence
WorldWeaver ($\mathbf{W}^2$) augments streaming autoregressive video diffusion with world state registers—learnable tokens shared across agents and updated step-by-step—to explicitly hold the global world state and per-agent status, grounded by bird's-eye views, scene text, and agent pose, with a Mixture-of-Transformers separating state and visual weights, improving both logical consistency and generation quality in two-agent Minecraft video.
Figure 1 — Interactive agents act in a shared world; a set of register tokens represents the evolving world state shared across agents, capturing agent status, bird's-eye views, and scene text, dynamically updated at each rollout step.
1. Background and Motivation
An interactive world model should not only generate visually consistent observations but also maintain a world state that persists across agents and stays compatible across views. Mainstream autoregressive video diffusion pipelines carry forward observation history as conditioning context—implicitly stuffing "the world" into frame history.
In multi-agent, multi-view settings, each observer sees only a partial 2D projection of the same 3D world; observations across agents must remain compatible with the shared state, which evolves whether or not a particular observer is looking. Relying on frame history to repeatedly re-infer the world cannot satisfy cross-agent consistency.
The core motivation: model the world state explicitly within the generation process, rather than letting it be a byproduct of per-frame generation.
2. Core Method
Three innovations: world state registers (WSR) + three supervision signals + a Mixture-of-Transformers architecture, wrapped over a Self-Forcing streaming rollout.
2.1 Multi-Agent Setting
Following Solaris, $P$ agents act in a shared world; the model predicts each agent's future video stream from past observations and actions. A player axis is added to every diffusion tensor: per-step observation $\mathbf{x}^t=(\mathbf{x}^t_1,\ldots,\mathbf{x}^t_P)$, action $\mathbf{a}^t=(\mathbf{a}^t_1,\ldots,\mathbf{a}^t_P)$, $N$ latent frames $\mathbf{x}\in\mathbb{R}^{P\times N\times H\times W\times C}$, with conditioning $c$ bundling first-frame visual embedding, masked first-frame latent, and each agent's action sequence.
2.2 World State Registers (WSR)
Registers are learnable tokens living alongside frame tokens in a causal Transformer, capturing shared world information, shared across agents, updated incrementally as new frames are generated. Two key properties:
- Persistence: shared across agents and rollout steps.
- Dynamic updateability: refreshed when new observations arrive.
All agents' observations contribute to state updates, encouraging temporal and cross-agent consistency at the world-state level. Frames and registers interleave as $[\mathbf{x}_0,\mathbf{r}_0,\mathbf{x}_1,\mathbf{r}_1,\ldots,\mathbf{x}_{F-1},\mathbf{r}_{F-1}]$, making rollout causal at the state level: committed register $\mathbf{r}_i$ precedes and conditions the next frame $\mathbf{x}_{i+1}$.
Figure 2 — (A) Standard streaming AR diffusion with local KV cache. (B) Our model with world state registers. (C) Stage-2 causal attention mask. (D) Auxiliary decoders ground registers with agent status, BEV maps, scene text.
2.3 Three-Stage Training Curriculum
- Stage 1 Bidirectional Training: finetune a multi-player bidirectional teacher on synchronized clips with flow matching $\mathcal{L}_{\mathrm{flow}}=\mathbb{E}\!\left[\left\|\mathbf{v}_\theta-\mathbf{v}_{\mathrm{target}}\right\|^2\right]$ to learn cross-agent scene structure.
- Stage 2 Causal Training: convert the teacher into a causal student, continuing flow matching with register supervision via heads $h_m(\mathbf{r}_i)$, targets $\mathbf{y}_i^m$, metrics $d_m$.
- Stage 3 Self-Forcing: the student rolls out on its own predictions; after each frame the register is committed, exposing register drift alongside frame drift; DMD-style distribution matching from the teacher while retaining state supervision.
2.4 Three Supervision Signals
Flow matching alone doesn't specify what WSR should retain, so three auxiliary groundings:
| Supervision | Head output | Target | Metric $d_m$ |
|---|---|---|---|
| Agent status | $\hat{\mathbf{y}}_i^{\mathrm{agent}}$ | position/velocity/orientation | $\|\hat{\mathbf{y}}-\mathbf{y}\|_2^2$ |
| Bird's-eye view | $\hat{\mathbf{y}}_i^{\mathrm{bev}}$ | aligned BEV | $1-\cos(\hat{\mathbf{y}},\phi_{\mathrm{DINOv2}}(\mathbf{y}))$ |
| Scene text | token logits | textual scene description | cross-entropy CE |
Total register loss and Stage-2 objective:
$$\mathcal{L}_{\mathrm{reg}}=\sum_{i,m}\lambda_m\,d_m\!\left(h_m(\mathbf{r}_i),\mathbf{y}_i^m\right),\qquad \mathcal{L}_{\mathrm{S2}}=\mathcal{L}_{\mathrm{flow}}+\lambda_{\mathrm{state}}\mathcal{L}_{\mathrm{reg}}$$
Prediction heads are training-only; discarded at inference, adding no rollout cost.
2.5 Mixture-of-Transformers
When dense weights must both generate pixels and update registers, frame and state objectives compete—especially as the register is pushed toward richer semantic supervision. MoT gives the register branch separate weights with joint attention but decoupled computation. This matters most under scene text supervision: the dense model's world score drops to 91.3 while MoT reaches 103.2—richer semantic states are easier to process with independent parameterization.
flowchart TB
subgraph Inputs
F0["First-frame obs
+ actions per agent"]
end
subgraph WSR["World State Register path"]
R0["r0 init"] --> R1["r1 commit"]
R1 --> R2["r2 commit"]
R2 --> R3["r3 commit"]
end
subgraph FramePath["Frame generation path"]
X1["x1 denoise"] --> X2["x2 denoise"] --> X3["x3 denoise"]
end
F0 --> X1
R1 --> X2
X1 --> R1
R2 --> X3
X2 --> R2
R3 --> X4["x4 denoise"]
subgraph Supervision["Training-only heads"]
R1 -.-> H1["agent status"]
R1 -.-> H2["bird-eye view"]
R1 -.-> H3["scene text"]
end
style WSR fill:#dbeafe,stroke:#2563eb
style Supervision fill:#fef9c3,stroke:#ca8a04
3. Experiments
3.1 Main Results
On the Solaris test split, the model receives both players' first-frame observations and full action sequences, then generates the remaining rollout. Metrics: VLM accuracy (does the generated video preserve queried world relations, e.g., relative object position, cross-player consistency) and FID (visual realism), plus auxiliary WorldScore:
$$\mathrm{WorldScore}=\frac{\frac{1}{K}\sum_{k=1}^{K}\mathrm{VLM}_k}{\frac{1}{K}\sum_{k=1}^{K}\mathrm{FID}_k},\quad K=5$$
| Method | Movement VLM↑ | Grounding VLM↑ | Memory VLM↑ | Building VLM↑ | Consistency VLM↑ | WorldScore↑ |
|---|---|---|---|---|---|---|
| Frame concat | 77.1 | 53.1 | 37.5 | 0.0 | 49.5 | 49.1 |
| Solaris* | 79.7 | 81.3 | 43.8 | 9.4 | 57.8 | 81.0 |
| W² (Ours) | 82.8 | 93.8 | 46.9 | 28.1 | 76.6 | 105.1 |
WorldScore rises to 105.1. Gains are strongest on state-sensitive categories: Grounding 81.3→93.8, Building 9.4→28.1, Consistency 57.8→76.6—persistent registers don't just improve fidelity but help maintain a logically coherent, verifiable world model across players and steps.
Figure 3 — Generated multi-agent rollouts with decoded world states: agent coordinates/trajectories, BEV maps (DINOv2 features PCA-projected to RGB), scene text.
3.2 MoT Architecture Ablation
| Model | Supervision | Movement VLM | Grounding VLM | Building VLM | Consistency VLM | WorldScore |
|---|---|---|---|---|---|---|
| Dense | none | 76.6 | 87.5 | 12.5 | 73.4 | 95.5 |
| MoT | none | 90.6 | 81.3 | 21.9 | 62.5 | 93.8 |
| Dense | scene text | 81.3 | 81.3 | 9.4 | 71.9 | 91.3 |
| MoT | scene text | 85.9 | 84.4 | 25.0 | 73.4 | 103.2 |
Without supervision MoT doesn't beat the dense backbone; under scene text supervision the dense model drops to 91.3 while MoT rises to 103.2. After freezing the visual Transformer and updating only the register Transformer in Stage 3, mean VLM rises from 63.8 to 68.1.
3.3 Semi-Supervised Extension
Fixed 1K labeled clips (with BEV) and gradually more unlabeled clips (diffusion loss only). Stage-2 trained 20K steps to avoid overfitting labeled data.
| Split (labeled/unlabeled) | Building VLM↑ | Memory VLM↑ | Consistency VLM↑ | WorldScore↑ |
|---|---|---|---|---|
| 1K / 0K | 15.6 | 21.9 | 71.9 | 63.2 |
| 1K / 2.5K | 15.6 | 65.6 | 56.3 | 64.4 |
| 1K / 5.0K | 28.1 | 81.3 | 71.9 | 82.3 |
| 1K / 10.0K | 25.0 | 84.3 | 68.8 | 90.3 |
Adding unlabeled data raises WorldScore from 63.2 to 90.3, with consistent VLM/FID improvement—a small supervised subset anchoring register semantics lets abundant unlabeled rollout data continue improving generation quality.
4. Main Contributions
- World state registers: persistent, dynamically updatable, cross-agent tokens explicitly carrying shared world info, unlike traditional approaches that bury state in frame history.
- Supervision space: from per-agent local status to global signals (BEV, scene text), grounding registers systematically.
- MoT architecture: separate weights for state and frame pathways in video diffusion, jointly generating world state and frames, improving performance and reducing objective competition.
- Semi-supervised extension: a small labeled subset anchors semantics while unlabeled videos keep improving generation—toward realistic data regimes where explicit world-state labels are scarce.
5.
流匹配训练损失
$$ \mathcal{L}_{\mathrm{flow}}=\mathbb{E}\left[\left\|\mathbf{v}_{\theta}(\mathbf{x}_{t},c,t)-(\epsilon-\mathbf{x}_{0})\right\|_{2}^{2}\right] $$
递归状态更新
$$ \mathbf{r}_{i}=G_{\theta}(\mathbf{r}_{i-1},\mathbf{x}_{i-W+1},\ldots,\mathbf{x}_{i},a_{i}) $$
智能体头损失
$$ d_{\mathrm{agent}}=\|\hat{\mathbf{y}}_{i}^{\mathrm{agent}}-\mathbf{y}_{i}^{\mathrm{agent}}\|_{2}^{2} $$
Limitations and Future WorkOur analysis: Experiments are limited to two-agent Minecraft; real-world complexity (multi-robot, physical interaction, open world) is unverified. WorldScore as an auxiliary metric is sensitive to FID scale and unbounded, so primary conclusions rest on VLM/FID. BEV supervision relies on a simulator god-view, requiring alternative supervision for real scenes. VLM accuracy itself depends on a vision-language model judge with its own uncertainty. MoT's separate learning-rate schedule (freeze visual, update register only) is an empirical engineering choice whose optimality at larger scale needs further study.
6. Conclusion
WorldWeaver's core idea: explicitly maintain a shared world state during streaming video generation instead of letting each agent repeatedly re-infer the world from frame history. The world state register persists across agents, updates with observations, and is grounded by agent pose, bird's-eye views, and scene text. With a Mixture-of-Transformers separating state and visual weights, the model lifts WorldScore from 81.0 to 105.1 on two-agent Minecraft, with the largest gains on Building (9.4→28.1) and Consistency (57.8→76.6). Semi-supervised experiments further show that once a small labeled subset anchors register semantics, abundant unlabeled video keeps improving generation—pointing toward data regimes where explicit world-state labels are hard to collect.



