Code World Model: Coding Agent as World Brain

The hard problem begins after the camera leaves

Most video world models frame interaction as conditional video prediction. Given previous frames and an action or prompt, the model generates the next observation. This formulation has produced increasingly convincing movement, camera control, and local interaction, but an observation exposes the outcome of a process rather than the rules that produced it. A rendered attack does not explicitly reveal range checks, collision rules, cooldowns, hit points, or the program that combines them.

The missing information becomes critical over long horizons. The paper uses an open-world example in which a player assassinates a city ruler and then leaves. Succession, public order, trade, faction relations, and the goals of many characters should keep changing off-screen. When the player returns, the world must reflect events that were never continuously visible. A short visual context or a retrieved frame can preserve appearance, but it does not by itself execute this causal chain.

The authors therefore argue that visual history is an awkward place to store a complete world. A practical world model needs state that can be inspected, updated, executed, tested, and revised. Language models have broad knowledge and can reason about exceptional events, yet calling one for every coordinate update or collision would be expensive and inconsistent. Code is a better substrate for frequent, repetitive, deterministic operations.

Code World Model is consequently more than an agent wrapped around a video generator. It assigns three computational regimes distinct responsibilities. The coding agent makes sparse, semantically difficult decisions. Executable code advances dense low-level state. A generative video model realizes the resulting world visually. This division of labor is the paper's central contribution.

Figure 1. The coding agent writes and runs world logic, the proxy exposes coarse spatial constraints, and the video model renders a detailed observation. The paper states that this overview image is AI-generated for illustration.

An executable state and a visual state

Let $S_t$ be the complete world state at time $t$, $A_t$ an action from a player, the environment, or another agent, and $O_t$ the visual observation. A conventional world model is written as:

$$S_{t+1}\sim p(S_{t+1}\mid S_t,A_t),\qquad O_{t+1}\sim p(O_{t+1}\mid S_{t+1}).$$

The action first changes the world and the updated world then produces an observation. In a video-only implementation, however, the complete $S_t$ is usually unavailable as an inspectable object. Visual history or latent activations are asked to carry state, memory, transition logic, and rendering at once.

Code World Model decomposes the state into executable and visual components:

$$S_t=\left(S_t^{\mathrm{exe}},S_t^{\mathrm{vis}}\right).$$

The executable state $S_t^{\mathrm{exe}}$ contains the evolving program, entity attributes, relationships, rules, and event history. The visual state $S_t^{\mathrm{vis}}$ contains generated appearance and motion information that should remain consistent. The first records what is true in the world; the second records how those truths have appeared.

The executable part advances through a joint agent-code transition:

$$S_{t+1}^{\mathrm{exe}}=\mathcal{T}_{\mathrm{AC}}\left(S_t^{\mathrm{exe}},A_t\right).$$

$\mathcal{T}_{\mathrm{AC}}$ should not be read as one model call per time step. Existing code repeatedly updates positions, collisions, schedules, and numerical rules. The coding agent intervenes when an event needs interpretation, a goal changes, an exception appears, or the operating mechanism itself must be revised. Reasoning frequency, code execution frequency, and video frame rate are deliberately decoupled.

The visual state is produced by a video model $G_\theta$:

$$S_{t+1}^{\mathrm{vis}}\sim G_\theta\left(S_t^{\mathrm{vis}},S_{t+1}^{\mathrm{exe}}\right).$$

This equation specifies a dependency rather than an input format. Raw source code and the complete database of world state are not sent directly to the neural network. The system selects observation-relevant information and communicates it through structured text and a frame-aligned visual proxy.

flowchart LR
    A[Interaction or Event] --> B[Coding Agent]
    B -->|write or invoke| C[Executable Code]
    C --> D[Persistent World State]
    D -->|semantic state| E[Structured Text]
    D -->|camera entities depth IDs| F[Proxy Compiler]
    F --> G[Frame aligned Proxy Video]
    E --> H[MiniMax H3 Video Model]
    G --> H
    H --> I[Visual Observation]
    I -->|feedback| B
    C -->|high frequency updates| D

Figure 2. An interaction is translated into code, code updates persistent state, and the same state produces complementary text and proxy conditions for visual generation.

Proxy: a narrow bridge from code to pixels

Structured text is the obvious interface. Identity records can store appearance, roles, goals, and relationships, while code updates position, orientation, and combat status. The authors report that recent video models still fail to recover precise frame-wise trajectories, occlusion, spatial relations, and camera paths from dense changing text. Language could theoretically describe every pixel, but doing so would be token-inefficient and too slow for interaction.

A complete explicit 3D scene provides stronger spatial control, but it also demands geometry, assets, rigging, animation, materials, lighting, physics, and a production renderer. In an open-ended world, every unforeseen object or behavior expands that asset pipeline. The final visual ceiling is then bounded by prepared content and simulator coverage.

The proxy occupies the middle ground. It preserves only the coarse structure that the current observation must obey: cameras, entity positions, poses, trajectories, spatial relationships, occlusion, and interaction-relevant state. A deterministic compiler rasterizes simple geometric primitives into a proxy video. Texture, material, detailed lighting, and small-scale motion remain unspecified so that the video model can supply them from its learned priors.

Text and proxy are derived from the same executable state but carry different information. Text communicates identity, appearance, and semantic intent. The proxy communicates where entities are, how they move, and how the camera sees them. Every signal is traceable to world state, leaving the state-to-condition path inspectable by the agent.

The design variable is condition bandwidth. A rich proxy grounds more details but forces the coding agent to maintain more primitives and parameters. Joint-level articulation, for example, would turn the agent into a motion controller, a task current coding systems do not perform reliably. The paper therefore seeks the minimum sufficient state that the observation must obey, falling back to a bounding box for complex objects and allowing text to resolve identity and appearance.

The implementation uses one quarter of the target resolution along each spatial dimension:

$$\frac{336\times192}{1344\times768}=\frac{1}{16}.$$

The proxy therefore has one sixteenth as many pixels as the target. This is the appeal of the interface: it supplies direct spatiotemporal grounding at much lower bandwidth than a full rendered world, while being more precise than prose.

Constructing aligned proxy-observation data

Games provide clean supervision because target RGB and runtime state can be captured from the same execution. The pipeline records camera state, entity position and orientation, approximate scale, layout, and interaction-critical variables. It then compiles a proxy along the exact target camera trajectory. Pixel-level instance maps bind identity descriptions in text to proxy regions, and one runtime recording can be recompiled with different proxy granularity without recollecting RGB video.

Real recordings lack game-engine state. The paper demonstrates an offline construction path on KITTI-360. Calibrated camera poses, accumulated 3D reconstruction, and 3D object annotations produce metric depth, surface normals, and coarse primitives. A shared depth buffer preserves alignment and occlusion. Reconstruction metadata is used to make the training condition and is not directly supplied to the video model.

Figure 3a. A game RGB target recorded synchronously with runtime state.

Figure 3b. Its aligned proxy: coarse people, vehicles, semantic regions, and depth specify structure rather than final appearance.

Figure 3c. A KITTI-360 target frame used to demonstrate proxy construction from real-world recordings.

Figure 3d. The offline-compiled real-data proxy preserves scene layout, object location, camera motion, and occlusion.

Training and inference

The authors adapt the official MiniMax-H3 Ref2VA backbone with LoRA. The dataset contains 157 gameplay takes, about 5.6 hours of source video. Sampling five-second clips every two seconds yields 9,420 training examples. Each target contains 124 frames at 1344×768 and 24 FPS; the aligned condition contains 124 proxy frames at 336×192 with fixed-log depth and semantic identity.

AdamW uses weight decay 0.01 and gradient-norm clipping at 1.0. The learning rate follows cosine decay from $2\times10^{-5}$ to $1\times10^{-6}$. Inference uses the step-3,534 checkpoint and 20 explicit-Euler sampling steps. GPT Image 2 generates a 1536×864 first-frame appearance anchor from the initial proxy and a coding-agent-written prompt; the anchor is resized to 1344×768 and supplied both as target latent slot zero and through the appearance-reference branch.

Long video generation uses overlapping 124-frame windows. The final 34 RGB frames of the previous window condition the next, so every continuation contributes 90 new frames. With $K$ windows, the stitched length is:

$$N_{\mathrm{out}}=124+90(K-1).$$

The overlap supports local temporal continuity, while reusing the appearance anchor helps preserve global identity and style. This technique reduces visible seams, but it should not be confused with evidence that arbitrarily long causal state remains correct.

What the released code establishes

The official repository releases the MiniMax-H3 inference component, CWM LoRA weights, 40 compact multi-window condition examples, and utilities for configuration validation, cache preparation, generation, recovery, and stitching. In src/cwm_h3_inference/duv.py, metric depth and a semantic identity map are converted into a three-channel condition. Depth uses a fixed logarithmic mapping and semantic IDs are restricted to twelve classes from void and sky through humans, animals, vehicles, and props.

# src/cwm_h3_inference/duv.py
normalized = (log(far) - log(depth)) / (log(far) - log(near))
depth_code = round(clip(normalized, 0, 1) * 65535)
condition = stack(depth_code, semantic_u, semantic_v)

src/cwm_h3_inference/engine.py implements the Retake34 continuation procedure. It decodes the previous window, takes the final 34 frames as a fixed prefix for the next latent sequence, denoises the remaining positions, and later discards the regenerated overlap during stitching.

# src/cwm_h3_inference/engine.py
tail = decoded_video[:, :, -OVERLAP_FRAMES:]
video_mask = ones_like(clean_video, dtype=bool)
video_mask[:, :, :VIDEO_PREFIX_LATENTS] = False
# Later windows omit their regenerated overlap during stitching.
pixels = pixels[OVERLAP_FRAMES:]

The release has a material limitation: it does not include the tools that turn coding-agent-controlled game execution into depth and semantic-ID proxy inputs. The authors state that this part depends on a closed-source game-code foundation that cannot be redistributed. The public package can reproduce inference on supplied conditions, but it does not independently reproduce the full agent-to-code-to-proxy claim.

What the experiments actually demonstrate

The evidence is primarily qualitative. Figure 4 and the project videos cover diverse characters, environments, actions, styles, and camera trajectories. The adapted model appears able to follow proxy-specified positions, layout, and movement while rendering appearance far outside the visual domain of the small GTA V fine-tuning set. A frame-aligned proxy plainly offers finer control than a single action label or a low-dimensional camera command.

The paper does not report FVD, trajectory error, identity consistency, proxy adherence, or human-preference scores. It also lacks ablations that isolate text, depth, semantic identity, the first-frame anchor, and the 34-frame overlap. The comparison explicitly excludes latency. Claims of greater precision and responsiveness therefore rest on selected demonstration videos rather than a statistically characterized benchmark.

More fundamentally, the experiments test whether a video model can understand the proxy interface. They do not test the complete open-world scenario used to motivate the work. There is no measured chain of consequences across locations, characters, and long periods, and no automated consistency check between executable state and generated pixels. If code says an object is inside a cabinet while the video shows it on a table, the system has no described procedure for detecting which representation failed or repairing the mismatch.

Why this matters for embodied AI

The decomposition is useful for embodied learning. A robot environment needs explicit task state, collision logic, and reward rules, but it also benefits from observations that approach real-world appearance, occlusion, and motion diversity. Executable state can remain an auditable source of truth; a video model can expand the observation distribution; and a programmable proxy can connect them without requiring production-quality assets for every visual variation.

Robot training, however, is unusually sensitive to contact correctness. Whether a grasp slips, a finger penetrates an object, or torque exceeds a limit cannot be decided by visual plausibility. A defensible near-term use would keep physics simulation or real sensor logs in charge of task truth and use generation for visual realization and domain variation. Treating the video generator itself as a dynamics simulator would require substantially stronger state-pixel consistency and sim-to-real evidence.

Limitations and open questions

The authors identify two primary limitations. Compute constraints keep the training scale and resulting visual quality limited, and the system does not perform autoregressive real-time generation. Every new observation requires generative inference, producing higher marginal latency and cost than conventional rendering. Current coding agents also cannot reliably implement complex collisions, driving systems, and large game mechanisms from scratch, so the prototype extends an existing code foundation rather than autonomously creating a complete open world.

The proxy may itself become a new engineering bottleneck. If it is too sparse, generation can violate state; if it is too detailed, the agent must maintain something close to a conventional 3D engine. The paper proposes adaptive selection of entities, regions, resolution, frame rate, and state types, but evaluates only one fixed proxy design.

A further limitation is the dual-state consistency problem. Executable and visual states need monitors, attribution, and repair when they disagree. For robotic use, future work should report trajectory deviation, contact consistency, reappearance after occlusion, long-horizon identity retention, intervention reproducibility, inference latency, and downstream policy transfer.

Conclusion

Code World Model is best read as a systems architecture. A coding agent and executable program preserve the causal skeleton of a world; a video model supplies its sensory surface; and a low-bandwidth proxy links the two. The proxy-conditioned MiniMax-H3 results and public inference implementation show that this interface is workable and visually compelling.

It remains a proof of concept. The work validates a key component of executable-state-driven observation generation, but not yet persistent open-world causality, real-time interaction, autonomous world construction, or a closed consistency loop. Calling it an operating-system sketch for future world models is more accurate than calling it a completed general simulator.