Do Coding Agents Need Executable World Models, Simplification, and Verification to Solve ARC-AGI-3?

Do Coding Agents Need Executable World Models, Simplification, and Verification to Solve ARC-AGI-3?

Paper: Do Coding Agents Need Executable World Models, Simplification, and Verification to Solve ARC-AGI-3?

Author: Sergey Rodionov (SingularityNET)

Links: arXiv:2607.15439

---

One-Line Summary

Through four nested Codex-based agent variants (textual baseline, executable world model, +simplification, +verification), this paper ablates the individual contributions of executable world modeling, simplification, and verification on ARC-AGI-3, finding that the verification variant ranks first in all settings but model capability and reasoning budget are the most consistent performance drivers — the gpt-5.6-sol verification variant fully solves all public games using fewer than half the human baseline's actions.

---

Background and Motivation

ARC-AGI-3 evaluates exploration, modeling, goal discovery, planning, and execution in novel abstract turn-based environments. Its primary metric RHAE (Relative Human Action Efficiency) rewards level completion while penalizing excess actions relative to a human first-contact baseline, turning abstraction into an online system-identification problem where each action is simultaneously a control decision, a possible experiment, and a scarce resource.

The author's previous agent bundled executable world modeling, scheduled simplification, and exact replay verification, leaving unclear which idea accounted for performance. This paper addresses the attribution question through four nested variants.

---

Four Nested Agent Variants

Figure 1: Prompt-level controller shared by all four variants. Simplification stages (asterisk) are enabled only for simplification and verification variants.

Table 1: Four nested agent variants

Variant	World-Model Representation	Simplification	Verification
twma_v1.2 (textual)	Textual model, no maintained executable simulator	No	No
ewma_v1.2 (executable)	Text + executable model and planning code, agent-chosen interfaces	No	No
ewma_s_v1.2 (simplification)	Same as above	Yes	No
ewma_sv_v1.2 (verification)	Text + fixed-interface executable engine, state reconstruction, renderer, planner	Yes	Yes

Controller: A thin prompt-level controller around the Codex runtime with no game-solving logic. It selects from three protocols: Normal (has new actions), Reset (GAME_OVER analysis + RESET), and Stuck (no new actions → reminder).

Simplification follows the Minimum Description Length principle, minimizing model description length while maintaining observation consistency:

$$ \mathcal{M}^* = \arg\min_{\mathcal{M}} \left[ L(\mathcal{M}) + L(\mathcal{O} | \mathcal{M}) \right] $$

where $L(\mathcal{M})$ is model description length and $L(\mathcal{O}|\mathcal{M})$ is observation encoding length given the model.

Verification requires maintaining a fixed-interface executable model $\hat{f}$ that exactly reproduces all observed transitions. For interaction history $\{(o_t, a_t, o_{t+1})\}_{t=0}^{T}$:

$$ \forall t \in [0, T]: \quad \hat{f}(o_t, a_t) = o_{t+1} $$

Any mismatch becomes a concrete counterexample without further environment interaction.

The RHAE metric is defined as:

$$ \mathrm{RHAE} = \frac{1}{|G|}\sum_{g \in G} \frac{1}{|L_g|}\sum_{l \in L_g} \max\left(0,\, 1 - \frac{a_{\mathrm{agent}}(l) - a_{\mathrm{human}}(l)}{a_{\mathrm{human}}(l)}\right) \times 100 $$

where $G$ is the game set, $L_g$ the levels of game $g$, and $a_{\mathrm{agent}}(l)$, $a_{\mathrm{human}}(l)$ the action counts.

The action efficiency ratio is $\eta(l) = a_{\mathrm{human}}(l) / a_{\mathrm{agent}}(l)$; $\eta > 1$ means the agent outperforms humans. In follow-ups, the verification variant used $7{,}758$-$8{,}347$ total actions versus a human baseline of $17{,}135$:

$$ \eta_{\mathrm{total}} = \frac{17{,}135}{7{,}758} \approx 2.21 $$

Cost-token proxy for resource use:

$$ C = \frac{T_{\mathrm{cached}}}{60} + \frac{T_{\mathrm{input}} - T_{\mathrm{cached}}}{6} + T_{\mathrm{output}} $$

flowchart TD
    A["Game Init
Get Initial Obs"] --> B["Send Main Prompt + Stop"]
    B --> C["Codex Executes
Inspect/Edit/Run/Submit"]
    C --> D{Check Stop Conditions}
    D -->|"Complete/GAME_OVER"| E["End"]
    D -->|"Continue"| F{State Check}
    F -->|"Normal (new actions)"| G["Optional Simplify
+ Continue"]
    F -->|"GAME_OVER"| H["Death Analysis
+ Simplify + RESET"]
    F -->|"Stuck (no new actions)"| I["Optional Simplify
+ Reminder"]
    G --> C
    H --> C
    I --> C
    style C fill:#e1f5fe
    style E fill:#e8f5e9

---

Results

Figure 3: Nested ablation structure — four variants progressively add executable model, simplification, and verification.

The main study evaluates a 4×2×2 design on 25 public ARC-AGI-3 games.

Figure 2: Mean RHAE across 25 public games, grouped by variant, model, and reasoning effort.

Table 2: Main study RHAE results (25 public games average)

Model	Effort	Textual	Executable	Simplification	Verification
gpt-5.4	high	34.36	33.74	31.42	39.98
gpt-5.4	xhigh	46.73	50.78	54.17	54.79
gpt-5.5	high	58.85	51.16	58.35	62.17
gpt-5.5	xhigh	72.51	69.70	73.09	74.78

Key findings:

• Capability and effort dominate: All 16 matched comparisons are consistent — xhigh > high, gpt-5.5 > gpt-5.4, no exceptions
• Executable deliverable not universally beneficial: For gpt-5.5, textual outperforms executable at both efforts (58.85 vs 51.16, 72.51 vs 69.70)
• Simplification usually helps: Improves in 3 of 4 settings (3.39-8.38 points), except gpt-5.4-high (-2.94)
• Verification always first: Ranks #1 in all 4 settings, but uses the most resources

Table 3: Cost tokens (millions), summed over 25 games

Model	Effort	Textual	Executable	Simplification	Verification
gpt-5.4	high	81.30	66.38	127.27	147.65
gpt-5.4	xhigh	97.65	103.59	176.51	204.01
gpt-5.5	high	63.17	53.87	138.06	205.10
gpt-5.5	xhigh	68.16	74.02	163.47	222.06

Exploratory Follow-up

With gpt-5.6-sol on v1.6 config, the verification variant fully solves all 183 public levels at both xhigh and max, achieving ~99% RHAE with fewer than half the human baseline's actions.

Table 4: Exploratory follow-up results

Variant	Model	Effort	RHAE	Unsolved	Actions	Cost(M)
Verification v1.5	gpt-5.5	xhigh	82.01	5/9	—	207.41
Textual v1.6	gpt-5.6-sol	xhigh	92.34	2/5	—	32.32
Verification v1.6	gpt-5.6-sol	xhigh	98.97	0/0	8,347	90.75
Textual v1.6	gpt-5.6-sol	max	95.97	0/0	10,111	30.60
Verification v1.6	gpt-5.6-sol	max	98.77	0/0	7,758	103.49

---

Limitations

1. Single run: Each game is played once per condition, unable to estimate run-to-run variability. Individual game performance can differ substantially across playthroughs.
2. Public set saturation: gpt-5.6-sol was trained after public games became available; fully solving the public set may reflect memorization rather than genuine generalization. Held-out performance is untested.
3. Nested ablation: The design follows a nested ablation ladder, not comparing all possible component combinations (e.g., verification without simplification is not evaluated). The verification variant changes both the verification objective and supporting workspace simultaneously.

---

Conclusion

This paper provides a systematic ablation of coding agents on ARC-AGI-3. The most robust conclusion: model capability and reasoning budget are the most consistent performance drivers, with no variant reaching a plateau within the tested range. An executable world model as a persistent deliverable is not universally beneficial, simplification usually helps, and verification consistently ranks first but at high resource cost. On gpt-5.6-sol, the verification variant fully solves all public games using fewer than half the human baseline's actions.

Key insight: "Stronger coding models and larger reasoning budgets would continue to improve performance across these architectures." The unusually consistent direction of all 16 matched capability comparisons is the core finding established directly by the experiment.