VISTA: A Visual Harness That Lets Multimodal Models Play ARC-AGI-3 From Raw Pixels

A Visual Harness, Not a Program Synthesizer

VISTA ("A Visual Harness for Reasoning in an Interactive World") is a small MIT project that asks a pointed question: how much of the machinery usually bolted onto multimodal agents is actually necessary? The answer it demonstrates on ARC-AGI-3 is "less than you think." The harness gives a general-purpose multimodal model three things and nothing else: raw PNG observations of the environment, free-form language to reason in, and a lossless archive of every frame the environment has ever returned. There is no world model compiled into code, no task-specific planner, no fine-tuning.

The team is Qiushi Han, Keya Hu and Linlu Qiu (co-leads) with Cathy Wu and Kaiming He at the Massachusetts Institute of Technology; the accompanying blog post is dated 5 August 2026 and the code ships under the MIT license. With Claude Opus 5.0 as the backend, VISTA completes all 25 public ARC-AGI-3 games, 183 levels in total, at a 100% win rate and a perfect Relative Human Action Efficiency (RHAE) score of 100, using 56% fewer game actions than first-time human players. With GPT-5.6 Sol through the Codex CLI it reaches a mean score of 98.27 across the same 25 games.

What ARC-AGI-3 Asks of an Agent

ARC-AGI-1 and ARC-AGI-2 present static input-output transformations: here is a grid, here are examples, infer the rule. ARC-AGI-3 replaces that with interactive visual games. Each game starts with no instructions, no stated rules and no stated goal. At every step the agent sees the current state, picks one of the available actions, and reads the resulting state as evidence about what its action did and how the world works. A game is a sequence of levels that share underlying mechanics while introducing new objects, layouts or constraints, so understanding inferred on level 3 has to survive into level 7.

Scoring combines completion with efficiency. Relative Human Action Efficiency credits an agent for solving levels in fewer actions than a first-time human player: a completed level earns full credit when the agent uses no more actions than the human baseline, partial credit when it completes but spends more, and zero when it does not finish. Later levels carry greater weight and the final score is averaged across games. Only environment actions count toward the total; internal reasoning and read-only inspection are free under the protocol, which is exactly the asymmetry VISTA's design leans on.

Design One: Perception Is Raw Pixels

The agent observes the world by reading 2D PNG images, with no symbolic preprocessing in between. Concretely it receives a 512 by 512 PNG of the current rendered state: a nearest-neighbor 8x upscale of the official 64 by 64 frame, with one-pixel grid lines drawn between cells. The agent is never told that the world is a 64 by 64 grid; the PNG is all it gets. It may additionally request an enlarged view of any rectangular region of the board, so its information access is in principle intact. Nothing is hidden, only not yet looked at.

The point of keeping perception raw is that the model has to bring its own visual priors to the task, and the authors observe that it does: the agents form and discover visual concepts on their own rather than being handed an object list. The design is also not tied to 2D; the same harness runs on a 3D rendering of the same games, which the results section returns to.

Design Two: Reasoning Is Free-Form Language

Observations become hypotheses in natural language, with no constraint on what the agent reasons about, which language it reasons in, or how long it spends. It takes notes, revises them as evidence accumulates, and uses them to choose the next action. The deliberate contrast is with program-based world models, the dominant recipe on the ARC-AGI leaderboards, where agents such as Schema, Tycho and Retrodict distill exploration traces into an executable reconstruction of the game: a state representation, transition rules and goal conditions that can be checked against past observations and searched over to pick an action.

VISTA's argument is that most of the reasoning that leads to a good action does not need to be executable. The blog makes the case with the jumping mechanic of game LF52. A faithful program for that one mechanic has to name the board, the coordinates of every object, the legality conditions of a move and the state update it produces; the released Schema world model for LF52 runs to roughly 4,000 lines of Python, and the move-generation function quoted in the post is one of 71 functions that calls three of the others. Three sentences the VISTA agent wrote into its notes convey the same rule well enough to play perfectly:

• Green pegs make standard orthogonal jumps on 48 px-spaced gray holes: click source, then an empty landing two cells away. A jumped standard green is removed.
• Purple/pink pedestal pieces are fixed persistent jump posts. Any movable piece can jump across one; the post remains.
• Red cross pieces are movable persistent jumpers/posts. Red and green can jump over each other without removing either, so alternating jumps move a persistent pair along a line.

One page of notes versus four thousand lines of Python, and the notes win on score. Language is implicit and fuzzy where a program is explicit and executable, and for this benchmark fuzzy turned out to be enough.

Design Three: Lossless Visual Memory

In a standard VLM or LLM agent the only memory is the KV cache: implicit, compressed, lossy, and bounded by the context window. Older visual detail is exactly what falls out of it first. VISTA replaces that with an explicit archive: every frame the environment returns is stored at full resolution, indexed by turn and frame index. Nothing is summarized away.

The agent reaches into that archive with two tools. inspect brings an earlier state, an intermediate animation frame, or an enlarged spatial region back into view through the same visual channel as the live observation, and several views can be requested together for read-only comparison. read_pixels returns exact color samples from a selected region, for small discrete details that are awkward to eyeball. The authors call this an explicit attention mechanism at frame, region and pixel level: the model decides which past moment matters, rather than a fixed pipeline deciding for it.

Memory mechanism	What is kept	What is lost	How it comes back
Context window	Recent turns, as tokens and KV-cache activations	Everything outside the window; compaction decides what goes, and older visual detail is compressed or dropped	Implicit attention over what still fits
Program world model	Observations distilled into code: state, transitions, goal conditions	Anything the reconstruction does not model, and the past frames themselves	Executing the reconstruction
Written text notes	The model's own description of what happened	Everything not written down; the model chooses what to drop	Re-reading its own text
Lossless visual memory (VISTA)	Every returned frame at full resolution, indexed by turn and frame	Nothing	inspect and read_pixels, on the model's own decision

Notes still exist, but as a complement rather than the memory itself: GUIDE.md holds a compact, high-level model of the game that should remain useful across levels, and WORKING.md is a scratchpad for the current level.

The Loop, the Notes, and a Four-Line Prompt

One agent plays each game from its first observation to completion. Each turn is a three-step cycle: observe the current visual state and available actions; reason in free-form language, hypothesizing, predicting and planning; act by calling play, which executes exactly one game action. Every frame a single action produces, including intermediate animation frames, enters visual memory in order, and the last returned frame becomes the next observation. Looking back is available but optional: whether to re-examine any past moment at all is the model's decision, not a fixed step of the loop.

Two words in the prompt carry most of the design weight. The agent is asked to build a compact model of the game, pushing it toward higher-level abstractions in the spirit of Occam's razor, and a revisable one, so that new evidence corrects the model instead of piling onto it. Before each play the prompt asks the agent to state what it expects to see, and afterwards to state every visible change, expected or not, a cheap prediction-error loop that keeps the game model honest. When the model approaches its context limit it writes a concise continuation state and resumes from the current visual state in a fresh context; notes, visual memory and the action history all remain available, so the handoff costs continuity but not evidence.

The whole instruction set is short enough to quote in full:

# Visual game task
Complete the game with as few game actions as possible.

Build and use a compact, revisable model of the game and its current state. Update it as new evidence changes what is supported.

Before each `play`, briefly state what you expect to see. Afterward, briefly state all visible changes, expected or not.

Keep concise, durable, revisable game understanding in `GUIDE.md`; use `WORKING.md` as a scratchpad when useful.

The tool surface is equally small: play to execute a game action, inspect to revisit selected frames and regions, read_pixels to sample exact colors, history to revisit prior actions and environment results, plus the two note files.

Results: 25 of 25 Games, 183 of 183 Levels

Both backends were run independently over all 25 public games with the same prompt, tools and reasoning setting throughout. With Claude Opus 5.0 through the Claude Code CLI at xhigh effort, VISTA completes all 183 levels across all 25 games: mean game score 100.00, 25 perfect games out of 25, and 7,542 game actions against the 17,135-action first-time-human reference, 56% fewer actions for a perfect sweep. With GPT-5.6 Sol through the Codex CLI at max effort it also completes all 183 levels, with a mean score of 98.27 and 22 perfect games; the missing 1.73 points concentrate in a handful of levels where the model spends extra actions discovering a mechanism or recovering from an incorrect game model, using 10,063 actions against the same human reference.

Backend	Runtime	Effort	Mean score	Games completed	Perfect games	Actions (agent / human)
Claude Opus 5.0	Claude Code CLI	xhigh	100.00	25 / 25	25 / 25	7,542 / 17,135
GPT-5.6 Sol	Codex CLI	max	98.27	25 / 25	22 / 25	10,063 / 17,135

The per-game tables show where the efficiency comes from. On ar25 the agent finishes in 270 actions against 748 human ones; on cn04, 235 against 789, including a level 4 solved in 30 actions where humans needed 300; on ft09 it clears level 1 in 4 actions against a human 43. The exceptions are instructive too: on bp35 level 5 the agent spends 56 actions where humans needed 33, and on g50t level 6 it takes 59 against 54, the levels where its inferred model needed repair.

Where It Sits on the Leaderboard

The interesting comparison is not VISTA against humans but VISTA against the program-synthesis systems that currently top the ARC-AGI Community Leaderboard, most of them concurrent work published within days of the blog post. VISTA is, to the authors' knowledge, the first system to reach a perfect or near-perfect score without program synthesis. The official organizer baselines, which also avoid program synthesis, score 30.16 with Opus 5.0 and 13.33 with GPT-5.6 Sol.

System	Program-based	Model	Reasoning effort	RHAE
Official minimal interface	No	GPT-5.6 Sol	max	13.33
Official minimal interface	No	Opus 5.0	high	30.16
Schema	Yes	GPT-5.6 Sol	xhigh then max	95.35
Schema	Yes	Opus 4.8 then Fable 5	max	98.98
ewma_sv_v1.6	Yes	GPT-5.6 Sol	xhigh	98.97
Retrodict	Yes	GPT-5.6 Sol	max	99.86
Tycho	Yes	GPT-5.6 Sol	max	100.00
Tycho	Yes	Opus 5.0	xhigh	100.00
VISTA (this project)	No	GPT-5.6 Sol	max	98.27
VISTA (this project)	No	Opus 5.0	xhigh	100.00

The fallback in the Schema row is worth reading closely: it runs Opus 4.8 and Sol at xhigh first, reruns any game scoring below 80 with Fable 5 and Sol at max, and keeps the higher per-game score. Against that, a page of notes plus a frame archive is doing the work of a compiler.

Swap the Representation, Keep the Agent

Because the harness treats observations as opaque sensory input, the same world can be handed over in different dimensions. The blog tests three: a 1D textual grid of 64 by 64 integers with one integer per color, the original 2D image, and a 3D rendering of the same game. The mappings are fully reversible and preserve the same information in theory, but they do not support the same computation in practice.

VISTA played S5I5 (extend two colored structures until they reach their targets) and CD82 (transform a canvas to match a reference) from each representation alone. In all three settings it inferred how both games worked and completed level 1; the three runs are independent trajectories, not identical action sequences. The text grid works but costs more tokens; the 3D rendering works too, and is the variant the authors point to as evidence that the design transfers toward embodied environments, where observations are closer to how humans perceive the physical world.

Running It

VISTA runs on Linux x86_64 with Python 3.12 or newer and Docker Engine, plus one of two agent runtimes: Codex CLI 0.145.0 or Claude Code 2.1.220. Online and competition runs need an ARC-AGI-3 API key from the ARC Prize platform; an offline mode plays local game files supplied through ENVIRONMENTS_DIR. From a checkout:

python3 -m venv .venv
.venv/bin/python -m pip install --upgrade pip
.venv/bin/python -m pip install -e .
cp .env.example .env
chmod 600 .env

Then add the ARC key to .env (ARC_API_KEY plus ARC_BASE_URL=https://three.arcprize.org), authenticate the chosen runtime (codex login for Codex CLI, or setup-token plus CLAUDE_CODE_OAUTH_TOKEN for Claude Code), and build the matching player image:

# Codex CLI player
docker build -t arc3-codex-player:0.1 -f Dockerfile.codex-player .

# Claude Code player
docker build -t arc3-claude-player:0.1 -f Dockerfile.claude-player .

A single game and a full sweep:

# one game, one backend
.venv/bin/python scripts/run_arc3_codex.py --game-id s5i5 --model gpt-5.6-sol --effort max
.venv/bin/python scripts/run_arc3_claude.py --game-id s5i5 --model opus --effort xhigh

# all 25 public games, two jobs in parallel
./scripts/run_batch.sh --runtime codex --mode online --model gpt-5.6-sol --effort max -j 2
./scripts/run_batch.sh --runtime claude --mode online --model opus --effort xhigh -j 2

Each run records the full trajectory, game actions, public model output, visual inspections and agent notes, which is what the blog's per-game replay links serve.

Caveats and What Comes Next

The authors are explicit about what the result does and does not show. The models' existing capabilities are the key factor; the harness is a simple but effective way of eliciting them. Both backends were released after the public ARC-AGI-3 games existed, so training-time contamination cannot be ruled out, and the private game set remains the real test of generalization. ARC-AGI-3 is a testbed, not the target.

The target is the design itself: raw high-dimensional observation, free-form language reasoning, and lossless recall of past experience are not ARC-specific. The authors position VISTA as a template for any interactive environment, and the 3D-representation result is the first step toward the case they care about, embodied environments closer to the physical world, where an agent has to discover how a place works by looking at it and acting in it.