How does Astra's computer use actually work?

Last week, OpenAI released GPT-6 Astra, its most capable model to date. The launch emphasized computer use, and the accompanying Blender demonstration made the model’s ability to operate a computer unusually visible.

The launch video extends “Put That There”, a 1979 demonstration from MIT’s Architecture Machine Group, the predecessor of the MIT Media Lab. In the original clip, researchers tell the computer to “put that there,” and it places shapes on a projector screen. At the time of writing, OpenAI’s post had reached 131 million views.

Computers using computers is an old ambition. Astra’s launch made it newly practical. Kyle Jeong, who has worked on deploying computer use at Browserbase for three years, traces why the new model works so well.

Computer use here means an AI system’s ability to autonomously control a computer-based system. The observation can be a screenshot, video, text, or a structured interface state; the action can be a click, a keystroke, or code that drives the system.

A brief history of computer use

Anthropic introduced computer use with Claude 3.5 Sonnet in October 2024. The first generation was primarily vision-based: the model received screenshots and returned pixel coordinates plus a JSON action.

{
  "action": {
    "type": "click",
    "x": 156,
    "y": 50
  }
}

Drivers such as Stagehand or Playwright translated that action into browser or operating-system interaction. OpenAI later shipped Operator and computer-use-preview, while Google DeepMind added computer use to Gemini 2.5 Pro.

The limitation was the pixel-centric design. Models were often post-trained against a fixed viewport; using a different window size could make them miss buttons or fail outright. Vision-only agents also struggled with controls and state changes that are not obvious from a screenshot alone.

Text-only and hybrid DOM-plus-vision designs continued to develop. Standard Intelligence’s FDM-1 is one experimental direction, encoding video rather than static screenshots for computer use.

What is different about Astra

Astra is the newest point in OpenAI’s 5.6 model family, and its Codex/ChatGPT harness is as important as the model itself. Computer use is jointly a research problem and an engineering problem: the model decides what to do; the harness observes the computer and executes the action.

Codex made the workflow feel materially better than earlier computer-use products. It can open an in-app browser or control the user’s browser, perform tasks in the background, write code, execute code, call tools, or build charts while continuing a task. Astra takes that 5.6 capability and makes it faster and cheaper, at least in its maximum-thinking configuration.

The key shift is the accessibility tree. Modern operating systems and Chrome expose a semantic representation of the interface for users who need accessibility support. It preserves the roles, names, states, and relationships of interface elements while omitting CSS and implementation classes.

That tree is compact and high-signal. It often costs fewer tokens than a screenshot while giving the model equal or better context. Astra uses it to dispatch actions such as click, type, and press. Chrome generates accessibility trees for sites automatically, so many web applications are compatible without custom integration.

The architecture

The control loop is straightforward once the model and the local harness cooperate.

When a session starts, Codex launches a Node REPL that persists session state. Its bindings cover browser control and native computer control. The agent chooses the binding appropriate to the task.

For a browser or native application, the agent observes text, a screenshot, or both, then writes code to choose and perform the next action. OpenAI’s documentation recommends using code execution for computer use rather than emitting individual pixel actions.

A call can therefore look like a JavaScript action instead of raw coordinates:

{
  "type": "function_call",
  "name": "exec_js",
  "call_id": "call_123",
  "arguments": "{\"code\":\"await page.getByRole('searchbox').fill('browser automation'); ...\"}"
}

The local service, CodexComputerUseIPC-5, executes it. The wrapper resolves an element selection to a native element ID, or coordinates if the model chooses them, then communicates over a native pipe transport with JSON-RPC messages and request IDs.

OpenAI’s sample app recommends Playwright for browser control and PyAutoGUI for native desktop control. After an action is sent, Astra requests another observation and compares actual state with intended state. Only then does the loop continue. The Node REPL remains alive for the whole task because computer use is inherently stateful.

Safety review

In the capability comparison above, Astra is the only model marked as requiring auto-review. Its Guardian policy reviews proposed computer actions before execution. GPT 5.6 Luna acts as a background classifier that scores the current workflow and upcoming risk. A high classification escalates later actions to a blocking reviewer, which performs a fuller evaluation.

{
  "risk_level": "high",
  "user_authorization": "low",
  "outcome": "deny",
  "rationale": "..."
}

The policy receives the proposed action and arguments, conversation evidence including user authorization, the parent environment and permission context, available REPL evidence and images, and the approval request. It does not always need the full accessibility tree to classify the action.

Common Guardian blocks include:

• Permission grants: specific authorization for the permission and recipient.
• Sign-in and consequential account actions: whether the user specifically authorized them.
• Sensitive data submission: permission for both the data and destination.
• Consequential clicks: interface state, effects, incorrect form inputs or settings, and consistency with instructions.
• Restriction bypass: whether an alternative route was authorized.
• Destructive actions: meaningful state loss or irreversible damage.
• Out-of-scope private-data access: whether access belongs to the authorized task.

On alignment benchmarks, Astra improves substantially over its predecessor.

Why it is faster

If Astra performs the same task family as GPT 5.6 and adds review, its speed advantage comes from needing fewer turns. It is a smarter model, heavily reinforcement-trained in computer-use environments, so it reaches the intended state in fewer observation-action cycles.

Astra was reportedly post-trained on 100,000 GB300s, so the compute investment was large. Raw inference speed matters less once a model exceeds roughly 300 tokens per second; action execution becomes the bottleneck. Harness optimizations such as WebSocket prewarming, connection reuse, and incremental requests using previous_response_id reduce startup latency, not the speed of each action.

Current failure modes

Astra remains fallible. It can receive an incomplete accessibility state, an insufficient screenshot, or a stale view. Interface state can drift between observation and action. In some applications the accessibility tree changes too frequently, invalidating the element the agent intends to manipulate.

Long-horizon computer use is also unsolved. Compaction allows Astra to run at high fidelity for a long time, but reliability over days or weeks remains unproven.

The frontier

Computer use is entering a more useful phase. Systems that once failed at trivial tasks can now complete substantial workflows, and public demonstrations range from Minecraft progress to multi-application work. The biggest gain came from moving beyond screenshots to semantic interface state.

Astra combines that shift with speed and guardrails. Each iteration moves production-ready computer use closer, especially for delegating routine digital work so people can focus on problems that still need human judgment.

Sources:

• OpenAI Codex repository
• GPT-6 Astra launch blog
• OpenAI computer-use sample app
• OpenAI computer-use API guide

Source: How does Astra’s computer use actually work? — Kyle Jeong. X post: @kylejeong/status/2097077446663372966.