Skip to content
←Back to Blog
Getting Robot-Use Agents to Run in Real Time
Robot-Use AgentsRobot HarnessReal-Time Inference

Getting Robot-Use Agents to Run in Real Time

FastRUA keeps the same robot agent model and redesigns its harness: fewer round trips, short context, faster tools, and motion-aware pipelining cut a cup task from 667s to 204s.

Shaoxiong Duan, Yang Yuanbo, Junda Chen, Quanquan Peng, Hao ZhangNovember 20, 202620 min read84 views
ZH
The post in 50 seconds. Opening footage: Anthropic; NVIDIA ASPIRE.

In brief. Robot-use agents are still far too slow for the physical world. Simulated benchmarks such as RoboDojo can hide that cost because the simulator waits while the model thinks. On a real robot, the clock keeps running and the robot remains idle until the next command arrives. FastRUA keeps the model unchanged and redesigns the harness around it: fewer requests, shorter context, faster tools, and motion-aware pipelining. Finished towel-folding runs fell from a median of roughly 18 minutes to roughly 6 minutes, while the robot moved during 84% of the cup task. The remaining problem is latency: the robot still acts on images that are about five seconds old.

Code: github.com/fast-rua/making-rua-fast contains the robot API, robot server, agent harness, and a small simulator for reproducing the experiments.

Over the past year, robot-use agents have become a serious alternative to end-to-end robot policies. GPT-6 Astra, controlling a robot through code, scored above all 40 public policies on RoboDojo [1]. Uni-robot-agent placed open-weight models such as Qwen3.8-Flash-Next on the same benchmark, well ahead of VLA and world-action-model baselines including π0.5 and OpenWAM-α [2]. Anthropic reports 64.9% on RoboDojo Sim for Fable 5.2 [3].

On a real robot, however, these agents are slow. In Fable 5.2's launch video, the robot solves a puzzle cube [3], but even with the pauses removed, the solve takes about an hour.

Benchmarks such as RoboDojo do not capture this gap. Their evaluation loop is synchronous: the simulator blocks until the policy returns its next actions, and episodes are limited by the number of steps rather than by wall-clock time [4]. In that setting, a long time to first token (TTFT) or slow decoding costs nothing. JustRUA, a controlled study of GPT-6 Astra agents on RoboDojo, says as much: it reports what an episode consumes rather than how long it takes, because simulation time can be hidden by evaluating episodes in parallel [5]. A physical robot has no such luxury. Its clock keeps running while the model thinks, and its arms wait for the next command.

Comparison of a simulation loop that pauses while the model responds with a real robot clock that continues while the model thinks
Figure 1. A simulator that waits for the model only counts actions. On a real robot, every model step is time the robot spends standing still.

This article describes the first attempts to close that gap on a real two-arm robot. The model was not changed. Every intervention lived in the surrounding harness: the tools the agent calls, the information placed in its context, and the schedule on which actions execute.

Experimental setup

The robot cell has two YAM arms and four cameras: an overhead view, a front view, and one view from each wrist. The agent is GPT-6 Astra running in Codex at the ultrafast service tier with medium reasoning effort. It receives a task description and a Python robot API. No scripted skills or demonstrations are supplied.

Two YAM robot arms at the back of a wooden table with a blue towel laid out in front of them
Figure 2, left. The two YAM arms, with a towel laid out for folding.
Four camera views tiled into one image: overhead, front, left wrist, and right wrist
Figure 2, right. The four camera views tiled into one image, in the form the agent receives them.

Like PyRUA-Lean [6], the project uses code as the action interface. The robot is a Python object, and the agent acts by writing short programs against it. The tools are deliberately designed as a white box. The robot API is ordinary Python source that the agent can read, and in practice it begins most tasks by reading that source and then writing helper functions, such as one that lifts a particular cup.

A separate robot server checks every command against a workspace fence and a minimum distance between the arms. That separation lets the agent execute code it wrote itself without a person reviewing each command first. Safety checks that require no judgment happen on the server: reach, fence boundaries, and arm separation are validated as commands arrive, and a bad command is rejected before the arm moves.

Architecture of the Codex sandbox, robot API and persistent Python kernel connected to an external robot server, cameras, and robot arms
Figure 3. The harness. Codex and everything the agent runs sit in a sandbox with no internet access. The agent reads the robot API source, writes its own helpers, and runs them in a long-lived Python process. The robot server validates every command, while results, images, and action reports return through the harness.

The sandbox has no internet access. Its only external interfaces are the model API, the robot server, and the cameras. The agent's tools include exec_command, write_stdin, and view_image; it has no web search or browser.

Where the time goes

The early towel-folding runs on the real robot took between 13 and 37 minutes. Figure 4 profiles one of the longest. Model work accounted for 78% of wall time, and tool calls accounted for nearly all of the remainder. The robot moved for only a quarter of the run, and much of that motion happened inside blocking calls while the agent was waiting.

Stacked timeline and breakdown showing model time, tool time, and robot motion in an early towel-folding run
Figure 4. Where the time went in an early towel-folding run. The summary hold is the API retaining each response until its reasoning summary was written; summaries were disabled for later runs. TTFT is estimated as model time minus decoding and includes hidden reasoning tokens.

Very little of the model's time went to producing output. About two thirds was TTFT. Measured from the harness, it includes prefill, network time, and any other server-side processing; through the API alone, the authors could not determine how the delay splits among those stages. A response with fewer than 100 output tokens still had a median TTFT of about two seconds, and the agent paid that cost on every tool-call round trip.

Many requests did very little. The agent would spend an entire round trip viewing an image, checking on a program it had already started, or asking where an arm had ended up. On average it made several requests per robot action, and the robot remained idle through all of them.

Agent workloads are input-heavy, especially robot-use agents that read images frequently: the context grows with every step, while each output is only a short tool call. TTFT, paid on every request, therefore outweighs decoding. The path to lower latency was to make fewer requests, make each request cheaper, and move the remaining requests so they overlap with robot motion rather than preceding it.

1. Faster decoding

In short timed requests, Astra's ultrafast tier generated tokens about four times faster than the Fast priority tier while keeping the same time to first output. In the towel runs, median model response time fell from 2.9 seconds to 1.9 seconds, roughly one third shorter. These were different episodes, so the comparison is approximate, but it matches the profile: decoding was the smaller part of each response, and a faster decoder leaves TTFT unchanged.

2. Fewer round trips

Because every request carries a fixed cost, the team looked for requests that were predictable. A request is predictable when the agent almost always makes it around a certain kind of action. Such requests can be merged into the request that writes the action itself. Two patterns stood out (Figure 5).

Before an action, the agent often reviewed its plan. It drew a ghost of the planned grasp into the camera image, looked at that image, and then ran the action in a second request. After an action, it spent several more round trips inspecting the result: it captured the cameras, read each image in a separate request, and asked for the robot's state. Neither process requires a new decision.

Before-and-after diagrams showing how separate review, camera, and state requests were merged into one action report
Figure 5. One action and the requests around it, shown illustratively. Before, the agent wrote the action, requested a visual review, then ran the action in a second request; after motion, it captured the cameras and read each image separately. After the redesign, the action starts as soon as it is written, the robot server checks commands while they run, and the result returns with the arm state, camera views, and the output of the agent's own checks.

The review now comes back with the action. A program that renders images returns them in its own tool result during the same step, so the model sees them without a separate request. Every action ends by printing the robot's state, including where each arm is, whether the grippers hold an object, and any faults, then attaching a camera view. The agent can add its own output through hooks: functions registered once that run at the end of every motion and add their printed and drawn results to the same response.

In the crumpled-towel run, one hook located the towel in the front camera and listed a dozen points on it. When an action ends, all of this information returns together as one message, called the action report (Figure 6).

Two smaller changes remove follow-up requests. Error messages now identify which limit was hit and by how much, so the agent can usually correct the problem on its next attempt. Checking whether a collection of candidate poses is reachable also takes one call for the whole list instead of one call per pose.

Code runs as soon as it is written. In the observed reviews, visual inspection changed the agent's plan in only about 4 of 99 cases, yet each review had cost a full round trip per action. The agent now starts an action in the same request that writes it and inspects the plan while the arm is already moving. If something is wrong, it can stop the action and hold the arms. Safety checks that require no judgment run on the robot server, which validates reach, fence boundaries, and arm separation as each command is sent. Section 5 extends this idea with pipelining: the agent writes the next actions while one is running.

3. Keep the context short

A robot agent reads many images. As those images accumulate, TTFT grows on every request even when earlier context is cached. The harness addresses this in two ways.

First, it sends the four camera views as one tiled image instead of four separate images. At the same effective resolution, the single tile produces noticeably lower TTFT (Figure 8 in the original article). Second, once about 20 images have accumulated, the harness moves the conversation to a fresh thread. Older images are replaced by short text notes, while only the newest few images remain. The first request after pruning cannot reuse the prompt cache, so pruning is performed between turns.

The result is not a substitute for a small context. It is a way to preserve the useful conversational record while preventing old observations from taxing every future request.

4. Faster tools

Once most of the extra requests were gone, time spent inside tools became visible at 6% to 10% of the run. Most of that time occurred between a visual inspection and the next action, where no other work could proceed.

The largest improvement came from running the agent's code in a persistent Python process. Imports are paid once, a call starts in tens of milliseconds instead of a few hundred, and variables and helper functions survive between calls (Figure 9). Camera capture was also made faster.

Three successive Python calls sharing helper functions and imported modules in one persistent kernel
Figure 9. Three calls from the crumpled-towel run, all sent to the same Python process, with code and output shortened. The second call uses the helper far defined by the first, and the third uses floor from the second. None of them imports robot, np, or viz. A new process for every call would spend hundreds of milliseconds importing and would lose every name when it exited.

5. Pipelining: hiding model time behind motion

Even after these changes, the agent still makes a few requests per action. As long as the next action cannot start until the model writes it, the robot must wait through every request. The final change lets the model think while the robot moves.

The agent writes each action as a Python function and passes it to the kernel. The robot executes actions in order in the background while the model plans. The model can change the plan at any point: it can replace everything after a specified action or stop the action that is currently running.

kernel.queue([lift_left, lift_right])             # run in order, in the background
kernel.queue(new_plan, replace_from="lift_right") # replace lift_right and what follows
kernel.action().stop()                            # stop now; the arms hold

The timing of feedback matters (Figures 10 and 11). The report for one action arrives as the next action starts. The model can therefore inspect how the previous action went while the robot is already busy, and it still has time to revise the action after that. If the model finishes a turn while the robot is still moving, the harness sends a fresh camera view so the model can prepare the next step instead of sitting idle.

Every action also starts from wherever the previous one ended. The agent therefore writes actions that verify their own preconditions, for example checking that a cup is still in the gripper before carrying it elsewhere.

Timeline showing action reports arriving while the next robot action runs, overlapping model thinking with motion
Figure 10. How pipelining overlaps thinking with motion. Each report arrives while the next action is already running, hiding model request time as long as the model writes a successor before the queue runs dry.
Detailed 36-second timeline of model planning, queued actions, action reports, and motion during the crumpled-towel run
Figure 11. Thirty-six seconds of the crumpled-towel run, with code and messages shortened. While smooth_first runs, the model adds the next stroke and inspects future plans. The report for smooth_first arrives as smooth_far starts. The model adds another stroke 2.8 seconds later, so it starts as soon as smooth_far ends.

The original article compares the same cup task in three harnesses on one time axis: the vanilla loop, sections 1 through 4 without pipelining, and the full harness with pipelining. Model time barely changes from the middle case to the final case; most of it simply happens while the arm moves. The first two timelines are replays of the real run, reconstructed under the measured request and tool costs.

Uni-robot-agent [2] uses a similar idea. Its tool calls can run in the background, so the model continues reasoning while the robot executes, and in simulation this substantially reduced episode times. The FastRUA version differs mainly in how results return. The agent does not ask for them. Each action report arrives as a message when the next action starts, while there is still time to change the action after that.

Results

The first comparison uses real runs. Every towel-folding run that either finished the task or continued for at least several minutes was grouped by harness version: the original; an intermediate version with fewer round trips, shorter context, and faster tools but no pipelining; and the full harness. Run time varies substantially because it depends on the towel's initial configuration and the choices the agent makes, and many runs were stopped before completion. The analysis therefore also reports two measures that do not require a finished episode: the fraction of time the robot moved, and the fraction of model time that overlapped motion.

Cutting round trips raised the robot's moving share from a median of 29% to 52%, because it spent less time waiting between actions. It hid none of the model's time, since every action still blocked. Pipelining hid the time: the median share of model time overlapping motion rose from almost none to about half. Runs that completed took a median of roughly 6 minutes with the full harness, compared with 16 to 18 minutes for the earlier versions.

Real runs vary too much to isolate each design choice, so the authors also took one real cup-task run made with the full harness and replayed it while removing one design at a time. The replay preserves the same model responses, tool calls, and action order, while adding back the requests removed by each design at the step and tool costs measured on the robot. Only the last bar in the original Figure 14 is a real run.

Together, the changes reduce this cup task from 667 seconds to 204 seconds, a 3.3× speedup. Before pipelining, almost all of the gain comes from reducing the number of model requests. Pipelining does not reduce model time at all, but it moves 137 of the 182 seconds of model work into periods when the robot is moving. From the first motion to the end of the task, the robot's moving share rises from 24% to 84%.

The following table lists the three runs from the demonstration videos, all made with the full harness. The final column is the median delay from the latest observation available to the model, either an action report or a camera view, to the moment it submits its next action.

Task, real run Time to finish Actions Robot moving* Model time hidden Observation to action
Three cups hide a pear, a ball, and nothing; put the pear under the empty cup 204 s 14 84% 75% 5.7 s
Spread a partly folded towel and fold it in half 143 s 11 67% 49% 4.7 s
The same task, starting from a crumpled towel 484 s 32 78% 74% 6.0 s

* From the first motion to the end of the task. Times and videos exclude the agent's initial reading of the robot API source.

JustRUA [5] reports two additional quantities for each simulated episode: what the episode consumed, and what the agent was doing during it. The authors measured the same quantities for the three real runs. Unlike the table above, these totals cover the entire episode, including the agent's initial reading of the robot API.

Run, whole episode Requests Input tokens Cached Output tokens Median context*
Cups 60 5.8 M 95% 14.0 k 80 k
Partly folded towel 44 7.4 M 96% 10.0 k 156 k
Crumpled towel 110 26.3 M 98% 36.7 k 195 k

* Median input tokens per request.

Like the simulated episodes, these runs are overwhelmingly input-heavy. A run reads between 5.8 and 26.3 million input tokens but writes only 10,000 to 37,000 output tokens, a ratio of 400:1 to 700:1. Nearly all input was sent earlier and came from the cache, so the marginal price is low. Each request still carries a context of 80,000 to 195,000 tokens, which is why TTFT rather than decoding sets the pace.

The second table groups model time in the three runs by the purpose of each request and shows how much of that time the robot was idle.

Request purpose, three runs Requests Model time Share Robot idle
Queue or run a motion 57 252 s 32% 30%
Capture or render camera views 62 250 s 32% 36%
Look up 3D points or IK 18 90 s 11% 6%
Read files 30 53 s 7% 85%
Wait on a running command 15 34 s 4% 19%
Message only 11 25 s 3% 22%
Read the robot's state 2 6 s 1% 0%
Other 19 81 s 10% 36%
All 214 792 s 100% 32%

Motions and camera views each account for about one third of model time, and in both cases the robot was idle for about one third of that time. Geometry lookups are almost fully hidden because the agent performs them while an earlier action is still running. Reading files is the opposite: most of it happens before the first motion, when there is no action behind which to hide the delay.

Video 1. The cup task at 2.5×. Below the cameras are the robot lane, the agent lane, and the number of actions waiting. The agent must reveal the hidden pear, identify the empty cup, move the pear, and cover it again.
Video 2. Spreading a partly folded towel and folding it in half, at 2.5×.
Video 3. The same task starting from a crumpled towel, at 3×. The first fold comes out crinkled, so the agent lifts the cloth until it hangs taut and lays it back down.

Limitations

Hiding model time behind motion is not the same as making the agent fast. Even when pipelining hides nearly all of it, every action is based on an observation that is already about five seconds old when the action is submitted (Figure 15). The robot moves most of the time, but it reacts like a player in a game with very high ping.

Median delay from the latest observation to submitted action for the cup and towel runs
Figure 15. The latency that pipelining cannot hide: the delay from the latest observation received by the model to the action it submits in response. Each point is the median for one of the three demonstrated runs.

This is also why pipelining helps some tasks much more than others. Lifting three cups in turn can be planned from a single look, so several actions can wait in the queue and the robot rarely stops. Cloth needs a new look after almost every grasp. At most one action can wait behind the running action, and every few actions the robot must wait while the model looks and thinks.

Tasks that require fine, live correction are the worst case. In ring-stacking runs, where the agent has to align a ring over a peg, the robot moved only 28% to 36% of the time even with pipelining, because each small correction depended on seeing the result of the previous one. At one point in the crumpled-towel run, the robot waited while the agent inspected the towel before folding, then waited again while it replanned after a sweep went wrong.

Closing this gap requires a shorter loop from observation to action. That means lower TTFT and less reasoning on routine steps. It may also require splitting the work so that a faster, smaller model or a reactive controller handles corrections that cannot wait several seconds.

What comes next

Throughout this work, Codex and OpenAI's serving stack were treated as a black box. The harness could change only what it sent and when. It could not even observe how the delay before the first output splits among prefill, queueing, and other server-side work. Much of the remaining latency lies on that side of the interface.

With a self-hosted model, the harness and serving system can be designed together. The authors identify several opportunities: process the next camera view while the arm is still moving, before asking the model about it; remove old images from the cache without paying for a full prefill; and schedule a robot's requests so that a needed correction does not sit behind unrelated work in a queue. Co-designing model serving with the robot harness could reduce the remaining latency substantially.

Contributors

Shaoxiong Duan, Yang Yuanbo, Junda Chen, Quanquan Peng, and Hao Zhang. The work is from UC San Diego and Hao AI Lab.

References

  1. An Unexpected Robot Policy: Early Evaluations of GPT-6 Astra on RoboDojo. arXiv:2609.24170.
  2. Democratizing Robot-Use Agents: A Unified Harness with Open Models (uni-robot-agent). arXiv:2611.01560.
  3. Anthropic, Claude Fable and Mythos 5.2. anthropic.com.
  4. RoboDojo. arXiv:2607.04434; evaluation loop in github.com/robodojo-benchmark/RoboDojo, src/eval_client/eval_env.py.
  5. JustRUA: A Controlled Study of GPT-6 Robot-Use Agents. arXiv:2611.16009.
  6. Fewer Tokens, Better Action: GPT-6 Astra Robot Agents with 14% Higher Success Rate but 65% Fewer Tokens (PyRUA-Lean). arXiv:2610.01939.

Citation

@misc{realtimerua2026,
  title        = {Getting Robot-Use Agents to Run in Real Time},
  author       = {Duan, Shaoxiong and Yang, Yuanbo and Chen, Junda and Peng, Quanquan and Zhang, Hao},
  year         = {2026},
  month        = nov,
  howpublished = {FastRUA Blog},
  url          = {https://fastrua.ai/blog/2026-11-realtime-rua/}
}

Source: “Getting Robot-Use Agents to Run in Real Time” by Shaoxiong Duan, Yang Yuanbo, Junda Chen, Quanquan Peng, and Hao Zhang, FastRUA, November 2026. Figures and videos are reproduced from the original article.

Related Articles

As an Amazon Associate, we earn from qualifying purchases.