PAPER DEEP DIVE
RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments
RobotWorld introduces a cross-embodiment robot-use benchmark spanning 84 tasks and evaluates whether multimodal agents can convert perception and computation into verifier-confirmed physical execution.
Paper information: Zhiqin Yang, Chenxin Li, Xiaomeng Hu, Yibin Liu, Weidong Huang, and 28 additional authors. arXiv:2610.10409. Official repository: robotworldai/robotworld. Repository snapshot inspected at commit 9e39068f49b5486fa39993982da2591a27a85f78.
One-sentence summary: RobotWorld evaluates five multimodal agents on 84 executable tasks across five robot domains and finds that useful perception and control routines exist, but they do not yet compose into reliable end-to-end robot use.
The central measurement is not whether an agent can discuss a robot. It is whether the agent can preserve the task state, repair ineffective actions, and move the physical world to a verifier-approved outcome within a finite interaction budget.
1. Why "robot use" needs its own evaluation
Digital agents have learned to inspect files, call tools, write programs, and revise a plan after observing an error. Those behaviors invite a simple extrapolation: attach a robot interface to the same agent and its digital competence should become physical competence. RobotWorld is designed to test that extrapolation rather than assume it. The paper defines robot use as the ability to turn an instruction and observations into action through a robot interface, respond to execution feedback, and satisfy an independently checked physical outcome.
This definition places the unit of evaluation above a single action prediction. A policy can estimate a grasp, an image-processing program can recover a useful coordinate, and a language model can explain why an insertion failed. None of those local successes proves that the complete task will succeed. The agent must also decide when the estimate is adequate, keep the relevant object relation intact while moving, recognize a rejected command, and stop only when the task is actually complete.
Existing embodied benchmarks often center on one robot, one scene family, or one class of manipulation. That is appropriate for comparing policies within a controlled embodiment, but it leaves a broader question open: does the same general agent remain capable when the body, camera configuration, action semantics, controller assistance, and temporal structure change? RobotWorld turns this into an empirical question by connecting 84 tasks from 20 source projects under a common interaction shell while retaining the native physics and robot interfaces.
The paper also separates task success from the evidence produced during execution. A score tells us whether the final state passed. A trace can show whether the agent approached the right object, relied on direct visual estimation or programmatic calibration, recovered after a planner rejection, exhausted the budget while navigating, or declared completion after changing an irrelevant part of the robot. For a benchmark intended to guide agent design, this distinction matters because two failures with the same binary outcome can demand completely different remedies.
RobotWorld is therefore not just a harder leaderboard. It is an attempt to characterize the distance between "the model has the ingredients for robot control" and "the model reliably performs robot work." The paper's strongest result is that this distance remains large: the best overall model solves 19.0% of the suite, and the union of all five evaluated models still leaves 63 of 84 tasks unsolved.
2. The benchmark model: a task is a contract
The paper represents task $i$ as a structured contract rather than a sentence paired with a simulator:
Here, $\mathcal{E}_{i}$ is the environment, $\rho_{i}$ specifies the initial-state distribution, and $g_{i}$ is the goal. $O_{i}$ and $A_{i}$ define what the agent can observe and command, while $C_{i}$ records controller assistance such as motion planning, inverse kinematics, native balance control, or low-level servoing. $B_{i}$ bounds interaction, and $V_{i}$ evaluates the executed trajectory. This tuple makes explicit a fact that a generic "robot tool" label can hide: two tasks may look equally simple at the API level while presenting very different control problems.
At interaction round $k$, the agent selects a response from its policy using the goal, interface documentation, and permitted history:
The response may be a robot request, a workspace computation, an image inspection, or text. The model parameters $\theta$ stay fixed throughout an episode. What changes is the evidence available to the next decision: returned observations, execution errors, notes, and files retained in the analysis workspace. This is an agentic formulation rather than a conventional single-shot policy. The model can construct a temporary solution around the task without updating its weights.
Robot execution is measured in native control steps. If request $k$ completes $d_{k}$ steps, cumulative progress is:
Offline computation, image analysis, and text-only decisions set $d_{k}=0$. A control step may itself contain multiple physics substeps, so control steps, tool calls, and wall-clock minutes are not interchangeable units. The paper deliberately reports all three. A model may spend little simulated control time but many minutes waiting for inference, or issue many zero-step analysis requests without advancing the physical episode.
Success is defined by $V_{i}$, not by the agent's completion message. Checkers can depend on object pose, support, joint opening, contact, release, velocity, event order, path history, or the duration of a condition. They can accept different physical strategies if those strategies establish the same required relation. At the same time, the checker never supplies private state or a reference solution to the agent. The interface can expose a task-specific robot state, but evaluator-only cameras and internal goal state remain outside the agent channel.
The task contract also gives the benchmark a disciplined way to describe variation. A changed goal or physical constraint can create a separate task, while simple aliases or alternative control modes do not inflate the count. This rule prevents the suite from appearing broader merely because one behavior has several names. It also means that the reported 84 tasks are intended to represent distinct goals or constraints, not merely distinct calls to the same simulator.
3. Runtime design: a common shell over heterogeneous robots
RobotWorld gives each agent a similar interaction surface: a task instruction, interface documentation, camera observations, permitted robot state, an execution tool, and a separate analysis workspace. A turn proceeds from evidence to optional computation to a bounded robot request, after which the environment returns a new observation, the number of completed steps, and any applicable errors. The framework is shown in Figure 3.
Figure 1: Paper Figure 3. The common protocol connects multimodal analysis and bounded robot execution while preserving each source environment's physics, action semantics, and native controller assistance.
The workspace is not a route to simulator internals. It supports image reading, geometry, numerical calculation, and the development of control logic using observations and files the task permits. It cannot read private evaluator state or advance physics. This separation lets the benchmark test whether an agent can make useful computations while keeping the physical loop auditable. It also cleanly distinguishes auxiliary code from code control inside the robot loop.
Direct action is always available. A call contains action values or targets and a requested number of steps. The environment validates the request, runs it through the native controller, and returns feedback. That feedback can distinguish a rejected call from a call that executed but failed to achieve its intended effect. Optional code control, where supported, runs an agent-generated feedback program for up to a fixed number of control steps. The program can read new permitted state at each step without another model call. The five-model study disables environment-side code control, so the main ranking primarily tests agents organizing direct robot requests. The agents can still analyze images and write offline code.
The environment interaction loop is intentionally split into three paths. The agent sees observations, proposes computation or action, and receives feedback. Robot execution advances the native controller and physics. A separate evaluator watches task state and events without becoming part of the action interface. This separation is essential for both measurement and security: success cannot be inferred from an attractive final camera frame if the required temporal or contact conditions were absent.
Figure 2: Paper Figure 4. Physics pauses during reasoning and offline computation, advances during bounded execution, and is checked independently for task success.
Observation design is task-specific. A manipulation interface may expose joint state and object geometry; a driving task may expose speed and route information; a legged controller may expose body velocity and projected gravity. The common benchmark does not force every robot into an RGB-only setting. What is shared is the agent-facing protocol and the requirement that the task be completed through the declared robot interface.
The observation history is also controlled. In the evaluated setting, the agent receives the current observation and up to four earlier observations sampled every two observation rounds. Text-only continuation preserves the latest packet without falsely implying that the scene has changed. This gives the model memory for correction and subgoal tracking while preventing an unlimited transcript of every physical tick from replacing state estimation.
Interaction limits include a task-specific control horizon and a separate non-action budget. For 83 tasks, the default limits are 15 consecutive action-free events and a total of 30, 60, or 120 such events. Volleyball 1v1 uses 20 consecutive events and 150 total. Completed robot execution resets the consecutive counter, while API retries do not consume the budget. The design admits necessary analysis but prevents an agent from replacing physical work with an unbounded chain of comments or failed calls.
Controller assistance remains part of the task. RoboDojo can use motion generation such as cuRobo; other robots rely on low-level servo control or learned balance policies. The benchmark does not erase these differences to manufacture a false equality. It records them and warns against treating the aggregate score as a pure measure of one identical low-level control ability across every body. This choice is scientifically honest, but it becomes one of the paper's main limitations.
The resulting pipeline can be summarized as follows:
flowchart TB A["Task contract
goal, interfaces, budget"] --> B["Observation packet
images and robot state"] B --> C["Multimodal agent"] C -->|analysis| D["Isolated workspace
vision, geometry, computation"] D --> C C -->|direct request| E["Native controller
bounded action segment"] E --> F["Physics and state update"] F --> G["Feedback
new observation, steps, errors"] G --> C F --> H["Independent checker
states, events, time"] H --> I["Outcome and trace record"] E --> H
4. The task suite: breadth with explicit composition
The 84 tasks come from 20 source projects and are grouped into five primary domains: 38 manipulation, 20 mobile manipulation, 11 locomotion, 11 driving, and four aerial tasks. Manipulation and mobile manipulation together account for 69.0% of the suite. The aggregate score therefore reflects a manipulation-heavy sample, and the paper consistently asks readers to inspect the domain columns rather than infer universal robot competence from the total.
Figure 3: Paper Figure 6. Each inner segment represents one of the five primary domains; each outer segment represents a primary task type. Tasks are counted once, and percentages use all 84 tasks.
Manipulation includes insertion, stacking, pouring, drawer operation, object rearrangement, and interactions with articulated or deformable objects. Mobile manipulation draws on RoboCasa and BEHAVIOR-1K and exposes interfaces that combine a mobile base and an arm. A task can remain in this domain even when the successful action is mostly arm motion, because the contract tests the combined body's interface rather than whether both subsystems moved the same distance. Locomotion covers legged motion, whole-body recovery, balance, carrying, and ball interaction. Driving emphasizes routes, gates, drifting, and parking. Aerial tasks test load stabilization, inverted-pendulum tracking, and repeated ball contact.
The benchmark also labels overlapping control requirements. Spatial goals require a correct position, orientation, destination, or path. Constrained contact covers insertion, hanging, pouring, and manipulation of joints where unconstrained pick-and-place is insufficient. Multiple subgoals require preserving progress across stages or objects. Balance and tracking require sustained regulation of posture or a moving reference. Timed interaction requires coordination with an independently moving target or gate. One task can belong to several of these groups, so the labels are not alternative domain partitions.
Figure 4: Paper Figure 2. RobotWorld combines a shared agent protocol with native simulators, diverse embodiments, trajectories, and task-level analysis.
The paper distinguishes inherited task objectives from benchmark-added interfaces and success conditions. When a source environment already has a suitable executable goal, RobotWorld can retain it. When the source is organized around a continuing reward, survival or reward alone may not prove that the instruction was completed. Selected locomotion and aerial tasks therefore receive explicit objectives, such as reaching a destination after a disturbance or stabilizing a payload within a set of bounds. Reaching the horizon is not automatically a success.
The suite remains finite and uneven. Several domains have small samples, and many tasks derive from a limited number of source projects. The authors acknowledge that public artifacts may also overlap with model training data. These are not incidental bookkeeping issues. A difference of one task in the four-task aerial domain changes the domain rate by 25 percentage points, while the same one-task difference moves the overall benchmark by only 1.2 points. Domain percentages and aggregate percentages therefore answer different questions.
5. Evaluation protocol and the scoring equation
The study evaluates GPT-6 Astra, Claude Opus 5.5, Kimi K3, DeepSeek V4.1 Flash, and Gemini 3.8 Flash. Every model attempts all 84 tasks. One episode is retained for each model-task pair, creating 420 scored episodes. The models share the task initialization, control-step horizon, success rules, history convention, and disabled environment-side code control used for this study.
Tasks are weighted equally rather than by trajectory length, reward magnitude, or action count. For a task set $\mathcal{G}$ and model $m$, the reported success rate is:
In this expression, $\xi_{i,m}$ is the model's executed trajectory on task $i$, and $S_{i}$ is the task verifier's binary result. Partial credit is not assigned, and native dense reward is not averaged into the main result. The formula is simple, but it makes the benchmark's inferential limit visible: one retained outcome per pair estimates a snapshot, not the distribution of outcomes under repeated trials.
The paper uses the outcome at the applicable interaction boundary. Some recorded videos continued past the scoring boundary, so successful-looking states after that cutoff cannot be retroactively substituted for the scored result. Twenty-one records required adjudication. Nine lacked a Boolean success value and remained failures; twelve retained the historical Boolean value, with three historical successes changed to failures because they occurred outside the earlier budget boundary. This procedure reinforces a basic rule: a success that arrives after the budget does not count.
Resource logging includes wall-clock elapsed time, robot requests, shell calls, explicit image views, and control steps. Images embedded in robot feedback are not counted again as explicit image views. The statistics include failed runs, so a short episode can mean efficient success or rapid failure. Totals may also extend beyond the retrospectively adjudicated scoring cutoff. The authors therefore present resource use as an observational profile rather than a direct efficiency score.
The protocol does not standardize all information available to every agent. Interface documentation differs by environment, and the observation channels differ by task. The shared instruction says what is permitted, but the model still has to read and use task-specific interfaces. This is closer to realistic robot use than a benchmark with a single universal action vector, but it also means that provider harness differences and interface understanding are part of the measured system.
6. Main result: 19.0% is the ceiling, not a plateau
Astra records the highest overall result with 16/84 successes, or 19.0%. Opus 5.5 follows with 13/84, or 15.5%. Kimi K3 solves two tasks, while DeepSeek V4.1 Flash and Gemini 3.8 Flash solve one each. The leader therefore still fails 68 of 84 tasks. The result is not a narrow ranking among models that have mostly saturated robot use; it is a broad completion gap across every evaluated system.
Figure 5: Paper Figure 1. Overall success and domain counts for the five evaluated agents. The first two domains are pooled in the figure, while the paper's detailed table reports them separately.
| Model | Manipulation | Mobile manipulation | Locomotion | Driving | Aerial | Overall |
|---|---|---|---|---|---|---|
| Astra | 9/38 | 4/20 | 0/11 | 2/11 | 1/4 | 16/84 (19.0%) |
| Opus 5.5 | 7/38 | 0/20 | 1/11 | 2/11 | 3/4 | 13/84 (15.5%) |
| Kimi K3 | 0/38 | 1/20 | 0/11 | 1/11 | 0/4 | 2/84 (2.4%) |
| DeepSeek V4.1 Flash | 0/38 | 0/20 | 0/11 | 0/11 | 1/4 | 1/84 (1.2%) |
| Gemini 3.8 Flash | 1/38 | 0/20 | 0/11 | 0/11 | 0/4 | 1/84 (1.2%) |
The successful sets are not interchangeable. Astra and Opus 5.5 share eight successes. Astra has eight additional unique successes, and Opus has five. The five-model union covers 21 tasks, or 25.0% of the suite, leaving 63 tasks unsolved by any model. All successes from Kimi, DeepSeek, and Gemini lie inside the Astra-Opus union, so the lower-ranked models do not increase total coverage in this evaluation. That union is a descriptive ceiling over available models, not a deployable router that knows which model to call.
Separating mobile manipulation changes the interpretation of the top-two gap. Astra solves 9/38 manipulation tasks and 4/20 mobile manipulation tasks. Opus solves 7/38 and 0/20 respectively. Across the two manipulation domains, Astra reaches 13/58 while Opus reaches 7/58. Astra's four mobile-manipulation wins are therefore a major source of its three-task overall lead. The aggregate label "manipulation" in a compressed chart would obscure that distinction.
The direction reverses in several dynamic and timing-sensitive groups. Astra leads spatial goals 13/45 to 6/45 and constrained contact 6/41 to 1/41. Opus leads balance and tracking 4/20 to 1/20 and timed interaction 4/7 to 2/7. Opus also solves 3/4 aerial tasks, while Astra solves 1/4. Opus records the only locomotion success in the five-model study, the quadruped push-recovery task. A single overall score would compress these complementary profiles into a ranking that is easy to quote and easy to misinterpret.
| Overlapping requirement | Astra | Opus 5.5 | Interpretation |
|---|---|---|---|
| Spatial goals | 13/45 | 6/45 | Astra is stronger on positions, orientations, paths, and placement relations. |
| Constrained contact | 6/41 | 1/41 | The difference appears in insertion, hanging, pouring, and articulated-object tasks. |
| Balance and tracking | 1/20 | 4/20 | Opus more often sustains posture or a changing reference. |
| Timed interaction | 2/7 | 4/7 | Moving targets and timing constraints favor Opus in this sample. |
Driving and locomotion remain especially sparse. Astra and Opus each solve courtyard and hairpin driving for 2/11; Kimi solves courtyard driving for 1/11. The remaining nine driving tasks are unsolved by every model. Ten of eleven locomotion tasks are also unsolved. The problem is not that every dynamic-control task fails uniformly. It is that success on one route or one recovery does not transfer into broad competence across the domain.
7. Cost, time, and tool use do not produce a simple scaling story
The resource data reject any easy claim that more reasoning, more calls, or more elapsed time reliably creates better robot use. Astra records the top score with an estimated list-price cost of USD 9,912.80, 941M tokens, and 54.6 cumulative hours. Opus records a lower score of 15.5% with lower estimated cost of USD 2,916.43, but more tokens at 1.63B and more cumulative time at 140.3 hours. Token counts and dollar estimates are not common currencies across providers, so neither should be treated as a controlled measure of reasoning effort.
| Model | Tokens | Estimated cost | Cumulative elapsed | Median episode | Median robot requests |
|---|---|---|---|---|---|
| Astra | 941M | USD 9,912.80 | 54.6 h | 30.9 min | 58 |
| Opus 5.5 | 1.63B | USD 2,916.43 | 140.3 h | 54.0 min | 37 |
| Kimi K3 | 2.00B | USD 624.09 | 163.8 h | 67.6 min | 42.5 |
| DeepSeek V4.1 Flash | 907M | USD 34.49 | 35.7 h | 16.0 min | 22 |
| Gemini 3.8 Flash | 303M | USD 66.68 | 28.7 h | 12.8 min | 11 |
Kimi is the clearest warning against treating consumption as capability. It uses the most reported tokens, 2.00B, and the most cumulative elapsed time, 163.8 hours, but solves only two tasks. Astra has the highest estimated cost and the most successful outcomes. Opus uses more than a billion additional tokens relative to Astra and more than twice the cumulative elapsed time, yet trails by three solved tasks. Provider pricing, caching, harness overhead, task length, and early failure all enter these totals, so the table describes operating cost under the recorded systems rather than a causal law of agent performance.
Median behavior reveals different control styles. Astra takes a median 30.9 minutes and 58 robot requests per task. Opus takes 54.0 minutes and 37 robot requests. Kimi takes 67.6 minutes and 42.5 requests. In the full retained logs, Opus makes 2,095 shell calls and 568 explicit image views, compared with Astra's 1,435 shell calls and 56 explicit image views. Opus more often externalizes geometry or computation into code. Astra more often turns visual evidence into a robot request without an explicit image-view call.
Conditioning on shared successes narrows the comparison. Among the eight tasks solved by both Astra and Opus, Astra uses fewer tool calls in seven and fewer control steps in six. The ordered block-stacking task reaches a comparable first-placement milestone with 23 calls and 270 control steps for Astra versus 46 calls and 268 control steps for Opus. Fruit placement uses 22 calls and 284 steps for Astra versus 58 calls and 419 steps for Opus. Cube placement reverses the step ordering: Astra uses fewer calls but spends more physical steps after an initially slipped grasp.
| Shared successful task | Astra calls / steps | Opus calls / steps | Observed tradeoff |
|---|---|---|---|
| First block placed and retracted | 23 / 270 | 46 / 268 | Similar physical progress with much different analysis cost. |
| Cube placed left of bowl | 24 / 286 | 31 / 239 | Fewer calls did not imply fewer recovery steps. |
| Three fruits arranged | 22 / 284 | 58 / 419 | Opus adds inspection and adjustment at both call and execution levels. |
Tool call count and physical execution are therefore different costs. One request can cover a long action segment, while another can execute one step or be rejected. Offline computation does not advance physics but can alter the next several requests. A useful efficiency evaluation must report outcome, calls, control steps, information acquisition, and failure timing together. RobotWorld does not collapse those dimensions into a single composite efficiency score.
8. What the traces reveal about success and failure
The successful traces show that general agents can assemble sophisticated perception and control routines. They fit camera models, segment objects by color, calculate joint relations, build approximate dynamics, allocate rotor thrust, and write controllers. The interesting question is not whether these routines exist. It is where they become connected to task progress and where the connection breaks.
Astra's ordered-stacking episode illustrates late calibration. It approaches the blue block from successive camera observations, closes the gripper at step 157, lifts at step 179, and only introduces camera-projection fitting at step 204 after moving the block toward red. Opus writes color-segmentation code before moving and uses sampled arm motions to fit camera geometry and compute targets. Both reach the first placement-and-retraction milestone at almost the same physical step, 270 versus 268, but their information-gathering orders differ sharply. Astra completes the stack at step 635; Opus reaches the auxiliary-call budget at step 313 without completion.
Feedback can also trigger a real reconfiguration rather than a verbal apology. In charger insertion, Astra tries to rotate an idle wrist camera downward at step 196, and the planner rejects the command. The agent lowers and repositions the idle arm by step 207, retries the rotation at step 230, and eventually reaches insertion and return success at step 323. The trace does not prove that lowering the arm was the unique cause of success, but it does show the requested action being changed into an executable configuration before the same motion is retried.
Temporal tasks expose a different requirement: a good final frame is not enough. Opus stabilizes a suspended payload after combining offline linear-quadratic control with repeated state feedback. Payload error, payload speed, suspension swing, and drone tilt remain jointly inside their bounds for 2.15 seconds, exceeding the two-second requirement. Astra's inverted-pendulum episode ends with a 2.45 cm final tip error, but the final stable interval lasts only 0.565 seconds. The trajectory also violates the 20 cm tracking boundary before recovering below the terminal 10 cm bound. The final state looks close, while the required interval does not hold.
Drone juggling distinguishes one-off contact from sustained contact coordination. Astra and Opus both perform offline dynamics computation and both make three valid contacts near step 143. Astra terminates at step 188 because the ball becomes too low, ending with three valid and one height-qualified contact. Opus continues to 800 steps with 15 valid and 14 height-qualified contacts. Both use a median of three executed steps per action, so the difference is not simply that Opus issues shorter command segments. The stronger trace adjusts thrust, attitude, and position around repeated impact states.
Failure traces expose several different bottlenecks. In Kimi's pouring episode, the first approach knocks the bottle onto its side. The agent notices that its closure missed, repeatedly tries the left arm, switches to the right arm at step 240, and reaches the 400-step limit while still repositioning. The agent changes arm and pose, but it does not restore the object state needed for pouring. In the chopping episode, all 69 positive-step calls are spent surveying, navigating, or clearing furniture. The task never reaches a cutting attempt, so the trace diagnoses workspace access rather than the unaudited cutting skill.
Other failures come from completion judgment. Kimi calls a block-sweeping task complete at step 861, then issues 69 calls that only change the empty left arm's height over 139 more control steps. The arm moves, but the task-directed action does not resume. DeepSeek's stove-navigation episode concludes that further motion is unnecessary, then sends zero base velocity for the final 105 steps while the native task remains unfinished. Tool use continues; corrective search does not.
Opus provides a useful contrast in infeasibility reasoning. After dropping a mug during a hanging attempt, it returns home, makes another approach with 83 steps remaining, and later judges that the final 35 steps are insufficient to grasp and hang the mug. It spends those steps returning the arm home. This is not a false completion claim. The trace cannot prove recovery was impossible, but it distinguishes an explicit budget judgment from asserting success.
Recovery timing is another failure axis. Astra's humanoid table-tennis episode ends at 3.58 seconds when the base falls below 0.50 m despite balance corrections. The agent can recognize local state and issue a response without producing a recovery that arrives before the constraint is irreversibly broken. RobotWorld's value here is diagnostic: the same binary failure can now be attributed to object-state loss, workspace access, temporal coordination, completion judgment, or late recovery.
9. Repository evidence: executable checks without one-click reproduction
The official repository is public, but its existence should not be confused with full reproduction of the paper's five-model comparison. The README states that the release registers 84 tasks across 20 integrations. Registration does not guarantee that every task has a verified successful trajectory. The default release protocol runs three rollouts per task, while the paper retains one scored episode per model-task pair. Environment-side code control is normally disabled, physics pauses during inference, and the L40S 48GB machine is described as reference hardware rather than a compatibility certification. The release also states that no top-level license had been selected at the inspected snapshot, leaving third-party code and assets under their own terms.
The executable checker implementation supports the paper's strongest methodological claim. In environment/evaluation/world_success/profiles.py, task profiles declare horizons and temporal predicates. A quadruped push-recovery profile requires the final two seconds to satisfy a 0.25 m goal-error bound, a 0.10 m/s horizontal-speed bound, and a 20 degree tilt bound. A drone payload-hover profile requires payload error, payload speed, suspension swing, and drone tilt to hold together for the final two seconds.
add('go2_push', 'quadruped_push_recovery', 1000, 50,
hold('S01', 2, *target(.25, .10, 20)))
add('omnidrones', 'drone_payload_hover', 500, 62,
hold('S01', 2,
('payload_error', '<=', .15),
('payload_speed', '<=', .20),
('swing', '<=', radians(10)),
('drone_tilt', '<=', radians(15))))The tireless success monitor in environment/evaluation/world_success/monitor.py enforces contiguous control ticks, finite measurements, and a fail-closed interpretation of holds. A broken hold resets its start time; a violation in a fixed interval is not erased by a later recovery; and a native failure overrides any final success. The implementation acknowledges that the checker observes sampled states rather than a mathematically continuous physical trajectory.
if type(step) is not int or step != self.last_step + 1:
raise ValueError(f'Expected contiguous tick {self.last_step + 1}, got {step}')
if rule['operator'] == 'hold':
row['samples'] += 1
if ok:
if row['since'] is None:
row['since'] = t
else:
row['since'] = None
if row['first_violation'] is None:
row['first_violation'] = t
row['passed'] = ok and t - row['since'] >= rule['seconds'] - 1e-10The repository also makes low-level interface semantics inspectable. The Go2 push profile uses 12 joint-position residual inputs. The native target is the default posture plus a quarter-radian scaled command:
This equation is not a learned policy. It defines the action contract between the agent and the native proportional-derivative controller. The four-channel OmniDrones interface maps a normalized input $u$ into an internal throttle target:
Thrust then depends on $s^{2}$, and motor lag remains active. A value of zero is half maximum steady-state thrust, not hover. Such details matter for interpreting failures: the same abstract "apply action" call can hide very different physical units, coordinate frames, actuators, and controller assistance. The paper's formula layer and repository mappings are therefore part of the benchmark's scientific content, not implementation trivia.
environment/evaluation/world_success/runtime.py keeps evaluator hooks outside agent adapters and treats privileged traces as private. environment/evaluation/runner.py routes tasks to the appropriate suite and native horizon. scripts/validate_world_success.py separates predicate tests from optional zero-action probes. These files support inspection of the evaluation machinery, but they do not reconstruct the model providers, reasoning parameters, harness timing, or the exact paper ranking. The correct status is "publicly inspectable infrastructure with an evaluation snapshot," not "the reported model results are available by one command."
10. Validity limits and alternative interpretations
The authors identify controller assistance as a major boundary. Some manipulation tasks provide motion generation or inverse kinematics, some mobile tasks retain native balance policies, and some vehicles or drones expose lower-level control than others. A shared protocol standardizes the interaction surface, not the physical difficulty of every action. Cross-domain success rates therefore combine model behavior with different controller contracts. RobotWorld reports this rather than pretending that every domain measures an identical low-level control skill.
The authors also note that physics pauses during inference and offline computation. Real robots must continue holding a posture, waiting safely, or tracking a moving object while the model thinks. Communication delays, sensor noise, contact-model error, actuator friction, and hardware-specific dynamics add further differences. A 2.15-second simulated hold is evidence that the agent can organize a controller, but it does not establish that the same loop meets real-time constraints on a physical drone.
Coverage is finite and uneven. Manipulation and mobile manipulation dominate the suite, several task families inherit the same source environment, and some embodiments have only a few representatives. Public tasks, code, videos, and traces may overlap with model training corpora. The study cannot rule out familiarity effects or estimate generalization to a fresh hidden scene family. Its claims should remain tied to the selected suite rather than expanding into a universal ordering of physical intelligence.
An independent validity concern is the single retained episode per model-task pair. The three-task difference between Astra and Opus is meaningful as a description of this snapshot, but it does not provide a confidence interval for repeated runs. A different initialization, one planner rejection, or one boundary decision could change a task outcome. The 63-task unsolved union is robust as a description of the evaluated records, but the precise ranking should not be treated as stable without paired repeated trials.
The comparison is also not a controlled single-variable model experiment. Providers differ in tokenization, pricing, inference stack, harness latency, and tool behavior. Kimi's token and cost aggregates extrapolate from 57 valid records, while the other models have 84; costs use list prices rather than bills. Opus uses more shell and explicit image-view calls, while Astra issues more robot requests per median episode. These correlations can suggest a strategy difference, but they cannot isolate model weights as the only cause of performance.
Finally, a verifier is only as complete as its encoded physics. A grasp may be considered stable even if it would damage a real object; a liquid event may approximate the result without modeling the material; a path can satisfy a simulated boundary while violating a hardware safety constraint. More executable checks improve auditability, but the choice of checks still shapes which policies succeed. Hidden initializations, adversarial perturbations, repeated trials, and physical-hardware transfer would test the conclusions more strongly.
11. What should improve next
The paper's failure analysis suggests four priorities for agent design. First, perception and computation should stay grounded in task state. A camera fit or visual estimate is useful only if the resulting action preserves the object and subgoal relation the task requires. Second, execution feedback should change the next task-directed action. Detecting a rejection or a missed grasp is not recovery until the agent changes the configuration or strategy that caused the failure.
Third, temporal constraints should be represented as conditions that persist, not as a final-frame target. A payload hover succeeds only when all bounds hold through the required interval; a tracking task can fail despite a favorable last sample. Fourth, completion judgment needs a reliable distinction between "I have finished," "the task has terminated," and "the remaining budget is insufficient to recover." The traces contain examples of all three states.
For benchmark design, the next step is not simply to add more tasks. Repeated paired episodes, hidden initializations, confidence intervals, and systematic reporting of controller assistance would make model differences more interpretable. Real-time and hardware studies should follow the simulation suite where representative tasks can expose latency and contact effects. A stronger benchmark would preserve RobotWorld's linking of outcome to trace while making reliability and generalization explicit variables rather than assumptions.
RobotWorld's contribution is to make the missing integration visible. The evaluated agents already contain many of the local operations that a robot-use system needs. They can inspect, calculate, code, and act. What they do not yet do consistently is maintain the chain that connects those operations to a complete physical outcome. That is a more useful target than another demonstration that a capable model can perform one impressive short-horizon behavior.
Golden quote: Reliable robot use is not the sum of clever local steps. It is the preservation of task state, the correction of action, and the persistence of physical evidence across the entire execution budget.
