Show-Harness: Just a VLM Agent Can Play Robots

Paper Metadata

• Title: Show-Harness: Just a VLM Agent Can Play Robots
• Authors: Yanzhe Chen, Zechen Bai, Zhijun Cao, Wenzheng Zeng, Kevin Qinghong Lin, Yiqi Lin, Guoqiang Liang, Kevin Yuchen Ma, Qiming Huang, Mike Zheng Shou (Show Lab, National University of Singapore; the first four authors contributed equally)
• Links: arxiv.org/abs/2609.10522 | project page showlab.github.io/Show-Harness
• Code status: released. The public repository github.com/showlab/Show-Harness contains the harness itself, the GUMI collector, every plugin, and the training pipeline; six LoRA adapters are published on HuggingFace as showlab/Show-Harness-VLMs and the demonstration corpus as showlab/Show-Harness-Data

One-Line Takeaway

Rewrite robot control as a small set of discrete semantic action units that a vision-language model already knows how to read, then let an embodiment-specific deterministic interpreter turn each symbol into a bounded Cartesian motion. Closed-source frontier models can drive a robot zero-shot, and a 2B open model becomes a usable policy after a few GPU-hours of LoRA tuning.

Background and Motivation

Foundation vision-language models already encode most of what robot manipulation needs: recognizing objects, judging spatial relations, decomposing a long-horizon goal into steps. The hard part is converting that knowledge into robot behavior. Vision-language-action models take one route, pulling the VLM down toward low-level control by fine-tuning it to regress a continuous, embodiment-specific action. The price is that broad semantic knowledge gets compressed into an opaque pixel-to-torque mapping, so a new task family, a new environment, or a new arm usually means another round of adaptation.

Hierarchical and programmatic systems go the other way. The VLM stays on top and emits subtask-level calls, or writes programs over a hand-designed control API. Semantics survive, but physics is delegated to a downstream controller and a system-specific grounding pipeline. The model says what should happen without seeing how it happens, and the direct link between semantic intent and physical execution is weakened by the intermediary.

The authors' diagnosis is that the missing piece is not a stronger model but a better interface. That interface has to satisfy two requirements that pull in opposite directions: it must be semantically sufficient for a VLM to reason over directly, and it must be fine-grained enough for physical control that every model decision actually lands on end-effector motion. Show-Harness is built around exactly this interface, an embodied harness in which the model is accountable for physical decisions step by step inside a perception-reasoning-action loop, rather than outsourcing those decisions to an opaque executor.

The idea has precedent in the digital world. Computer-use agents and game agents act through compact, human-shared interfaces such as the mouse, the keyboard, and the gamepad, and discrete direction primitives are the native language of navigation simulators and benchmarks. Real manipulation has never had a comparable interface, because control is embodiment-specific, high-dimensional, and demands millimeter-level spatial precision. This is the gap Show-Harness fills: grounding fine-grained semantic actions deterministically into real robot motion while keeping the same action space open to both VLM agents and people.

Once the interface is human-shareable, the shape of data collection changes too. The authors extend it into GUMI, a web GUI for manipulation in which a human uses the keyboard, a computer-use agent drives the browser, and a VLM agent predicts action units directly. All three collect demonstrations in the same semantic space with no dedicated teleoperation hardware. Interface design therefore stops being an implementation detail and becomes the through-line across control, adaptation, and data.

Preliminaries

Embodied harnesses and agentic systems. A harness is the runtime wrapped around a foundation model: it organizes observations, maintains memory, translates model decisions into robot behavior, and feeds execution results back, closing the perception-decision-execution loop. Existing systems differ mainly in how a decision gets executed: composing API programs, selecting skills from a library, issuing language subgoals to a VLA, or handing off to a symbolic planner and a numerical optimizer. The Show-Harness position is that a harness should not make physical decisions on the model's behalf; it should expose the how, transparently and deterministically, to the model.

The action-representation spectrum. At the low end sit continuous action chunks (regression, diffusion, flow matching), discrete motor tokens, task-space keypoints, and latent action codes learned from trajectories. At the high end sit subgoals, keypoints, affordance targets, and spatial constraints. Semantic action units land in between: they are discrete symbols, but each symbol corresponds to one small, bounded, observable physical change, so they keep semantic readability while retaining the granularity that closed-loop control requires.

Cartesian poses and incremental motion. The interpreter maintains a 6-DoF pose setpoint $s_{t}=(\mathbf{x}_{t},Q_{t})$, where $\mathbf{x}_{t}\in\mathbb{R}^{3}$ is position and $Q_{t}\in\mathrm{SO}(3)$ is orientation. Rotation increments are written with the exponential map $\exp(\theta[\cdot]_{\times})$, where $[\cdot]_{\times}$ is the skew-symmetric matrix operator. This notation lets "rotate 15 degrees about an axis" become one clean left-multiplication, and it composes naturally with workspace projection and safety limits.

Method

The closed loop. Given a language instruction $\ell$, at step $t$ the system collects an observation $o_{t}=(\mathcal{I}_{t},p_{t})$, where $\mathcal{I}_{t}$ is the set of camera views and $p_{t}$ is proprioceptive state. The current observation plus a compact interaction history $h_{t}$ first pass through a configurable set of reasoning plugins $\mathcal{P}$, producing a reasoning-refined context

$$c_{t}=\Phi_{\mathcal{P}}\!\left(\ell,\,o_{t},\,h_{t}\right) \tag{1}$$

The central VLM then selects a semantic action over that context

$$a_{t}=\pi(c_{t})\in\mathcal{A} \tag{2}$$

where $\mathcal{A}$ is the entire semantic interface. An embodiment-specific, model-agnostic interpreter grounds that semantic decision into executable control deterministically

$$u_{t}=g_{E}(a_{t};\,s_{t}) \tag{3}$$

Here $E$ is the embodiment and $s_{t}$ is the interpreter's internal setpoint state. Executing $u_{t}$ updates the robot and the environment, and the new observation and execution state flow into the next step. Along the whole chain the model only ever faces symbols and physics is only ever handled by deterministic code; there is no learned black box between them.

The semantic action space. The shared space $\mathcal{A}$ consists of a small number of semantic action units. At each step the system first fixes a reference view from the current observation, which defines the direction of end-effector motion. Relative to that view, MV_FWD and MV_BACK, MV_LEFT and MV_RIGHT, MV_UP and MV_DOWN move the end effector one step along the corresponding direction; ROTATE_CW and ROTATE_CCW each take a specified axis (x, y, or z) and rotate the end effector incrementally about it when reorientation is needed; GRASP and RELEASE close and open the gripper; DONE declares the task finished. The vocabulary is designed around three properties. It is incremental: each unit produces only a small local physical change, so the model stays in the loop and can converge on precise behavior through successive corrections, and the experiments confirm that VLMs switch between step granularities without fine-tuning. It is interpretable and embodiment-agnostic: actions are semantic symbols rather than numeric targets, with all embodiment-specific low-level control pushed down into the interpreter. And it is visually groundable: motion directions are defined relative to an observable view, so spatial reasoning maps directly onto actions.

The deterministic interpreter. Once a semantic decision $a_{t}$ enters the interpreter $g_{E}$, the pose setpoint is updated as

$$s_{t+1}=\Pi_{E}\!\left(\mathbf{x}_{t}+\sigma_{t}R_{E}d_{a},\;\exp\!\left(\theta_{t}[R_{E}r_{a}]_{\times}\right)Q_{t}\right) \tag{4}$$

$d_{a}$ and $r_{a}$ encode translation and rotation respectively: $d_{a}=0$ for a rotation, $r_{a}=0$ for a translation, and otherwise $r_{a}$ is one of the six signed unit axis vectors. The calibrated increments $\sigma_{t}$ and $\theta_{t}$ set translation and rotation magnitude (2 cm for a fine step and 4 cm for a coarse step by default, 15 degrees per rotation unit), $R_{E}$ maps semantic directions and axes into the motion frame of embodiment $E$, and the projection $\Pi_{E}$ enforces embodiment-specific workspace and per-step limits. Gripper units bypass the pose update and map directly to open or close. Different embodiments implement the same semantic units with different low-level controllers: impedance control tracks the Cartesian setpoint on the Franka, inverse kinematics with streamed joint targets drives AgileX, and simulation environments execute the corresponding operational-space commands. Safety boundaries (workspace limits, a table-height floor) are configured inside the interpreter, so an out-of-bounds action is rejected before execution. Porting to a new robot therefore means writing one new interpreter, with no change at all to the model-side interface.

Perception-stage plugins. Multi-View Guidance tells the model what role each camera plays and how to prioritize them: a global exocentric or egocentric view supplies scene-level context, while wrist views provide fine-grained alignment and close-range evidence for manipulation. Proprioception translates the robot's internal state into concise textual feedback: current gripper height, the displacement caused by one action step, staged execution hints (for example, descend first while the position is still high), and contact plus gripper open-close state.

Reasoning-stage plugins. Subtask Planning first has the VLM act as a planner and decompose the instruction $\ell$ into ordered subtasks with completion criteria; during execution the model owns advancement, checking the criterion against the current image at every step, continuing to act while it is unmet, and declaring completion and moving on once it is met. Situated Planning handles conditional branches: the initial plan leaves uncertain decisions open, and a branch is resolved and the remaining plan updated only when the evidence becomes observable during execution, which avoids replanning at every step. Action Chunking lowers the model call rate when fine-grained feedback is not yet necessary because the goal is still far, letting the VLM emit a short sequence of semantic actions to be executed open-loop. Adaptive Step varies step size with goal visibility: a 2 cm fine step when the target is visible in the wrist view, otherwise a 4 cm coarse step. Visual Prompt spends one dedicated model call to turn a vague language goal into a visual reference that highlights the relevant affordance, which subsequent reasoning then grounds on; it is enabled only when the interaction region is hard to describe in words.

Action-stage plugins. Action History carries the most recent actions (five by default) into the next step as lightweight context, together with simple usage guidance such as avoiding oscillation between opposite actions, giving the loop temporal memory. Failure Recovery detects grasp failures automatically: an empty grasp is recognized, the gripper state is reset, and the corresponding grasp subtask is rolled back and retried, so the model does not carry on transporting an empty gripper.

flowchart TD
  L["Language instruction task"] --> P["Perception
Multi-View Guidance
Proprioception as text"]
  P --> R["Reasoning plugins
Subtask and Situated Planning
Action Chunking / Adaptive Step
Visual Prompt"]
  R --> C["Reasoning-refined context c_t"]
  C --> V["Central VLM decision
a_t = pi(c_t)"]
  V --> A["Semantic action units
MV_* / ROTATE_*
GRASP / RELEASE / DONE"]
  A --> I["Embodiment interpreter g_E
incremental pose setpoint update
workspace projection and safety limits"]
  I --> X["Robot execution
Franka impedance control
AgileX inverse kinematics
sim operational-space commands"]
  X --> M["Action-stage plugins
Action History
Failure Recovery empty-grasp rollback"]
  M --> P
  A --> G["GUMI web interface
human keyboard / computer-use agent
demos in the same semantic space"]

Mode 1: zero-shot frontier VLM agent. The same harness lets a closed-source frontier VLM control the robot with no fine-tuning at all. The default backend is Gemini-3.1 Pro with a thinking budget of low, medium, or high at inference time, medium by default. The significance of this path is scalability: each generation of improvement in the foundation model is inherited by physical control, with no robot policy to retrain.

Mode 2: lightweight fine-tuning of a small model. The same interface also supports training a small open VLM directly into a policy. Given demonstrations $\mathcal{D}$ collected in the shared action space, the policy minimizes token-level cross-entropy over the target unit

$$\mathcal{L}(\theta)=-\!\!\sum_{(\ell,\,o,\,h,\,a)\,\in\,\mathcal{D}}\!\!\log\pi_{\theta}\!\left(a\mid\Phi_{\mathcal{P}_{\mathrm{min}}}(\ell,\,o,\,h)\right) \tag{5}$$

$\mathcal{P}_{\mathrm{min}}$ deliberately keeps only the minimal decision context (instruction, multi-view observation, short action history) so comparisons stay controlled. The key point is that semantic actions are predicted through the VLM's native vocabulary: no action head and no special tokens. A rank-64 LoRA is therefore attached only to the language-model linear layers, with the vision encoder and multimodal projector frozen, updating roughly 3% of parameters. Training on 7.9K single-arm samples for 40 epochs (learning rate 1e-4, cosine schedule, 0.1 warmup, bf16, 256x256 views, effective batch 32) takes under two hours on a single H200 and is affordable on a 24GB card. The default backbone is Qwen3.5-2B.

GUMI: a collection interface shared by humans and machines. Because the semantic action space is discrete and directly manipulable, it exposes naturally as a lightweight web GUI in which every unit corresponds to a labeled control and a key binding. A human plays the robot with a keyboard, a computer-use agent operates the same GUI, and a general VLM agent predicts units directly. At each step GUMI records the pre-execution observation and the chosen semantic action, yielding $(o_{t},a_{t})$ pairs a policy can consume as-is; since every semantic unit is deterministically grounded by the interpreter, a rollout also retains the corresponding low-level commands and trajectory, so one demonstration can train either a semantic-action policy or a continuous-control policy. GUMI supports step-by-step control, queued action chunks, single and dual arms, and mixed human-agent collection where a person can intervene at any time to correct an agent rollout. Unlike teleoperation pipelines that depend on dedicated hardware or simulator-specific widgets, it records demonstrations directly in the shared semantic space, so the same batch transfers across embodiments that implement the same units, the same workflow works in simulation and on real hardware, and remote collection comes for free.

Experiments

Setup. Two real platforms: a 7-DoF Franka Research 3 with one exocentric RealSense D435 facing the worktable and one wrist-mounted RealSense D405; and a bimanual AgileX with two 6-DoF arms and three Orbbec Dabai DC1 cameras (one shared egocentric view plus one per wrist). Local models run on a single RTX 5090, frontier models are called over API. The benchmark is ten real-robot manipulation tasks, five objects crossed with two containers (a plate and a bowl), spanning rigid geometry (a block), irregular shape (a banana), rolling dynamics (a tennis ball), deformability (a teddy bear), and precision manipulation (a chess piece). The fine-tuned model only ever sees block, banana, and tennis ball, so teddy and chess are held out for out-of-distribution evaluation. Each task runs ten trials with randomized object placement, a 50-step episode cap, and a timeout counted as failure.

Figure 1: Show-Harness releases the generalization of foundation VLMs across tasks, scenes, and embodiments; the strip below compares per-step token cost and inference frequency across models.

Main result: three axes of generalization. Across tasks, the zero-shot mode (ZS, Gemini-3.1 Pro with medium thinking) averages 89.0% success over the ten tasks and the fine-tuned mode (FT, Qwen3.5-2B) 86.0%, while the strongest baseline RATS reaches only 57.0%, with H-VLA at 50.0%, CaP-X at 44.0%, pi0.5 at 39.0%, GR00T at 35.0%, and G-VLA at 13.0%. The gap is starkest on the held-out teddy and chess tasks: for Chess to Plate, ZS scores 10/10 and FT 9/10, against 1/10 for pi0.5 and 0/10 for GR00T. Across environments (20 trials each for background, lighting, viewpoint, and distractor perturbations), ZS is 20/20 on all four for a 100.0% average, FT reaches 88.0%, and RATS 65.0% with H-VLA at 63.8%. The simulation-to-real row is reported only for trainable methods: FT reaches 13/20 on a real Franka using only demonstrations GUMI collected in simulation, where pi0.5 and GR00T score 0/20 on the same data. Across embodiments, ZS obtains 48/50 on the Franka and 45/50 on AgileX (93.0% average) purely by swapping the interpreter, while FT is co-trained on both embodiments' demonstrations and evaluated directly at 87.0%.

Cross-task setting (10 trials/task)	pi0.5	GR00T	H-VLA	G-VLA	CaP-X	RATS	ZS	FT
Block to Plate	6/10	5/10	7/10	2/10	5/10	6/10	10/10	10/10
Banana to Plate	7/10	7/10	8/10	3/10	8/10	8/10	10/10	10/10
Tennis to Plate	3/10	3/10	3/10	1/10	4/10	6/10	8/10	8/10
Teddy (held out) to Plate	5/10	4/10	5/10	1/10	4/10	6/10	10/10	8/10
Chess (held out) to Plate	1/10	0/10	3/10	0/10	2/10	4/10	10/10	9/10
Block to Bowl	5/10	4/10	6/10	2/10	5/10	5/10	10/10	10/10
Banana to Bowl	6/10	6/10	7/10	2/10	7/10	8/10	6/10	7/10
Tennis to Bowl	2/10	2/10	3/10	1/10	3/10	5/10	7/10	8/10
Teddy (held out) to Bowl	3/10	4/10	5/10	1/10	4/10	5/10	8/10	7/10
Chess (held out) to Bowl	1/10	0/10	3/10	0/10	2/10	4/10	10/10	9/10
Average success (%)	39.0	35.0	50.0	13.0	44.0	57.0	89.0	86.0

Cross-environment and cross-embodiment settings	pi0.5	GR00T	H-VLA	G-VLA	CaP-X	RATS	ZS	FT
Background perturbation (20 trials)	11/20	10/20	15/20	4/20	12/20	14/20	20/20	18/20
Lighting perturbation (20 trials)	9/20	9/20	14/20	5/20	11/20	13/20	20/20	19/20
Viewpoint perturbation (20 trials)	10/20	7/20	10/20	2/20	10/20	12/20	20/20	19/20
Distractors (20 trials)	10/20	8/20	12/20	1/20	9/20	13/20	20/20	19/20
Simulation to real (20 trials)	0/20	0/20	--	--	--	--	--	13/20
Cross-environment average (%)	40.0	34.0	63.8	15.0	52.5	65.0	100.0	88.0
Franka 7-DoF (50 trials)	22/50	19/50	26/50	7/50	23/50	30/50	48/50	45/50
AgileX 6-DoF (50 trials)	19/50	17/50	23/50	4/50	20/50	22/50	45/50	42/50
Cross-embodiment average (%)	41.0	36.0	49.0	11.0	43.0	52.0	93.0	87.0

Figure 2: The Show-Harness architecture. A modular perception-reasoning-action loop connects a foundation VLM to robot control through the shared semantic action interface.

Physical adaptability: precision, composition, extrapolation, workspace. Block stacking and peg-in-hole insertion need finer motion than standard pick-and-place. The authors change only the interpreter step from 2 cm to 1 cm, leaving interface and model untouched: ZS rises from 60% to 82% and FT from 40% to 65%, while pi0.5 on the same demonstrations sits at 18% and needs an extra round of fine-grained training to reach 62%. That is the dividend of separating semantic decisions from metric execution, since a new precision requirement touches only the interpreter. Action composition behaves similarly: on the five Plate tasks, composing two orthogonal translation units into one diagonal displacement cuts step count substantially with almost no loss in success rate, and a new composition can be introduced without retraining. Rotation extrapolation is even more telling. In a carrot-grasping task the carrot is placed at 0, 45, and an unseen 90 degrees relative to the gripper, with 15 degrees per rotation unit: ZS is robust at all angles, FT reaches 70% at the unseen 90 degrees despite seeing only 0 and 45 degree demonstrations, and pi0.5 manages 20%. Incremental rotation covers unseen angles by recomposing units, whereas continuous action regression is bound to the angular range of its demonstrations. For workspace shifts, expanding from the central 25% region (S@1) to near-boundary 90% (S@3) degrades Show-Harness only mildly while pi0.5 drops sharply, indicating that visually grounded semantic decisions depend less on the training distribution. In bimanual coordination, on two 20-trial tasks (tidy the table and hand over a banana), a policy that predicts both arms jointly clearly beats two independent single-arm agents and eliminates collisions.

Figure 3: Capability analysis. The left column covers physical adaptability (fine-grained control, rotation extrapolation, action composition, workspace shift, bimanual coordination); the right column covers semantic adaptability (reasoning-intensive tasks, in-context video learning).

Semantic adaptability: reasoning and in-context learning. On two tasks that require reasoning before manipulation, finding a block hidden under one of three upside-down cups and arranging scattered letters into SHOW, ZS with Situated Planning reaches 85%, while FT and pi0.5 alone manage only 10% and 0%. Giving FT the same Gemini-generated subtask instructions lifts it to 70%, and pi0.5 stays at 5%. The semantic interface lets a new instruction be grounded through a reusable action space, preserving the instruction-following elasticity of the VLM. For in-context video learning, where the agent must tidy three objects in the order shown by a demonstration, ZS reaches only 20% without a video because the order is unspecified, and 20/20 once a human or robot demonstration video is supplied; FT attains 95% conditioned on a task outline extracted by the same planner.

Plugin ablations. Leave-one-out ablations run on the real Franka with the zero-shot agent over the five Plate tasks (default configuration: 96% success, 30 steps per episode on average). Removing Subtask Planning drops success to 60%, with the typical failure being dragging the object toward the plate without lifting it. Turning Action Chunking off still yields 96% but with more model calls, while forcing it on throughout reduces calls and drops success to 74%, so selective chunking is the right policy. Adaptive Step with fine steps only is precise but times out, and coarse steps only are fast but overshoot; adaptive switching achieves both 96% and 30 steps. Removing Action History lowers success and raises timeouts, with the typical failure being visible oscillation between opposite actions. Removing Failure Recovery drops success to 72%, hitting the hardest-to-grasp objects most, and the dominant failure mode is an undetected empty grasp. Multi-View Guidance improves substantially over a global view alone, with the largest gains on precise alignment of small objects, and removing Proprioception degrades performance noticeably on objects with ambiguous visual cues. Two plugins that are off by default stand out in their target scenarios: Visual Prompt lifts handle-aware grasping from 40% to 85%, with the gain coming from explicitly aligning the language instruction to the marked interaction point rather than from the marker itself, and Situated Planning lifts hidden-object search from 35% to 85%.

Plugin (real Franka, zero-shot agent, five Plate tasks)	Intervention	Result	Reading
Default configuration	--	96%, 30 steps/episode on average	Adaptive Step: 2 cm fine, 4 cm coarse
Subtask Planning	removed	60%	model often drags without lifting; no explicit stage decomposition
Action Chunking	off / forced always on	96% but more calls / 74%	selective chunking is optimal
Adaptive Step	fine only / coarse only	precise but times out / fast but overshoots	switching on wrist visibility balances the two
Action History	removed	lower success, more timeouts	oscillation between opposite actions becomes a visible loop
Failure Recovery	removed	72%	empty grasps go undetected; carries an empty gripper onward
Visual Prompt	added (handle grasping)	40% to 85%	aligns the instruction with the marked interaction point
Situated Planning	added (hidden-object search)	35% to 85%	defers open branches until visual evidence appears

Why the interface itself works: an action-representation ablation. A 2x2 design varies only how the six translation units are represented, leaving the rest of the harness fixed: A semantic action names plus written conventions (the default), B semantic names only, C arbitrary symbols plus conventions, D arbitrary symbols only. D provides no action prior at all, so the agent can only probe unknown symbols, infer effects from before-and-after observations, and record a mapping. C essentially matches the default and B is usable but less efficient, which says the conventions carry most of the grounding and semantic names mainly act as a useful prior. D succeeds once in 20 episodes and infers only 23.3% of mappings correctly: inferring physical action effects from visual change alone is too ambiguous, and explicit conventions are necessary. On frontier models and thinking budgets, zero-shot performance rises broadly with model capability; increasing the thinking budget mainly removes redundant interaction steps with limited gains in success rate, and can cost more wall-clock time (up to 3.4x for GPT-5.6-sol); across all models more than 98% of responses produce a valid action unit. On backbone scaling, 2B is already a good balance between accuracy and response speed: larger backbones help mainly on fine-grained tasks such as stacking and peg-in-hole, while 1B-class models over-adjust locally near the goal and lengthen episodes, yet on the tennis ball task the smaller model is actually better because correcting a moving object in time matters more than fine precision.

Figure 4: The GUMI interface. Humans and frontier agents collect demonstrations through the same semantic controls, recording the pre-execution observation and the chosen action unit at every step.

The demonstration corpus. All demonstrations are collected through GUMI with a uniform 2 cm translation step. Real robot: 164 episodes and 7774 steps, of which Franka contributes 9 tasks and 101 episodes (4969 steps, 49.2 per episode) and AgileX 10 tasks and 63 episodes (2805 steps, 44.5 per episode). Simulation: 230 episodes and 13523 steps, with ManiSkill at 100 episodes (5840 steps) and RoboLab at 12 tasks and 130 episodes (7683 steps). A further 19 Franka episodes (594 steps) are recorded specifically for grasp recovery: the gripper starts offset forward, backward, or sideways from the target, and after an empty grasp it lifts and re-approaches instead of transporting nothing; the recordings are trimmed to the corrective segment, which is why they average about 31 steps versus roughly 53 for freely collected episodes.

Platform	Tasks	Episodes	Episodes/task	Steps	Steps/episode
Franka (7-DoF, real)	9	101	11.2	4969	49.2
AgileX (6-DoF, real)	10	63	6.3	2805	44.5
Real total	19	164	8.6	7774	47.4
ManiSkill (simulation)	1	100	100.0	5840	58.4
RoboLab (simulation)	12	130	10.8	7683	59.1
Simulation total	13	230	17.7	13523	58.8

Figure 5: Qualitative demonstrations. New objects, background and lighting changes, cluttered scenes, spatial reasoning (arranging letter blocks into SHOW), and bimanual drawer opening with object placement all reuse the same interface without redesign.

Limitations

Stated by the authors. Evaluation so far concentrates on single-arm and bimanual manipulation with parallel-jaw grippers; extending to more complex embodiments such as humanoids and dexterous hands is a worthwhile direction. The perception side also does not yet include embodied modalities such as touch and force feedback, so contact-rich interactions that need force control remain out of scope.

Our reading. First, closed-loop granularity is paid for in wall-clock time: with a 50-step cap and roughly 30 steps per episode, a single pick-and-place costs tens of model calls, and even with Action Chunking the throughput is hard to match against a VLA that emits an action chunk in one pass; the zero-shot mode adds closed-API latency and cost on top. Second, a training-free interface is not a free adaptation: the step size sigma, the rotation increment theta, the motion-frame mapping R_E, and the workspace limits all have to be calibrated and configured by hand per embodiment, so interpreter quality directly sets the ceiling. Third, the simulation-to-real row of the main table is filled only for FT (13/20), with agentic and code-as-policy baselines left blank in that cell, making this comparison weaker than the other rows; and with ten trials per task the success-rate confidence intervals are wide. Individual results also cut against the headline, such as ZS scoring 6/10 on Banana to Bowl, below several baselines, which suggests the semantic interface is not always ahead on every placement combination.

Conclusion and Outlook

The contribution of Show-Harness is not a new model but a new interface: discrete action units that are semantically transparent to a VLM and physically grounded by a deterministic interpreter, wrapped in a plugin harness that organizes perception, reasoning, and action into one loop. It validates both operating modes at once, a closed-source frontier model controlling a robot zero-shot and a 2B open model becoming a usable policy with about 3% of parameters updated over a few GPU-hours, and it beats representative VLA and agentic baselines on generalization across tasks, environments, and embodiments as well as on simulation-to-real transfer. GUMI then extends the same interface to data collection, letting humans, computer-use agents, and VLM agents share one keyboard-operable semantic space. Looking forward, three questions decide whether this moves from playable to practical: whether the interface can carry the higher-dimensional action spaces of dexterous hands and humanoids, whether it can absorb tactile and force modalities, and whether the throughput gap between semantic steps and continuous action chunks can be closed by smarter chunking.

Golden Quote

The right interface can unlock substantial embodied capability already present in foundation VLMs, without requiring additional model capacity or costly embodiment-specific pretraining.