Skip to content
RobotWorld
← Tags

#Franka Research 3 (2)

WaterfallPagination
Show-Harness: Just a VLM Agent Can Play Robots

Show-Harness: Just a VLM Agent Can Play Robots

Foundation vision-language models (VLMs) exhibit broad intelligence about the world, yet translating this intelligence into robot control remains challenging. We present Show-Harness, an Embodied Harness that enables VLMs to "play" robots through a compact semantic interface linking intent to action. Show-Harness exposes discrete semantic action units that VLMs can naturally reason over, while embodiment-specific interpreters deterministically ground them into local robot actions, keeping the VLM directly responsible for fine-grained physical decisions. Through the same interface, Show-Harness demonstrates the feasibility of (1) directly unlocking closed-source frontier VLMs for zero-shot robot control, and (2) adapting small-scale open-source VLMs for low-cost deployment with just a few GPU-hours of fine-tuning. We further develop GUMI (GUI Manipulation Interface), which extends the same semantic action space to GUI-based demonstration collection, allowing humans and agents to "play" robots across embodiments without specialized teleoperation hardware. Extensive experiments show that Show-Harness-equipped VLM agents generalize robustly across tasks, embodiments, and environments, outperforming representative agentic and VLA paradigms. These results suggest that the right interface can unlock substantial embodied capability from foundation VLMs, without requiring additional model capacity or costly embodiment-specific pretraining.

Yanzhe Chen, Zechen Bai, Zhijun CaoSep 9, 2026
Vision-language modelsAgentic HarnessSemantic Action InterfaceSep 9, 2026
Memory as Plans: World-Action Modeling with Memory-Grounded Planning

Memory as Plans: World-Action Modeling with Memory-Grounded Planning

Most robot policies predict actions from the current observation or a short fixed window, yet long-horizon manipulation is non-Markovian: the evidence a decision needs may already be out of view. MaP-WAM treats memory as planning-time evidence instead of executor input. Completed segments are stored as structured multimodal records pairing a language instruction with sparse visual context (8 uniformly sampled frames per segment); a fine-tuned Qwen3.5-4B language planner proposes the next segment-level language plan, and a causal world model initialized from WAN-2.2-5B turns the long-term visual context into a matching visual plan. The two form a memory-grounded plan that conditions a World-Action-Progress executor, a Mixture-of-Transformers extension of a video DiT that jointly predicts action chunks, future visual latents, and execution progress, with progress as a first-class modality. Because the executor only sees a fixed-length plan prefix, its context stays constant as history grows, and block-causal attention makes both planning and execution KV-cacheable; plan-observation alignment calibrates recursively predicted progress against visual-plan frames, and a progress gate (threshold 0.95) triggers segment transitions that write resampled real observations back into memory. On RMBench MaP-WAM reaches 83.3% average success over nine memory-dependent tasks (best baseline LingBot-VA 77.1%), and 78.0% on two real Franka Research 3 tasks (88% Find Button, 68% Press Buttons), while a full-context executor runs out of memory at 1700 history frames and WAP holds an approximately constant 827 ms per action chunk. Limitations: segment structure is taken from benchmark annotations rather than discovered automatically, and plan-observation alignment uses a lightweight training-free pixel-difference metric.

Sizhe Zhao, Haozhe Xie, Weiyu ZhaoSep 10, 2026
World-action modelsLong-Horizon MemoryMemory-Grounded PlanningSep 10, 2026