Addressing the Orchestration Gap in Generalist Robots via Physical Agency

Paper: Addressing the Orchestration Gap in Generalist Robots via Physical Agency
Authors: Liane Galanti, Dhruv Shah, Tri Dao
Institution: Princeton University, Together AI
Links: arXiv:2607.21725 · Project Page · Code
Code Status: Open source

1. Background and Motivation

General-purpose robots need to reason about their actions—combining perception, world knowledge, planning, success detection, recovery, and low-level control. Imagine instructing your home robot: "a child is coming over—put the toys on the plate, and the unsafe items in the box." A capable visuomotor policy might fail: it must distinguish a toy from a hazard (world knowledge), find items hidden behind others (perception), recover when a grasp slips (closed-loop control), and stop only when the table is genuinely safe—not when it appears tidy. General-purpose manipulation is a full stack, and the hard part is rarely the motion itself.

The dominant route to generalization is scaling robot data. VLA policies (π0, π0.5, OpenVLA, GR00T, DROID) train on larger datasets to learn increasingly capable visuomotor skills. This data is essential: contact, grasping, and embodiment-specific control must be learned from interaction. But robot data is expensive and not always the most direct way to teach task-level capabilities such as negation, progress tracking, and recovery. When a policy fails on "put everything on the plate except the cup," the missing ingredient is often not the low-level motion but negation, world knowledge, decomposition, progress tracking, or recognizing that a grasp failed and must be retried.

Prior work addresses parts of this stack but rarely the whole loop. VLA scaling sharpens control and grounding, yet direct prompting still asks one network to perceive, reason, plan, verify, recover, and act in a single forward pass. Code-as-policies and task planners add structure but rely on symbolic APIs, hand-designed primitives, or privileged simulator state. Reasoning-VLAs push more deliberation into the policy itself, at the cost of additional training, and still leave success detection and recovery outside the loop. What is missing is not a better motor policy or a single reasoning module, but a process that closes the loop—deciding what to do, checking whether it worked, and repairing when it did not.

Pigey's core insight: decompose these capabilities at inference time rather than baking them into learned weights. A frontier VLM runs as a closed-loop agent on a fixed instruction: it reads the current observation and interaction history, emits one tool call, incorporates the result, and decides again—until it declares the task complete or a tool-call budget is reached. The agent supplies the task-level loop: it decomposes the instruction into short subgoals, maintains memory of what it has done, selects which frozen backend executes each subgoal, verifies the outcome from sensor and visual feedback, and recovers when verification fails. It never emits motor commands itself; all motion is delegated.

2. Preliminaries

Understanding Pigey requires two foundational concepts. First is the VLA policy formulation. Let $\mathcal{I}$ be the space of natural-language instructions, $\mathcal{O}$ the observation space, and $\mathcal{A}_{\mathrm{motor}}$ the continuous motor-action space. A VLA policy is a function:

$$\pi: \mathcal{O} \times \mathcal{S} \rightarrow \mathcal{A}_{\mathrm{motor}}$$

where $\mathcal{S} \subset \mathcal{I}$ is the subspace of short, concrete, visually grounded subgoals on which the policy is reliable. A user instruction $I \in \mathcal{I}$ is solved when the final world state satisfies a task-specific success predicate:

$$\mathbb{1}_I(\mathrm{scene}) = 1$$

The standard approach collapses interpretation, planning, and execution into a single VLA call $\pi(o, I)$. This works when $I$ is already VLA-legible but degrades when $I$ requires intermediate reasoning. Pigey instead factors interpretation, planning, and verification into an inference-time process operating over the same observation channel.

Second is the concept of the orchestration gap: the increase in success rate when the same frozen motor policy is invoked through the agent rather than prompted directly. This gap quantifies the benefit of inference-time orchestration—rather than new motor learning.

3. Methodology

3.1 System Overview

Pigey's agent $\phi$ runs as a closed-loop agent on a fixed instruction $I$: at each step it reads the current observation and interaction history, emits one tool call, incorporates the result, and decides again. Formally:

$$\phi: \mathcal{O} \times \mathcal{H} \times \mathcal{I} \rightarrow \mathcal{T}$$

The agent maps observation $o \in \mathcal{O}$, history $h \in \mathcal{H}$, and instruction $I \in \mathcal{I}$ to a tool call. Each frozen backend executes a short subgoal:

$$\pi_b: \mathcal{O} \times \mathcal{S} \rightarrow \mathcal{A}_{\mathrm{motor}}, \quad b \in \{\textsc{tamp}, \textsc{vla}\}$$

Backends receive only the subgoal $s$, never the full instruction $I$. The agent supplies the task-level loop: it decomposes $I$ into short subgoals, maintains memory of completed operations, selects frozen backends to execute each subgoal, verifies outcomes from sensor and visual feedback, and recovers on failure.

Figure 1: Example reasoning behaviors enabled by Pigey: obstacle reasoning, safety reasoning, clearing an occupied goal, long-horizon memory, and spatial reasoning. All panels are single, uncut real-robot rollouts; the orchestrator drives frozen skills with no new training.

3.2 Tools and Grounding

The agent acts through five tools: Perceive (return camera views, robot state, and detected object labels), Pick($\ell$) and DropAbove($\ell$) (grasp/place a labeled object via the TAMP backend), VLARollout($s$) (execute subgoal via the VLA backend), and Done (terminate).

After every call, the agent receives a wrist image, end-effector pose, gripper aperture, a binary is_grasped flag from the gripper-width sensor, and object labels from the open-vocabulary detector. These labels are the agent's vocabulary—every Pick/DropAbove argument must be one of them, forcing the agent to ground semantic categories from the instruction ("unsafe," "vegetarian," "the smallest") onto concrete detected objects rather than inventing names. Backends also return typed failures—no grasp found, motion-planning failure, unreachable, step-budget exhausted—so the agent learns why a step failed, not merely that it did.

3.3 Verification

Verification is what separates the agent from open-loop prompting, drawing on two complementary signals. The first is deterministic: a Pick counts as successful only if is_grasped is true at the gripper-width sensor, and motion-planning failures are surfaced as explicit flags. The second is visual: the post-action wrist image is returned to the VLM $\phi$, which confirms the intended object is in the jaws and gone from the table. The two are combined conservatively—if a backend reports success but the sensor reads an empty gripper, the step is overridden to a failure, so an optimistic backend cannot mislead the agent.

Only a verified outcome advances the plan; an unverified one triggers recovery. A place (DropAbove) is issued only after the preceding grasp is verified, so the agent never transports and releases an object it failed to grasp. This "verify-then-place" design converts TiPToP's open-loop pick-and-place into a closed loop—a direct source of Pigey's gains over it.

3.4 Planning and Recovery

Pigey picks a backend per subgoal, verifies, and escalates on failure. Key strategies include:

Path planning by object type: Rigid, table-resting targets go to TAMP; deformable or cable-like objects, and objects inside containers or stacked on another object, go straight to the VLA—geometric grasp planning is unreliable for these.

Verify, retry, escalate: An unverified Pick is retried once (re-perceiving and re-planning); a second failure escalates to the VLA. Escalation is bidirectional: if a VLA rollout makes no progress, the agent falls back to a TAMP Pick on a freshly perceived scene.

Recover: On a wrong-object grasp, the agent returns the object to the table (never the destination) and retries; when a target is hidden, it treats visible objects as occluders and uncovers it; when the destination holds items that don't belong in the goal state, it clears them first.

Table 1: Operational profile across modular manipulation systems. Pigey provides context memory, state tracking, retry, and verification—all without added training.

flowchart TB
    subgraph AGENT["Pigey: Frontier VLM Orchestrator"]
        OBS["Read observation o
+ history h"]
        OBS --> REASON["Reason: decompose I
into subgoal s"]
        REASON --> SELECT["Select backend:
TAMP or VLA?"]
        SELECT --> CALL["Emit tool call"]
        CALL --> VERIFY{"Verify outcome:
sensor + visual"}
        VERIFY -->|success| NEXT["Next subgoal
or Done?"]
        VERIFY -->|failure| RECOVER["Recover:
retry / escalate
/ clear / uncover"]
        RECOVER --> OBS
        NEXT -->|not done| OBS
        NEXT -->|done| SUCCESS["Task complete"]
    end
    subgraph BACKENDS["Frozen Motor Backends"]
        TAMP["TAMP: Pick/DropAbove
(rigid objects)"]
        VLA["VLA: VLARollout
(deformable / recovery)"]
    end
    CALL -->|Pick/DropAbove| TAMP
    CALL -->|VLARollout| VLA
    TAMP -->|views + flags| VERIFY
    VLA -->|views + flags| VERIFY
    style AGENT fill:#e8f0fe,stroke:#2563eb
    style BACKENDS fill:#fef3c7,stroke:#d97706

Above: Pigey's closed-loop orchestration flow. VLM reads observation, reasons to decompose subgoal, selects backend, emits tool call, verifies, and on success continues or completes, on failure recovers. Two frozen backends (TAMP/VLA) execute complementarily.

4. Experimental Analysis

4.1 LIBERO-PRO Simulation Benchmark

On LIBERO-PRO, Pigey achieves the highest mean success rate across six perturbation suites under a zero-shot protocol (no task-specific memory, reference-seed exploration, or fine-tuning), improving the frozen π0.5-LIBERO baseline from 12.8% to 53.3%—over 4×. The π0 baseline scores 0% across all suites, and CaP-Agent0 only 18.2%.

Notably, even though LIBERO-PRO requires minimal reasoning, Pigey's error recovery and provided tools can orchestrate the weak base policy to success, indicating the orchestration gap extends beyond high-reasoning tasks to scenarios requiring robustness and recovery.

4.2 Real Robot: The Orchestration Gap

Across 30 DROID real-robot tasks, Pigey lifts the frozen π0.5-DROID policy's overall success from 16.7% to 97.3%. TiPToP achieves 48.7%. Gains concentrate on reasoning-limited probes:

On reasoning-limited probes, the raw VLA averages only 4.6% while Pigey reaches 96.9%—using the same VLA. This indicates the primary bottleneck lies in instruction interpretation, decomposition, memory, and recovery, rather than low-level control. On simple pick-and-place tasks, the absolute increase is only 5 percentage points (95→100%), showing orchestration supplies missing reasoning without degrading already-easy tasks.

Table 2: LIBERO-PRO success rate (%) across six perturbation suites. The same frozen π0.5-LIBERO weights are used throughout; only the inference-time process changes.

4.3 Pigey vs. TiPToP: Orchestrating a Visuomotor Policy

Another dimension independent of high-level reasoning comes from how the agent uses low-level primitives. TiPToP uses the same open-vocabulary grounding and motion planning but runs pick-and-place as a single open-loop primitive: it is blind to execution, and a slip or mislocalized grasp can lead to failure. Pigey exposes grasping and placement as two separate tools with verification between them—committing to a place only after the grasp is sensor-confirmed.

The cost of staying open-loop is visible even when grounding is trivial: on simple pick-and-place, TiPToP reaches only 80%—below the raw VLA—because subtle control mistakes can lead to failure, while the closed loop detects the empty gripper and retries, achieving 100%. When adversarial perturbations are introduced, the gap widens: if the target is nudged after planning, an open-loop plan grasps where the object used to be; if initially hidden, the plan never adapts. Pigey observes the changing world, re-perceives and re-plans, turning guaranteed failures into successes.

5. Discussion

Pigey's core contribution lies in reframing general robot manipulation from "full-stack end-to-end learning" to "inference-time decomposition plus closed-loop orchestration." This shift in perspective has profound implications. Traditional methods fold perception, reasoning, planning, verification, recovery, and control entirely into a single network's forward pass, making it impossible to localize which component is missing when failure occurs. Pigey explicitly separates these capabilities: frozen motor backends handle "how to act," while the VLM orchestrator handles "when, why, in what order, and with which skill to act."

The verification mechanism's "deterministic plus visual" dual-signal design is the foundation of Pigey's reliability. The deterministic signal (gripper-width sensor) provides objective binary judgment immune to VLM hallucination; the visual signal (VLM confirmation) supplements semantic-level verification. The two combine conservatively—either signal vetoing triggers failure—preventing false positives from optimistic backends or hallucinatory reasoning. This design transfers directly to other robot orchestration scenarios.

The bidirectional escalation mechanism also embodies important design wisdom. TAMP and VLA each have complementary strengths: TAMP provides geometric precision and verifiable grasp signals but lacks closed-loop robustness; VLA provides closed-loop robustness but unreliable geometric planning. Pigey doesn't presuppose which backend is better but selects the initial path by object type and escalates bidirectionally on failure—TAMP failures escalate to VLA, VLA stalls fall back to TAMP. This "competitive backends" design makes the system's overall capability exceed either backend's individual ceiling.

6. Limitations

Author-stated limitations: Pigey is limited by the skills it orchestrates—while it can compensate for weak policies, there is a ceiling set by the "support" of the low-level tools. Verification is also imperfect—partial observability from occlusions can hide a poor grasp, allowing a false success to propagate downstream. Finally, because Pigey relies on API calls to a frontier model for orchestration, it adds per-step latency and cost, making it challenging for low-latency or high-speed applications.

Independent assessment: Pigey's effectiveness depends heavily on the frontier VLM's reasoning quality. While the paper tests 7 different reasoners and shows the effect holds across models, all are large-scale proprietary models—whether open-source small models can serve as orchestrators is unverified. The tool set (5 tools) is relatively limited; more complex tasks may require richer action spaces. Additionally, the 30 real tasks and LIBERO-PRO are all tabletop manipulation scenarios—Pigey's applicability to navigation, mobile manipulation, or bimanual cooperation is unvalidated. The paper proposes distilling successful traces into smaller orchestrators but this direction is not yet realized, currently relying on online API calls.

7. Conclusion and Outlook

Pigey proposes an inference-time orchestrator that drives frozen TAMP and VLA backends in a closed loop through a frontier VLM, achieving dramatic improvements in frozen skill success rates—4× on LIBERO-PRO and 16.7% to 97.3% on real robots—without collecting new data or training policies. The paper defines this gain as the "orchestration gap": the difference in success rate between frozen motor skills prompted directly and invoked through the agent loop.

From a broader perspective, Pigey represents a significant validation of the "inference-time decomposition" route in robot learning. It demonstrates that a frontier VLM, never trained for embodiment, can ground its reasoning in live camera observations tightly enough to tell a completed grasp from a failed one, a present object from a hidden one, and a finished task from an unfinished one—from general pretraining alone. This finding has guiding implications for the development trajectory of robot AI: rather than baking all capabilities into a single network, separating reasoning from control and combining them through a closed-loop process at inference time may be more effective. Future work may explore distilling successful traces into smaller models to reduce inference cost, and extending to richer action spaces and more diverse robotic scenarios.