Robot Planning and Situation Handling with Active Perception (VAP-TAMP)
SUNY Binghamton + CMU + Ford Research + Agility Robotics. A TAMP (task and motion planning) framework with VLM-based active perception. Core problem: TAMP assumes a fully observable world, but execution constantly surfaces unforeseen situations — a door only half open, one lemon half fallen out of the plate. Prior open-world planning implicitly assumed situations are fully observable, while real robots rely on unreliable onboard sensors. VAP-TAMP’s key novelty is the bidirectional interaction between VLMs and action knowledge: predicates from the current action’s PDDL preconditions/effects construct VQA prompts for the VLM, and VLM outputs update the action knowledge. Three modules: (1) scene graph generator (RGB-D point cloud + instance segmentation + CLIP embeddings; geometric rules for on/inside/near; updated via both action-driven and observation-driven channels); (2) active perception (N semantically equivalent paraphrases per predicate with majority voting; inconsistent responses → the VLM suggests a viewing direction (left/right/closer/above) → navigate and re-observe, looping within budget K until the view is sufficient); (3) situation handling (verify-execute-verify loop: preconditions checked before, effects after; any failure → correct the scene graph → PDDL replanning; diagnoses whether discrepancies stem from perception error, world state change, or execution failure). Real-world: Segway RMP + UR5e + Robotiq gripper, 4 household tasks × 25 trials (100 total), 88% success (OK-Robot 76%, COWP 69%, Closed-World 33%); 72/100 trials contained at least one unforeseen situation; VLM-guided viewpoint selection averages 1.61 views vs 4.35 for greedy exploration (63% fewer); failure-mode analysis shows VAP-TAMP failures concentrate in manipulation hardware (42%) — the perceptual bottleneck has been shifted away. Simulation (OmniGibson, 5 tasks with injected failure probabilities) shows predicate-based verification beats SuccessVQA/AffordanceVQA and their combination, with effect verification contributing more than precondition verification (grasp/cut carry 50% combined failure rates detectable only post-execution).