#VLM (17)

OneCanvas: reproject video frames onto a single panoramic canvas so a stock 2D VLM reasons in 3D — SOTA on SQA3D, VSI-Bench and SPBench at an order of magnitude less compute

@unknown · 452 views · 2026-06-19

Robot Planning and Situation Handling with Active Perception (VAP-TAMP)

SUNY Binghamton + CMU + Ford Research + Agility Robotics. A TAMP (task and motion planning) framework with VLM-based active perception. Core problem: TAMP assumes a fully observable world, but execution constantly surfaces unforeseen situations — a door only half open, one lemon half fallen out of the plate. Prior open-world planning implicitly assumed situations are fully observable, while real robots rely on unreliable onboard sensors. VAP-TAMP’s key novelty is the bidirectional interaction between VLMs and action knowledge: predicates from the current action’s PDDL preconditions/effects construct VQA prompts for the VLM, and VLM outputs update the action knowledge. Three modules: (1) scene graph generator (RGB-D point cloud + instance segmentation + CLIP embeddings; geometric rules for on/inside/near; updated via both action-driven and observation-driven channels); (2) active perception (N semantically equivalent paraphrases per predicate with majority voting; inconsistent responses → the VLM suggests a viewing direction (left/right/closer/above) → navigate and re-observe, looping within budget K until the view is sufficient); (3) situation handling (verify-execute-verify loop: preconditions checked before, effects after; any failure → correct the scene graph → PDDL replanning; diagnoses whether discrepancies stem from perception error, world state change, or execution failure). Real-world: Segway RMP + UR5e + Robotiq gripper, 4 household tasks × 25 trials (100 total), 88% success (OK-Robot 76%, COWP 69%, Closed-World 33%); 72/100 trials contained at least one unforeseen situation; VLM-guided viewpoint selection averages 1.61 views vs 4.35 for greedy exploration (63% fewer); failure-mode analysis shows VAP-TAMP failures concentrate in manipulation hardware (42%) — the perceptual bottleneck has been shifted away. Simulation (OmniGibson, 5 tasks with injected failure probabilities) shows predicate-based verification beats SuccessVQA/AffordanceVQA and their combination, with effect verification contributing more than precondition verification (grasp/cut carry 50% combined failure rates detectable only post-execution).

Austine Oloo, Zainab Altaweel, Yohei Hayamizu · 468 views · Apr 28, 2026

TAMPActive PerceptionVLMApr 28, 2026

0:45

HomeBody: a frontier VLM drives a Unitree G1 skill library directly, with no learned VLA in between

@CyberRobooo · 918 views · 2026-09-28

HumanoidUnitree G1VLM

World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal

HKUST(GZ), CUHK and Knowin AI present World Action Agent (WAA), a multi-agent harness that changes what the VLM sees and how its decisions take effect — rather than the VLM itself — so a general-purpose VLM can pilot a robot with basic tools (point, drag, preview, execute), making every decision inside one visual action workspace. Three properties: Contact views, whose camera parameters are solved from interaction-region visibility under occlusion, framing compactness, view redundancy and stability, with the feasible set constraining the two views to orthogonal horizontal projections so any alignment error can be read along two independent directions — requiring only a base-frame point cloud, so the perception source (simulation, fused RGB-D, or VGGT reconstruction) is irrelevant; action rehearsal, where each action is an editable proposal planned by cuRobo, overlaid as a translucent robot in every view with a feasibility report, refined by an Imagination Agent in a separate context while the physical scene stays unchanged, and only proposals with executable plans become motion; and in-view correction, where the agent drags from a reference point to the desired location in the Contact view where the error is observed — the reference can be a visible point on the held object, so no conversion into an absolute gripper pose is needed. Skills evolve from expert videos and human Canvas-GUI teaching through Learner, Editor and Reviewer roles under evidence-citing and independent-review constraints. On LIBERO-Pro with Gemini 3.7 Flash, using skills evolved only from LIBERO-90 and frozen before evaluation, WAA reaches a state-of-the-art 75.6% average success, above ASPIRE (72.0%), end-to-end VLAs and the same-backbone Show-Harness (6.7%), scoring 80.0 / 73.3 on the two Spatial splits; an episode costs just 31 model calls, 150 s and $0.1996 versus 120 calls, 874 s and $0.5021 for Show-Harness. The frozen skills transfer to robosuite at 100.0%, and LoRA fine-tuning Qwen3.5-9B on 112 trajectories (1,774 decision steps) as the main agent only raises out-of-domain success from 1.7% to 43.3%.

Yehang Zhang, Haojian Huang, Yifan Chang · 776 views · Sep 24, 2026

World Action AgentWAAVLMSep 24, 2026

0:07

Ludi₀.₁: An Agentic System for Socially Intelligent Robots

Ludo Robotics presents Ludi₀.₁, their first agentic system integrating perception, navigation, and manipulation with interactive speech, dialogue, memory, and social reasoning — powered by a fine-tuned Qwen3.5 VLM, GR00T VLA policies, and KISS-ICP navigation.

BLOG

agentic systemLudo RoboticsVLMNVIDIA GR00TQwen3.5

AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation

Vision-and-Language Navigation (VLN) requires an agent to ground language instructions to its own movement within a visual environment. While state-of-the-art methods leverage the reasoning capabilities of Vision-Language Models (VLMs) for end-to-end action prediction, they often lack an explicit and explainable understanding of the relationships between the agent, the instruction, and the scene. Conversely, explicitly building a scene map for heuristic planning is intuitively appealing but relies on additional 3D sensors and hinders large-scale vision-language pre-training. To bridge this gap, we propose AwareVLN, a novel framework that equips the navigation model with a self-aware reasoning mechanism, enabling it to understand the agent's state and task progress in a fully end-to-end and data-driven manner. Our approach features two key innovations: (1) a structural reasoning module that fosters spatial and task-oriented self-awareness, and (2) an automatic data engine with progress division for effective training. Extensive experiments on various datasets in Habitat simulator show our AwareVLN significantly outperforms previous state-of-the-art vision-language navigation methods. Project page: https://gwxuan.github.io/AwareVLN/.

Wenxuan Guo, Xiuwei Xu, Yichen Liu · 377 views · May 21, 2026

VLMVLNVision-language navigationMay 21, 2026

0:18

Microsoft Mage-VL: 4B Streaming VLM for Real-Time Live Video Understanding

@berryxia · 320 views · 2026-07-28

VLMVision-language modelsStreaming Understanding

3D-Aware VLMs with Implicit and Explicit Geometries

Despite rapid progress, most existing vision-language models (VLMs) built from 2D visual inputs often struggle when handling various 3D tasks that require fine-grained spatial understanding and reasoning. To bridge this gap, we present VLM-IE3D, a unified framework that enhances the 3D spatial awareness of VLMs by equipping them with both implicit and explicit 3D geometries learned from RGB videos. Our VLM-IE3D introduces Implicit Geometry Tokens (IGTs) that capture high-level geometric priors from input videos, as well as complementary Explicit Geometry Tokens (EGTs) that encode detailed geometric structures from reconstructed 3D attributes. On top of that, VLM-IE3D comes with a 3D-aware adapter that effectively fuses the two types of geometric representations with 2D visual cues. This RGB-only design injects strong 3D inductive biases for fine-grained spatial understanding and reasoning without requiring any additional 3D inputs. Extensive experiments show that VLM-IE3D achieves superior performance consistently across various 3D tasks including 3D video detection, 3D visual grounding, 3D dense captioning, and spatial reasoning. Code and models are available at https://github.com/Vegetebird/VLM-IE3D.

Wenhao Li, Xueying Jiang, Quanhao Qian · 388 views · Jul 23, 2026

3D-AwareVLMGeometryJul 23, 2026

0:18

Open-set object detection recognize unseen objects

@ctorobotics · 472 views · 2026-07-26

PerceptionOpen SetDetection

1:09

TOPReward zero-training robot reward model

@IlirAliu_ · 405 views · 2026-07-26

VLAReward ModelVLM

0:39

Astribot releases Lumo-2 World-Action Model

@LeoKharon · 492 views · 2026-07-23

WAMLumo-2Astribot

Exo2EgoPose: Leveraging Exocentric Demonstrations for Vision-Language guided Egocentric 3D Hand Pose Forecasting

Perceiving multimodal cues and forecasting fine-grained actions from an egocentric (Ego) perspective is vital for applications like robot manipulation. However, previous studies either rely mainly on under-informed visual inputs to predict coarse human motions or follow the VRM/VLA paradigm, which suffers from insufficient robot data and the gap between human and robot embodiments. We observe that 3D hand pose naturally serves as a unified representation to bridge human-robot actions. Hence, we investigate an under-explored Vision-Language guided Egocentric 3D Hand Pose Forecasting (VL-EHPF) task, which aims to predict future Ego 3D hand poses from visual observations, a language instruction, and pose states. To overcome the limited field-of-view and highly dynamic motions in the Ego view, we propose a framework dubbed Exo2EgoPose, which innovatively leverages holistic and stable exocentric (Exo) demonstrations as guidance to compensate for partial and dynamic Ego-view cues. Specifically, we introduce a Dual-level Exocentric Reconstruction Module (DERM), which incorporates the paired Exo videos as supervision to reconstruct their video-level and chunked frame-level representations, thereby modeling spatial contexts and temporal dynamics. Then, the Global-to-Local Modulation Module (GLMM) utilizes the reconstructed hierarchical Exo representations for progressive feature refinement via attention mechanisms and adaptive modulation, enabling comprehensive Exo guidance for accurate Ego hand pose forecasting. Extensive experiments on AssemblyHands, Ego-Exo4D, and our newly constructed EgoMe-pose benchmarks show the superiority of our method, which outperforms state-of-the-art methods by a large margin. Moreover, it demonstrates an effective human-to-robot transfer capability and yields improvements on the CALVIN dataset. Code will be released.

Zhaofeng Shi, Heqian Qiu, Lanxiao Wang · 362 views · Jul 17, 2026

First-personHand PoseVLMJul 17, 2026

As an Amazon Associate, we earn from qualifying purchases.