Vision-Language-Motion Maps: An Open-Vocabulary, Uncertainty-Aware, Queryable Motion Attribute for 3D Scene Maps

Background and Motivation

Open-vocabulary 3D maps let robots answer language queries about "what" and "where" in scenes, but they assume a static world and cannot answer queries about "how" scene elements behave. For example, robots need to know "which door can open," "what is currently moving," "what could move but is currently still"—these motion attribute queries are critical for navigation and manipulation planning in dynamic environments. Existing map systems each have limitations: open-vocabulary maps like VLMaps and ConceptFusion have only semantic features without motion info; Khronos builds 4D metric-semantic maps from geometric motion detection but lacks semantic priors, per-element uncertainty, and natural-language motion queries; Dewan et al. fuse learned objectness priors with observed LiDAR motion in a Bayes filter but are closed-set and not language-queryable; DualMap carries CLIP anchor-vs-volatile movability priors but does not fuse priors with measured geometric motion and stores no motion uncertainty.

This paper proposes Vision-Language-Motion Maps (VLMM), an open-vocabulary, natural-language-queryable 3D map where each element carries a fused motion attribute—combining VLM/LLM semantic movability priors with geometrically observed cross-frame motion, along with per-element uncertainty. Queries reduce to attribute filters distinguishing "seen to move," "could move but hasn't," and "stays still." On the AI2-THOR simulation benchmark (three scene types, exact ground truth), ablation proves schema fields are non-substitutable: a semantic-only baseline fails motion queries even with strong features, and neither motion field substitutes for the other (the prior cannot answer "what is moving," observed motion cannot answer "what could move"). On real dynamic RGB-D data (TUM and Bonn, six sequences), the uncertainty channel—the key differentiator from prior fused-motion work—consistently improves moving-vs-static detection average precision and reduces false motion flags.

Map Schema and Element Representation

VLMM map elements carry a seven-tuple schema:

$$e_i = \left(\mathbf{p}_i, \mathbf{f}_i, o_i, \kappa_i, \rho_i, y_i, c_i\right)$$

where $\mathbf{p}_i \in \mathbb{R}^3$ is position, $\mathbf{f}_i$ is CLIP semantic feature, $o_i \geq 0$ is observed motion score, $\kappa_i \in [0,1]$ is observed motion confidence, $\rho_i \in [0,1]$ is semantic movability prior, $y_i \in \mathcal{Y} = \{\text{static}, \text{movable\_static}, \text{moving}\}$ is motion class, and $c_i$ is class confidence. Semantic features are obtained via CLIP image encoder:

$$\mathbf{f}_i = \text{CLIP}_{\text{img}}\left(\text{crop}(I, \mathcal{M}_i)\right) / \|\cdot\|_2$$

Language query $q$ matches elements via text-image similarity: $s_i(q) = \mathbf{f}_i^\top \text{CLIP}_{\text{txt}}(q)$. This design lets motion and semantic attributes coexist in the same map element, supporting compound queries like "the door that is moving" (semantic=door ∧ motion=moving).

Observed Motion Channel

Observed motion is computed via optical flow and depth back-projection. For pixel $u$ in frame $a$, the world coordinate is:

$$\mathbf{X}^w_a(u) = R_a \cdot z_a(u) K^{-1}[u_x, u_y, 1]^\top + \mathbf{t}_a$$

Optical flow $\mathbf{w}_{a \to b}(u)$ finds the corresponding pixel $u' = u + \mathbf{w}_{a \to b}(u)$ in frame $b$, with forward-backward consistency gating $\|\mathbf{w}_{a \to b}(u) + \mathbf{w}_{b \to a}(u')\| < \epsilon$ filtering unreliable matches. The geometric motion vector is $\mathbf{m}(u) = \mathbf{X}^w_b(u') - \mathbf{X}^w_a(u)$. Ego-motion compensation estimates $(\hat{R}, \hat{\mathbf{t}})$ via RANSAC + Kabsch alignment:

$$\hat{R} = U \cdot \text{diag}(1, 1, \det(UV^\top)) \cdot V^\top, \quad \hat{\mathbf{t}} = \bar{\mathbf{X}}_b - \hat{R}\bar{\mathbf{X}}_a$$

The compensated residual motion is $\tilde{\mathbf{m}}(u) = \mathbf{X}^w_b(u') - (\hat{R}\mathbf{X}^w_a(u) + \hat{\mathbf{t}})$. Uncertainty propagates through the depth noise model—depth variance $\sigma_z^2(z) = (a + b(z - z_0)^2)^2$, Jacobian $J = [\frac{z}{f_x}\mathbf{e}_1, \frac{z}{f_y}\mathbf{e}_2, \frac{1}{z}\mathbf{X}^c]$, covariance $\Sigma^c = J \cdot \text{diag}(\sigma_{\text{px}}^2, \sigma_{\text{px}}^2, \sigma_z^2) \cdot J^\top$, world-frame covariance $\Sigma^w = R \Sigma^c R^\top$, motion covariance $\Sigma_m = \Sigma^w_a + \hat{R}\Sigma^w_b \hat{R}^\top$.

Mahalanobis Distance Motion Detection

Motion significance is measured via Mahalanobis distance, normalizing observed motion to uncertainty scale:

$$d_M(u) = \sqrt{\tilde{\mathbf{m}}(u)^\top \Sigma_m^{-1} \tilde{\mathbf{m}}(u)}$$

Under the null hypothesis $H_0: \mathbb{E}[\tilde{\mathbf{m}}] = \mathbf{0}$, $d_M^2 \sim \chi^2_3$, with motion threshold $\tau = \sqrt{\chi^2_{3, 1-\alpha}}$ ($\alpha = 0.05 \Rightarrow \tau \approx 2.80$) and motion probability $p_{\text{mot}}(u) = \Pr(\chi^2_3 > d_M^2)$. Per-object aggregation: observed score $o_i = Q_{75}\{d_M(u) : u \in \mathcal{P}_i\}$ (75th percentile), motion pixel fraction $\phi_i = \frac{1}{|\mathcal{P}_i|}\sum_{u \in \mathcal{P}_i} \mathbf{1}[d_M(u) > \tau]$, motion direction consistency $\gamma_i = \|\frac{1}{|\mathcal{P}_i|}\sum_{u \in \mathcal{P}_i} \frac{\tilde{\mathbf{m}}(u)}{\|\tilde{\mathbf{m}}(u)\|}\|$, observed confidence $\kappa_i = \phi_i \cdot \gamma_i$.

flowchart TD
    RGB["RGB-D frames a, b"] --> FLOW["RAFT optical flow
w_{a→b}(u)"]
    FLOW --> CONSIST["F-B consistency gate
||w_a→b + w_b→a|| < ε"]
    CONSIST --> DEPTH["Depth back-projection
X^w = R·z·K⁻¹u + t"]
    DEPTH --> EGO["Ego-motion compensation
RANSAC + Kabsch"]
    EGO --> RESID["Residual motion m̃(u)"]
    RESID --> UNCERT["Uncertainty propagation
Σ_m = Σ_a + R̂Σ_bR̂ᵀ"]
    UNCERT --> MAHAL["Mahalanobis dist d_M(u)"]
    MAHAL --> AGG["Object aggregation
o_i=Q75, κ_i=φ_i·γ_i"]
    VLM["VLM/LLM
movability prior ρ_i"] --> FUSE
    AGG --> FUSE["Fusion classification
y_i = f(o_i, κ_i, ρ_i)"]
    FUSE --> MAP["VLMM map element
(p, f, o, κ, ρ, y, c)"]
    QUERY["Language query q"] --> MAP

Fusion Classification and Semantic Prior

The semantic movability prior $\rho_i = \text{VLM}(\ell_i) \in [0,1]$ is given by VLM/LLM based on object category label $\ell_i$, as a cached per-category call (no per-frame cost). Fusion classification rules:

$$y_i = \begin{cases} \text{moving} & \text{if } o_i > \tau_o \wedge \kappa_i \geq \kappa_o \\ \text{movable\_static} & \text{if } y_i \neq \text{moving} \wedge \rho_i \geq \rho_\tau \\ \text{static} & \text{otherwise} \end{cases}$$

Confidence $c_i$ takes $\kappa_i$ for moving, $\rho_i$ for movable_static, and $1 - \rho_i$ for static. Language queries of the form $\langle s, \text{field}, \text{ret} \rangle$ return elements satisfying semantic similarity and field conditions: $\{e_i : s_i(s) \geq \theta_s \wedge \pi_{\text{field}}(e_i)\}$.

System Implementation and Compute Constraints

The system's constraint of running on a single laptop GPU reflects attention to practical deployment feasibility. The 8GB VRAM limit is addressed through three strategies: caching per-keyframe dense CLIP features to disk so peak VRAM never holds all frames simultaneously; VLM/LLM movability prior as cached per-category call eliminating per-frame inference cost; and avoiding any component requiring multi-GPU servers. This lightweight design enables VLMM to run on edge devices, critical for real robot deployment. RAFT optical flow with 2px forward-backward consistency gate filters unreliable matches, RANSAC 2cm inlier threshold and Kabsch alignment achieve robust ego-motion compensation. Per-object score $o_i = Q_{75}(d_M)$ uses the 75th percentile rather than maximum, providing robustness to outlier pixels.

Experimental Results

Simulation Ablation (AI2-THOR Exact GT)

On kitchen, living room, and bedroom scenes (67 objects) with exact ground truth labels, camera geometry validated to 1.1cm cross-view static-point alignment. Ablation proves schema fields are non-substitutable:

The semantic-only baseline achieves only AP=0.18 (random 0.09) on "what is moving"—appearance features do not encode instantaneous motion. VLMM, ranking by observed motion, exactly returns the ground-truth movers. DualMap's anchor-volatile mechanism answers "what could move" well (0.80) but cannot answer "what is moving." The two motion fields are non-substitutable: the prior cannot answer "what is moving," and observed motion cannot answer "what could move."

Real RGB-D Uncertainty Validation

On TUM (4 sequences) and Bonn (2 sequences) real noisy dynamic data, the uncertainty channel consistently improves moving-vs-static detection average precision and reduces false flags. Uncertainty weighting ties with the no-uncertainty version on exact depth, separating only under sensor noise—it is a real-noise property, not a clean-sim one. Raw confidence is not calibrated, but post-hoc isotonic calibration reaches expected calibration error ECE = 0.10.

Figure 1: VLMM system overview—queryable 3D map fusing semantic prior with observed motion

Figure 2: Motion detection pipeline—optical flow, depth back-projection, ego-motion compensation, Mahalanobis distance

Figure 3: Comparison of semantic-only baseline vs VLMM on "what is moving" query

Limitations and Future Work

Limitation 1: Real-data mover ground truth is a proxy. Moving-region ground truth in TUM/Bonn sequences comes from a person segmenter (proxy), not from non-person object motion ground truth. AI2-THOR has exact non-person object motion ground truth but is simulation. A real-world non-person-mover benchmark is an open problem.

Limitation 2: Confidence not pre-calibrated. Raw confidence requires post-hoc isotonic calibration (ECE=0.10) to reach usable levels. Future work needs released baseline comparisons and a real non-person-mover benchmark. The system runs on a single laptop GPU (RTX 4060 8GB)—an explicit design constraint, achieved by caching per-keyframe CLIP features to disk, VLM prior per-category caching, and avoiding multi-GPU components.

Summary and Insights

VLMM adds a queryable, uncertainty-aware, fused motion attribute to open-vocabulary 3D maps. Key technical contributions include: the seven-tuple element schema $e_i = (\mathbf{p}_i, \mathbf{f}_i, o_i, \kappa_i, \rho_i, y_i, c_i)$ unifying semantic features, observed motion, semantic prior, and motion class in a single map element; Mahalanobis distance motion detection $d_M(u) = \sqrt{\tilde{\mathbf{m}}^\top \Sigma_m^{-1} \tilde{\mathbf{m}}}$ ($d_M^2 \sim \chi^2_3$, threshold $\tau \approx 2.80$) normalizing motion significance to uncertainty scale, preventing distant objects with large depth noise from being misclassified as moving; per-object aggregation $o_i = Q_{75}(d_M)$, $\kappa_i = \phi_i \gamma_i$ combining motion pixel fraction and direction consistency; fusion classification rules distinguishing moving/movable_static/static; language queries $\{e_i : s_i(s) \geq \theta_s \wedge \pi_{\text{field}}(e_i)\}$ supporting compound semantic-motion queries. On AI2-THOR, schema field non-substitutability is proven (semantic-only AP=0.18 vs VLMM 1.00), and on six real RGB-D sequences the uncertainty channel consistently improves AP and reduces false flags, with isotonic calibration reaching ECE=0.10. This work fills the gap of motion attribute querying in open-vocabulary maps, providing fine-grained "what is moving / what could move / what is static" scene understanding for robot navigation and manipulation planning in dynamic environments. The uncertainty propagation $\Sigma_m = \Sigma^w_a + \hat{R}\Sigma^w_b\hat{R}^\top$ is the key differentiator from prior fused-motion work—making motion detection robust under real sensor noise rather than only effective with exact depth in simulation. The constraint of running on RTX 4060 8GB also reflects attention to practical deployment feasibility.