SuperMap: A Spatio-Temporal SLAM System for Visual-Language Navigation

SuperMap: A Spatio-Temporal SLAM System for Visual-Language Navigation

Authors: Shibo Zhao, Guofei Chen, Honghao Zhu, Zhiheng Li, Changwei Yao, Nader Zantout, Seungchan Kim, Wenshan Wang, Ji Zhang, Sebastian Scherer
Institution: Carnegie Mellon University, The Robotics Institute
Published: RSS 2022
Project: superodometry.com/supermap
Keywords: Spatio-temporal SLAM, open-vocabulary perception, scene graph, visual-language navigation, dynamic environment mapping

One-Sentence Summary

SuperMap is the first real-time, open-vocabulary, instance-level 4D spatio-temporal semantic SLAM system — fusing high-frequency geometric SLAM with asynchronous open-vocabulary perception through 3D-aware instance association/re-activation and confidence-driven existence-label updates to maintain stable object identities under occlusions and scene changes, producing a queryable 4D scene graph for VLM-based zero-shot language navigation.

Background and Motivation

Robots executing open-vocabulary navigation goals (e.g., "go to the monitor next to the whiteboard" or "return to the chair that was near the plant earlier") require more than pose estimation — they need an object-centric spatio-temporal map preserving instance identities, semantic attributes, and spatial relations over time. Real scenes are dynamic: objects may be occluded, relocated, removed, or newly introduced.

Existing methods have a fundamental limitation: most address either short-term dynamics (moving people) or static object recognition, rarely both simultaneously. Semantic mapping methods (ConceptGraphs, HOV-SG) require offline processing (minutes to hours); dynamic SLAM methods (Khronos) are limited to closed-set categories and cannot run in real time; open-vocabulary methods (OpenScene, OpenMask3D) assume complete scene data, precluding online use.

SuperMap's core insight: decompose the spatio-temporal SLAM problem into three chain-rule sub-problems — pose estimation (geometric), spatio-temporal instance association, and online map update — each with clear probabilistic semantics.

Problem Formulation: Probabilistic Decomposition

The system processes continuous RGB-D or point-cloud video streams $C_{1:T}$ and $D_{1:T}$, estimating camera poses $P_{1:T}$ and outputting a global map $M_t$ with semantic instance objects at each time $t$. The spatio-temporal SLAM problem: given observation $Q_t=\{C_t, D_t\}$ and existing map $M_{t-1}$, estimate pose $P_t$, instance IDs $I_t$, and updated map $M_t$ by maximizing:

Applying the chain rule yields three factors:

Method

1. Geometric Layer: Online 3D Reconstruction

SuperOdometry provides robot state $T_{WB}^{(t)} \in SE(3)$. Given extrinsic $T_{BC}$, the camera pose is:

2. Instance Layer: Spatio-Temporal Association

2.1 Hybrid Tracking State

The tracklet state $S_i(t) \in \mathbb{R}^6$ is defined as:

where $c_i(t) = [x, y]^\top$ is the 2D image centroid, $s_i(t) = [w, h]^\top$ is the bounding box size, and $\dot{c}_i(t)$ is the translational velocity.

2.2 3D-to-2D Motion Compensation

Standard 2D trackers fail under rapid ego-motion. SuperMap projects the 3D centroid $X_i$ from $M_{t-1}$ using current pose $P_t$:

The Kalman Filter uses this projection as prior:

2.3 Probabilistic Geometric Consistency

Depth residual for each map point $X_k$:

Occupancy state updated via log-odds:

Three-state classification based on $\Delta d$:

2.4 Bayesian Semantic Fusion

Each instance maintains a categorical distribution $P(L_j = c)$. Posterior update:

3. Topological Layer: 4D Scene Graph

Map $M_t$ abstracted as graph $G = (V, E_S, E_T)$:

• Spatial Edges ($E_S$): Class-dependent geometric predicates, e.g., $\text{On}(A, B) \Leftrightarrow (z_A^{\min} \approx z_B^{\max}) \wedge (\text{IoU}_{xy}(B_A, B_B) > \gamma)$
• Temporal Edges ($E_T$): Link nodes across time based on association results $I_t$, tracing object trajectories

Experiments

1. Semantic Quality on ScanNet

2. Instance-level Segmentation

3. Change Detection

4. Ablation Study

5. Visual-Language Navigation

The robot incrementally builds an instance-level semantic map during autonomous exploration. Given a language command, the VLM performs zero-shot reasoning via the serialized scene graph, returning target instance IDs. A parser extracts IDs and retrieves 3D centroids $X_j$ from $M_t$ as navigation waypoints.

System Pipeline

flowchart TD
    A["Sensors: RGB + Depth/LiDAR + IMU"] --> B["Geometric Layer: SuperOdometry"]
    B --> C["Pose P_t + Dense 3D Reconstruction"]
    C --> D["Instance Layer: GroundingDINO + SAM2"]
    D --> E["3D-to-2D Motion-Compensated Tracking"]
    E --> F["Geometric Consistency Update
Log-odds Occupancy + Change Detection"]
    F --> G["Bayesian Semantic Fusion
Recursive P(L=c|z) Update"]
    G --> H["Topological Layer: 4D Scene Graph"]
    H --> I["Spatial Edges E_S + Temporal Edges E_T"]
    I --> J["VLM Serialization Interface
Zero-shot Language Navigation"]
    J --> K["Waypoint Generation + Robot Execution"]
    style B fill:#e3f2fd
    style F fill:#e8f5e9
    style H fill:#fff3e0
    style J fill:#fce4ec

Limitations

• Detector dependency: Performance bounded by GroundingDINO and SAM2 quality; rare or unseen categories may be missed.
• Threshold sensitivity: Depth residual threshold $\tau_\epsilon$ requires tuning per sensor and environment.
• Compute requirements: Requires RTX 4090 + i9 for real-time operation; may not suit resource-constrained edge devices.
• Limited spatial predicates: Current relations (on, beside, under) are basic; complex spatial reasoning (e.g., "behind") may need richer geometric inference.

Conclusion

SuperMap's core contribution is the probabilistic decomposition of spatio-temporal SLAM into geometry-association-update, each with clear mathematical foundations. The 3D-to-2D motion compensation solves tracking under rapid ego-motion by projecting 3D centroids rather than predicting in 2D. The log-odds occupancy update with three-state classification (observable/unobservable/disappeared) distinguishes "not visible" from "removed" — critical for long-term map maintenance. Bayesian semantic fusion smooths open-vocabulary detector noise through cross-frame belief accumulation rather than single-frame decisions.

The 4D scene graph serves as "structured memory" between VLMs and the physical world: VLMs reason over symbolic spatio-temporal relations instead of raw point clouds or video. Spatial logic queries leverage metric edges for precise distance computation; temporal logic queries traverse $E_T$ for object history — capabilities that raw video input cannot reliably provide. SuperMap's open-source system provides a deployable baseline for open-vocabulary spatio-temporal mapping.