PAPER DEEP DIVE
IGGT4D: Streaming 4D Instance-Grounded Geometry Transformer
Real-world spatial intelligence requires agents to understand scenes from continuous video streams, where objects move, persist, disappear, and reappear over time. While recent spatial foundation models have enabled generalizable feed-forward 3D reconstruction, most streaming methods remain geometry-centric and lack temporally consistent object-level understanding. Meanwhile, existing semantic reconstruction and 3D-aware vision-language methods largely rely on externally extracted 2D semantic cues or loosely coupled geometry inputs, limiting unified geometry-instance learning in long dynamic scenes. In this paper, we propose IGGT4D, a streaming instance-grounded geometry Transformer for online 4D scene understanding. IGGT4D processes video frames sequentially, reuses historical context through causal spatial-temporal modeling, and incrementally updates a unified representation of camera motion, geometry, and object identity. This enables long-sequence feed-forward reconstruction with geometry-instance consistency in dynamic environments. To address the lack of high-quality 4D supervision, we further construct InsScene4D-147K, a large-scale dataset spanning real/synthetic and static/dynamic scenes, with RGB images, depth, poses, and temporally consistent instance masks generated by an automated geometry-guided annotation pipeline. Experiments on 3D reconstruction, pose estimation, instance spatial tracking, and open-vocabulary segmentation demonstrate that IGGT4D outperforms existing streaming baselines while maintaining scalable online inference for long dynamic sequences.
1. Overview
Real-world spatial intelligence is inherently online and dynamic. An embodied agent receives a continuous video stream where objects move, become occluded, leave the field of view, and reappear. Estimating camera motion and scene geometry alone is insufficient — the agent must also maintain temporally consistent object identities.
This paper proposes IGGT4D (Streaming Instance-Grounded Geometry Transformer), reformulating 4D scene understanding as streaming geometry-instance prediction: causally predicting camera motion, scene geometry, and persistent object identities from continuous video streams. It also constructs InsScene4D-147K, a large-scale dataset spanning real/synthetic and static/dynamic scenes with RGB images, depth, poses, and temporally consistent instance masks.
2. Related Work
Surveys streaming spatial foundation models (DUSt3R, VGGT, Spann3R, MUSt3R, CUT3R, Stream3R) and instance-aware 3D scene understanding (LangSplat, LSeg, Uni3R, IGGT). Existing streaming models remain geometry-centric without temporally consistent object identity maintenance; existing instance-aware methods use fixed image sets with global cross-view attention, preventing efficient long-video processing.
3. Method
3.1 Problem Formulation
Given RGB sequence $\{I_{t}\}_{t=1}^{N}$, IGGT4D performs feed-forward sequential inference:
$$\mathcal{F}_{\theta}:\left(I_{t},\tilde{\boldsymbol{\pi}}_{t}\right)\mapsto\left(\boldsymbol{\pi}_{t},R_{t},D_{t},S_{t}\right),\quad t=1,\ldots,N$$
Outputs: camera parameters $\boldsymbol{\pi}_{t}=[\mathbf{t}_{t},\mathbf{q}_{t},\mathbf{f}_{t}]\in\mathbb{R}^{9}$ (translation, rotation, field-of-view), ray map $R_{t}=[O_{t},V_{t}]$, depth map $D_{t}$, instance feature map $S_{t}\in\mathbb{R}^{H\times W\times 8}$.
3.2 Streaming Instance-Grounded Geometric Transformer
Adapts DA3's unified geometric Transformer from fixed-set 3D prediction to causal streaming 4D understanding. Each frame encoded into image tokens concatenated with a camera token, processed by 40 Transformer blocks with interleaved intra-view and cross-view attention, producing multi-scale features $\{\mathbf{F}_{t}^{(l)}\}_{l=1}^{4}$.
Key modification: causal masks on cross-camera and cross-view attention so each frame only attends to history. Geometry-Aware Attention (Geo-Attn) lets instance features benefit from geometric priors. First-Frame Geometric Normalization (FF-Norm) eliminates scale ambiguity in streaming reconstruction.
3.3 Efficient Streaming Instance Clustering
Replaces offline HDBSCAN with a lightweight clustering codebook, performing frame-by-frame cosine clustering with constant memory footprint. Objects marked as disappeared if projected area drops sharply for 5 consecutive frames, preventing stale ID propagation.
3.4 4D Scene Understanding Applications
Supports instance spatial tracking, open-vocabulary semantic segmentation, 4D QA scene grounding. Instance features matched with CLIP text features for open-vocabulary querying.
3.5 Training Objectives
Geometry: depth L1 loss $\mathcal{L}_{D}$, ray L1 loss $\mathcal{L}_{R}=\|O_{t}-O_{t}^{\ast}\|_{1}+\|V_{t}-V_{t}^{\ast}\|_{1}$, 3D point reconstruction loss $\mathcal{L}_{P}=\|P_{t}-P_{t}^{\ast}\|_{1}$ where $P_{t}=O_{t}+D_{t}V_{t}$, camera loss $\mathcal{L}_{\pi}=\|\boldsymbol{\pi}_{t}-\boldsymbol{\pi}_{t}^{\ast}\|_{1}$.
Instance: multi-view contrastive loss with intra-view pull/push and cross-view pull/push:
$$\mathcal{L}_{\mathrm{ins}}=\sum_{v}\Big(\lambda_{\mathrm{pull}}^{\mathrm{in}}\sum_{p\in M_{k}^{v}}\big[\|\mathbf{f}_{p}-\mu_{k}^{v}\|_{2}-\delta_{\mathrm{pull}}^{\mathrm{in}}\big]_{+}+\lambda_{\mathrm{push}}^{\mathrm{in}}\sum_{k\neq j}\big[\delta_{\mathrm{push}}^{\mathrm{in}}-\|\mu_{k}^{v}-\mu_{j}^{v}\|_{2}\big]_{+}\Big)+\text{cross-view terms}$$
Final: $\mathcal{L}=\mathcal{L}_{\mathrm{geo}}+\lambda_{\mathrm{ins}}\mathcal{L}_{\mathrm{ins}}$
4. InsScene4D-147K Dataset
147K video sequences across four domains: static-real (RealEstate10K, ScanNet++), static-synthetic (Aria Synthetic, SceneNet, Infinigen, Hypersim), dynamic-real (HOI4D, Waymo, Aria Digital Twin), dynamic-synthetic (RoboTwin 2.0, Kubric, Dynamic Replica, PointOdyssey, VKITTI2).
Geometry-guided annotation: DA3 estimates multi-view consistent depth → TSDF fusion reconstructs 3D mesh → project mesh vertices to image plane forming inheritance map → SAM2 generates masks → IoU matching for ID inheritance/creation. Dynamic scenes use available annotations to override projected pseudo-labels.
5. Architecture
flowchart LR
V[Video Frame Sequence I_t] --> ENC[Frame Encoder
Image Tokens + Camera Token]
ENC --> T40[40 Transformer Blocks
Interleaved Intra+Cross View
Causal Masks]
T40 --> MS[Multi-Scale Features F_t]
MS --> TDPT[Tri-DPT Head]
TDPT --> GEO[Geometry Outputs
Pose π + Ray R + Depth D]
TDPT --> INS[Instance Features S_t]
INS --> SC[Streaming Clustering
Lightweight Codebook + Cosine]
SC --> MASK[Instance Masks
4D Consistent]
GEO --> MERGE[Spatio-Temporal Consistent
Scene Representation]
MASK --> MERGE
MERGE --> DOWN[Downstream: Instance Tracking
Open-Vocab Segmentation
4D QA]
style T40 fill:#4f9eff,color:#fff
style SC fill:#ff6b6b,color:#fff
style MERGE fill:#51cf66,color:#fff
6. Experimental Results
6.1 Camera Pose Estimation and 3D Reconstruction
Evaluated on HiRoom, ETH3D, 7Scenes, ScanNet++. IGGT4D achieves best average pose estimation among streaming models (AUC@3: 0.4464), outperforming several full-attention baselines. 3D reconstruction F1 surpasses all streaming baselines in both predicted and ground-truth pose settings.
| Method | Type | AUC@3 ↑ | AUC@30 ↑ | F1(w/o p.) ↑ | F1(w/ p.) ↑ |
|---|---|---|---|---|---|
| DA3 | Full-attention | 0.6046 | 0.9352 | 0.7430 | 0.7980 |
| Stream3R | Streaming | 0.1871 | 0.7947 | 0.4551 | 0.5504 |
| LingBot-Map | Streaming | 0.3060 | 0.8647 | 0.5228 | 0.6082 |
| IGGT4D | Streaming | 0.4464 | 0.8924 | 0.6678 | 0.7225 |
6.2 Instance Spatial Tracking
On HOI4D, Waymo, ScanNet++, PointOdyssey. IGGT OOMs on long sequences; IGGT4D scales to 100 frames. Long-seq avg T-mIoU 58.84 (vs SAM2's 55.62), T-SR 85.41 (vs 76.07).
| Method | Long T-mIoU | Long T-SR | Short T-mIoU | Short T-SR |
|---|---|---|---|---|
| SAM2 | 55.62 | 76.07 | 59.21 | 79.57 |
| IGGT | OOM | OOM | 61.13 | 94.83 |
| IGGT4D | 58.84 | 85.41 | 63.45 | 96.88 |
6.3 Open-Vocabulary Semantic Segmentation
Highest mIoU and mAcc on Waymo and ScanNet++. Long-seq avg mIoU 40.08 (vs LSeg 31.95), mAcc 73.84 (vs 48.23).
| Method | Long mIoU | Long mAcc | Short mIoU | Short mAcc |
|---|---|---|---|---|
| LSeg | 31.95 | 48.23 | 32.06 | 50.54 |
| IGGT | OOM | OOM | 31.16 | 61.90 |
| IGGT4D | 40.08 | 73.84 | 39.11 | 72.25 |
6.4 Ablation and Efficiency
Removing Geo-Attn degrades instance/semantic performance. Removing FF-Norm causes scale ambiguity across all metrics. Streaming clustering: 100 frames in 7.43s with 0.7GB memory; IGGT's HDBSCAN OOMs at 32 frames.
| Clustering | N=8 Time(s) | N=8 Mem(GB) | N=100 Time(s) | N=100 Mem(GB) |
|---|---|---|---|---|
| IGGT (HDBSCAN) | 105.8 | 13.2 | OOM | OOM |
| IGGT4D (Streaming) | 0.78 | 0.7 | 7.43 | 0.7 |
7. Conclusion
IGGT4D jointly predicts camera motion, scene geometry, and temporally consistent instance features via causal spatial-temporal modeling and geometry-grounded instance prediction. InsScene4D-147K provides large-scale geometry-consistent instance annotations across real/synthetic and static/dynamic scenes. Experiments demonstrate improved object-level consistency across reconstruction, pose estimation, instance tracking, and open-vocabulary segmentation while preserving scalable streaming inference.
前向映射函数
$$ \mathcal{F}_{\theta}:\left(I_{t},\tilde{\boldsymbol{\pi}}_{t}\right)\mapsto\left(\boldsymbol{\pi}_{t},R_{t},D_{t},S_{t}\right) $$
相机位姿参数化
$$ \boldsymbol{\pi}_{t}=[\mathbf{t}_{t},\mathbf{q}_{t},\mathbf{f}_{t}]\in\mathbb{R}^{9} $$
渲染输出表示
$$ R_{t}=[O_{t},V_{t}]\in\mathbb{R}^{H_{r}\times W_{r}\times 6} $$
Limitations include needing broader data coverage and currently relying mainly on supervised learning.SOURCE LINKS
