ZeroSplat: Generalized Referring Segmentation in 3D Gaussian Splatting

ZeroSplat: Generalized Referring Segmentation in 3D Gaussian Splatting

Paper: ZeroSplat: Generalized Referring Segmentation in 3D Gaussian Splatting
Authors: Jiayu Ding, Meilu Song, Xiaoyi Zhang, Hongbo Jin, Yichen Jin, Xiangtian Si
Affiliations: Peking University, North China Electric Power University, China University of Geosciences, InkMind.AI
Links: arXiv:2607.18801 | Project Page
Code: Open source

One-Sentence Summary

ZeroSplat introduces the Generalized Referring 3D Gaussian Splatting Segmentation (GR3DGS) task supporting arbitrary target counts (0/1/N), and solves it with a training-free, zero-feature framework that lifts 2D VLM priors into 3D space via multi-view geometric constraints for intrinsic point-level understanding, substantially outperforming SOTA in both generalized and single-target scenarios.

Background and Motivation

3D Gaussian Splatting (3DGS) represents scenes with explicit 3D Gaussians for high-quality real-time rendering. Each Gaussian is characterized by mean position, covariance matrix (scale and orientation), color, and opacity. Rendering projects Gaussians onto the image plane in depth-sorted order, with pixel color computed via alpha compositing: $C(p)=\sum_{i=1}^{|\mathcal{G}_p|}c_i\alpha_i\prod_{j=1}^{i-1}(1-\alpha_j)$. Beyond visual synthesis, recent research has equipped 3DGS with semantic understanding.

Open-vocabulary 3DGS understanding methods follow pixel-based and point-based paradigms. Pixel-based methods adopt a "render-then-match" strategy, rendering scenes into dense 2D feature maps for image-space reasoning. LEGaussians, LangSplat, and Feature-3DGS distill 2D foundation model features into view-consistent 3D representations. However, these methods treat 3D representations primarily as rendering proxies, lacking structural scene understanding and unable to directly identify or manipulate individual Gaussian primitives. Point-based methods adopt "match-then-render," treating 3D Gaussian primitives as basic understanding units. OpenGaussian and InstanceGaussian learn 3D-consistent instance features using SAM 2D masks; Dr.Splat and LUDVIG lift 2D features directly onto 3D points via aggregation or graph diffusion. However, extending point-based advantages to referring 3DGS understanding remains largely unexplored.

Referring 3D Gaussian Splatting (R3DGS) fills the gap between open-vocabulary and free-form language, segmenting targets from elaborate text prompts. ReferSplat uses a position-aware cross-modal module to align 3D Gaussians with text queries. However, existing R3DGS paradigms have a critical limitation: strict "single-target" setting, presupposing each instruction corresponds to exactly one target. This severely hinders flexibility in real scenarios—user instructions are inherently uncertain, often involving multi-target requests (e.g., "find all red chairs") or no-target queries (object absent). When targets are absent, single-target methods still output a segmentation, producing false positives.

To address this, the paper introduces Generalized Referring 3D Gaussian Splatting Segmentation (GR3DGS), requiring parsing instructions to segment arbitrary numbers of targets (0, 1, or N), imposing higher demands on semantic discrimination and ambiguity robustness. The model must dynamically segment multiple targets, a single target, or output an empty mask if the queried object is absent. Two benchmarks are constructed: GR-LERF (pixel-level evaluation with 4 scenes: Ramen, Teatime, Figurines, Waldo) and GR-ScanNet (3D point-level evaluation based on ScanNet with complex spatial and functional queries). The annotation protocol has three stages: instruction generation (based on appearance, spatial relations, functions, target counting), mask annotation, and quality verification.

Beyond task formulation, existing paradigms encounter two critical bottlenecks on GR3DGS. First, lack of 3D point-level understanding—existing R3DGS methods use 2D rendered pixels as the semantic processing unit rather than operating on discrete 3D Gaussian points, remaining confined to 2D image-level understanding and unable to leverage explicit 3D geometric structure to resolve spatial ambiguities. Second, prohibitive computational and memory overhead—requiring per-scene optimization (typically 1-2 hours) and embedding high-capacity semantic features into millions of Gaussians (~3 GB extra storage), with peak VRAM of 20-28 GB, making them impractical for real-time and resource-constrained deployment.

ZeroSplat bridges these gaps with a training-free, zero-feature framework. The core insight: robust 3D semantic understanding does not require altering the intrinsic scene representation but can be achieved by lifting 2D foundation model priors into 3D space through geometric constraints. By projecting 2D VLM semantic cues onto the 3D structure, it directly filters and localizes targets within the original 3D Gaussian set, requiring no extra feature storage or per-scene optimization. This design stores zero semantic features, supports plug-and-play inference, with peak VRAM of only ~10 GB, making 3D semantic understanding deployable on edge devices and mobile platforms, significant for real-time scene perception of embodied agents.

Method Details

ZeroSplat is a three-stage end-to-end framework hierarchically connecting unstructured text to 3D geometric anchors: (1) Geometry-Guided VLM Semantic Parsing; (2) 2D Semantic Mask Generation and 3D Lifting; (3) Multi-View Verification and Spatial Refinement. Given a 3DGS scene $\mathcal{G}$ reconstructed from multi-view images and a free-form natural language expression $\mathcal{T}$, the goal is to assign a binary semantic label to each Gaussian $g\in\mathcal{G}$.

Figure 1: ZeroSplat method overview. (a) Keyframe extraction + VLM two-stage parsing (semantic label extraction → 2D localization). (b) Multi-view 2D mask generation and 3D lifting. (c) Multi-view verification and spatial refinement.

Geometry-Guided VLM Semantic Parsing. First, geometric keyframes are selected via curvature evaluation—voxelizing the scene and computing curvature variation of visible voxels per frame, greedily selecting $K=10$ frames covering high-value geometric regions, filtering redundant background views. Then a VLM (qwen3-vl-30b-a3b-instruct) performs two-stage interaction. Stage 1 semantic label extraction: input referring text $\mathcal{T}$ and keyframes $\mathcal{I}_{key}$ to extract a concise semantic label set $\mathcal{C}=\{C_1,\ldots,C_M\}$ from verbose descriptions—simplifying complex free-form language into category labels for 2D segmentation. Stage 2 2D geometric localization: feed labels $C_j$, original text $\mathcal{T}$, and keyframes back to output normalized 2D bounding boxes $\mathcal{B}_i$ per keyframe as spatial priors for filtering 3D Gaussian artifacts. Ablation shows removing semantic label extraction causes severe degradation (24.3 mIoU) due to high ambiguity from feeding free-form prompts directly to 2D segmentation.

2D Semantic Mask Generation and 3D Lifting. SAM3 generates multi-view 2D masks per category and frame. Adaptive view selection balances semantic purity and geometric coverage: high-confidence threshold $\tau_{high}=0.6$, base $\tau_{base}=0.3$, safe $\tau_{safe}=0.15$ (0 for open-vocabulary), prioritizing high-confidence views ($N_{target}=30$), falling back to lower thresholds ($N_{safe}=6$ minimum). SAM3 often over-segments, so sub-optimal masks are merged with dominant masks (confidence > $\tau_{merge}=0.8$), cross-category unioned for final masks $\hat{M}_i=\bigcup_{j=1}^L M_{i,j}$.

3D lifting leverages 3DGS volume rendering properties to assign 2D semantics to individual Gaussians. A single Gaussian's rendering contribution weight along a ray:

$$w(r,g_j)=T(r,g_j)\alpha(r,g_j)$$

where $T(r,g_j)$ is accumulated transmittance and $\alpha(r,g_j)$ is opacity. For each Gaussian $g_j$, multi-view semantic responses are aggregated across valid views to compute foreground ($k=1$) and background ($k=0$) global scores:

$$W_k(g_j)=\sum_{I_i\in\mathcal{I}_{sel}}\sum_{r\in\mathcal{R}_i}\mathbb{I}(M_i(r)=k)w_i(r,g_j)$$

Gaussians with foreground score strictly higher than background ($W_1(g_j)>W_0(g_j)$) are assigned to foreground set $\mathcal{G}_{fg}$, otherwise background. This rendering-contribution-weighted aggregation ensures occluded or edge Gaussians receive lower semantic confidence, reducing noise.

Cross-View Background Cropping. For each foreground Gaussian, the conflict ratio of projecting onto background regions across valid views:

$$\rho_g=\frac{N_{conflict}(g)}{N_{val}(g)+\epsilon}$$

If $\rho_g$ exceeds $\tau_{conf}=0.8$, the Gaussian is deemed a geometric artifact and reclassified as background. This leverages dense multi-view consensus to reduce boundary noise—a Gaussian projected onto background in most views is likely background itself.

graph LR
  A[3DGS Scene + Text Query] --> B[Geometric Keyframe Selection
Curvature Eval K=10]
  B --> C[VLM Two-stage Parsing
Label Extraction + 2D BBox]
  C --> D[SAM3 Multi-view Mask Gen
Adaptive View Selection]
  D --> E[3D Lifting
Volume Rendering Weights]
  E --> F[Cross-view Background Cropping
Conflict Ratio Filter]
  F --> G[Multi-view Verification
Out-of-bbox Ratio Filter]
  G --> H[3D KNN Spatial Diffusion
k=40 Neighbors Fill Voids]
  H --> I[3D Gaussian Segmentation Result]
  style E fill:#e3f2fd,stroke:#1565c0,stroke-width:2px
  style C fill:#fff3cd,stroke:#856404,stroke-width:2px

Figure 2: ZeroSplat workflow. VLM parsing (yellow) provides semantic priors; 3D lifting (blue) maps 2D semantics to Gaussian points via volume rendering.

Multi-View Verification and Spatial Refinement. Using keyframe 2D bounding boxes $\mathcal{B}_i$, each foreground Gaussian is verified. Projecting Gaussian $g$ to keyframe $I_i$ yields 2D pixel coordinates $\mathbf{u}_{g,i}$; cross-view valid observation frequency $N_{vis}(g)$ and out-of-bounds frequency $N_{out}(g)$ (visible in viewport but outside bounding box) are computed. The cross-view out-of-bounds ratio $R_{out}(g)=\frac{N_{out}(g)}{N_{vis}(g)+\epsilon}$; if $N_{vis}(g)\geq\tau_{views}=8$ and $R_{out}(g)\geq\tau_{box}=0.8$, the Gaussian is removed. This avoids over-cropping from single-view occlusions or 2D localization errors. Finally, 3D KNN spatial diffusion repairs internal voids from view-dependent occlusions—for each background Gaussian, checking the ratio of foreground Gaussians among k=40 nearest neighbors; if exceeding reliability threshold $\tau_{knn}=0.8$, it becomes foreground. Ablation shows KNN diffusion outperforms fixed-radius search (41.2 vs 39.8 mIoU) as Gaussian density varies drastically across scenes.

Experimental Results

Two new benchmarks: GR-LERF (4 scenes: Ramen, Teatime, Figurines, Waldo, pixel-level) and GR-ScanNet (ScanNet-based, 3D point-level). Evaluated on GR3DGS, single-target R3DGS, and open-vocabulary settings against pixel-based (LangSplat, GOI, ReferSplat) and point-based (OpenGaussian, InstanceGaussian, Dr.Splat, LUDVIG) methods.

Generalized Referring 3DGS Segmentation. ZeroSplat achieves 50.8 mIoU on GR-LERF, substantially outperforming runner-up GS-Grouping (28.3) and LUDVIG (23.8). On GR-ScanNet, 41.2 mIoU, leading InstanceGaussian (24.5) and OpenGaussian (19.8). On Ramen 46.5 (runner-up LUDVIG 25.3), Figurines 52.1 (LUDVIG 23.9). Qualitatively, baselines merge or miss nearby instances in multi-target scenes due to over-smoothed embeddings, while ZeroSplat accurately segments all text-specified targets with significantly higher recall. InstanceGaussian and Dr.Splat exhibit severe over-segmentation; OpenGaussian fails to localize targets effectively.

Method	Type	GR-LERF mIoU	GR-ScanNet mIoU
LangSplat	Pixel	18.6	-
GOI	Pixel	31.6	-
GS-Grouping	Pixel	28.3	-
ReferSplat	Pixel	8.8	-
OpenGaussian	Point	7.9	19.8
InstanceGaussian	Point	9.9	24.5
LUDVIG	Point	23.8	15.1
ZeroSplat (Ours)	Point	50.8	41.2

Table 1: GR-LERF and GR-ScanNet quantitative results. ZeroSplat leads substantially (GR-LERF +19.2, GR-ScanNet +16.7).

Single-target R3DGS. On Ref-LERF, ZeroSplat achieves 32.7 mIoU, outperforming fully supervised ReferSplat (29.2, +3.5) despite being entirely training-free. On Teatime 41.4 (ReferSplat 31.3), Figurines 37.8 (25.7). This proves geometry-guided 3D lifting is effective even in single-target scenarios, outperforming specially trained methods.

Method	Ramen	Teatime	Figurines	Waldo	Mean
GOI	27.1	22.9	16.5	15.7	20.5
ReferSplat	35.2	31.3	25.7	24.4	29.2
ZeroSplat	30.4	41.4	37.8	21.3	32.7

Table 2: Ref-LERF single-target referring segmentation. ZeroSplat is training-free yet outperforms supervised ReferSplat.

Open-vocabulary 3D Segmentation. On ScanNet, ZeroSplat achieves 44.5 mIoU on 19 classes (runner-up InstanceGaussian 40.7), 49.7 on 10 classes. On LERF open-vocabulary object selection, 52.4 mIoU, outperforming most point-based methods. This demonstrates competitiveness on non-referring tasks.

Notably, ZeroSplat performs relatively weaker on Waldo (48.4 mIoU) due to small, dispersed targets where VLM 2D bounding box localization is less precise. But on Ramen, Teatime, Figurines with medium-large targets, it reaches 46.5, 56.3, 52.1 mIoU respectively. On GR-ScanNet, ZeroSplat's advantage extends beyond mIoU to segmentation quality—baselines' over-segmentation produces background noise, while ZeroSplat produces crisp boundaries and high semantic purity, more suitable for downstream robotic manipulation. From a paradigm perspective, pixel-based methods are limited by 2D rendering viewpoints unable to resolve spatial ambiguities using 3D geometry; point-based methods operate in 3D but require per-scene optimization. ZeroSplat combines 3D geometric advantages of point-level understanding with zero-training efficiency, being the only framework effective in generalized referring scenarios.

Efficiency Analysis. ZeroSplat is the only method supporting generalized referring (0/1/N), with no per-scene optimization, zero training time, zero extra feature storage, and ~10 GB peak VRAM (baselines: 20-28 GB). This plug-and-play design provides a practical path for 3D semantic understanding on resource-constrained devices.

Figure 3: GR-LERF qualitative results. ZeroSplat accurately segments multiple targets; baselines merge or miss instances.

Figure 4: GR-ScanNet qualitative results. Baselines over-segment or fail to localize; ZeroSplat isolates queried instances with crisp boundaries and high semantic purity.

Ablation. Baseline mask lifting: 24.7 mIoU; adding VLM spatial anchoring: 38.6 (+13.9), effectively pruning cross-view false positives; KNN diffusion alone: 26.2 (+1.5); combined: 41.2 mIoU, proving global semantic localization and local geometric refinement are highly complementary. Removing semantic label extraction: 24.3 mIoU. Random keyframe selection: 39.6. Removing bounding box filter: 40.1. Fixed-radius search: 39.8 vs KNN 41.2.

像素颜色alpha混合

$$ C(p)=\sum_{i=1}^{|\mathcal{G}_{p}|}c_{i}\alpha_{i}\prod_{j=1}^{i-1}(1-\alpha_{j}) $$

透射率计算

$$ T_{i}=\prod_{j=1}^{i-1}(1-\alpha_{j}) $$

Limitations

First, ZeroSplat depends on 2D VLM and SAM3 prior quality—if these models fail on specific scenes or rare objects, 3D segmentation quality suffers. VLM 2D bounding box localization precision directly affects 3D verification accuracy—overly large boxes retain background Gaussians, overly small ones crop valid target geometry. VLM inference itself has computational overhead, though no per-scene optimization is needed, each query still requires running the VLM.

Second, multi-view geometric constraints may be insufficient in texture-sparse or heavily occluded regions. KNN spatial diffusion can fill some voids but may still miss internal regions under extreme occlusion. The training-free design means performance cannot be further improved through scene-specific optimization, potentially lagging behind optimized methods on certain fine-grained tasks. When many similar objects exist (e.g., identical boxes in a warehouse), VLM semantic label extraction may not distinguish individuals. Cross-view bounding box verification may fail under severe occlusion—if targets are occluded in most keyframes, insufficient valid observations ($<\tau_{views}=8$) prevent confirmation.

Conclusion and Future Work

ZeroSplat introduces GR3DGS and solves it with a zero-feature, training-free framework, lifting 2D VLM priors into 3D via multi-view geometric constraints for intrinsic point-level understanding. It substantially outperforms SOTA across generalized, single-target, and open-vocabulary scenarios—GR-LERF 50.8 mIoU (runner-up 28.3), GR-ScanNet 41.2 mIoU (24.5), Ref-LERF 32.7 mIoU (outperforming supervised methods)—while maintaining plug-and-play efficiency: zero training time, zero extra storage, ~10 GB peak VRAM. GR-LERF and GR-ScanNet benchmarks provide evaluation foundations for future research.

This framework provides a practical path for embodied AI and interactive 3D scene understanding, demonstrating that robust 3D semantic understanding does not require altering the intrinsic scene representation or expensive per-scene optimization, but can be achieved by lifting 2D priors via geometric constraints. This insight is inspiring for future 3D scene understanding research. Future work can explore more efficient VLM inference (e.g., distillation to smaller models), stronger small-target segmentation (e.g., multi-scale feature fusion), and downstream integration with robotic manipulation—precise 3D referring segmentation is a prerequisite for robotic grasping. Extension to dynamic scenes and real-time incremental segmentation is also important; the current framework targets static reconstructed 3DGS scenes, requiring continuous semantic updates in dynamic environments.

Golden Quotes

Robust 3D semantic understanding does not require altering the scene's intrinsic representation—lifting 2D priors into 3D via geometric constraints suffices.

Real-world instructions never guarantee exactly one target—generalized referring (0/1/N) is the true setting for 3D scene understanding.