FreeOcc: Training-Free Embodied Open-Vocabulary Occupancy Prediction

Paper Metadata

Title: FreeOcc: Training-Free Embodied Open-Vocabulary Occupancy Prediction

Authors: Zeyu Jiang (equal contribution), Changqing Zhou (equal contribution), Xingxing Zuo (MBZUAI), Changhao Chen (The Hong Kong University of Science and Technology, Guangzhou; corresponding author)

Links: arXiv:2604.28115 (https://arxiv.org/abs/2604.28115 , v1, 30 Apr 2026, cs.RO, accepted to RSS 2026); project page https://the-masses.github.io/freeocc-web/

Code and data status: fully released. The repository https://github.com/the-masses/FreeOcc ships the complete four-layer pipeline, and the paper's central claims can be checked against concrete code anchors. src/gaussian_mapping.py lines 869-932 implement the ray-aligned anisotropic initialization: the config key frame_gaussians_ray_aspect is exactly the elongation ratio $\gamma$ of Equation 4, and quat_from_z_to_dir_cam rotates each Gaussian's local +Z axis onto the viewing direction. Line 152 defines gaussians_to_occ(), which together with the CUDA kernels under src/gs2occ/localagg_prob/ realizes the probabilistic Gaussian-to-occupancy projection that sustains the reported 25 FPS online rate. Line 1134 holds _init_ov_model(), which loads the default open-vocabulary segmentation stack Trident: SAM ViT-B for mask refinement, CLIP ViT-B/16 for text alignment, and DINO as the visual foundation model. Nothing in the occupancy task itself is learned, so the release is a configuration of existing components rather than a checkpoint download.

Figure 1: The FreeOcc framework (paper Figure 1). Layer 1 is a SLAM backbone that estimates poses and sparse geometry; layer 2 performs geometrically consistent Gaussian updates to build a dense 3DGS map; layer 3 associates open-vocabulary semantics with the Gaussian primitives; layer 4 projects Gaussians into a voxel occupancy field that can be queried online. In the right-hand example, "window" and "chair" are correctly localized by FreeOcc, while the ground-truth labels (red boxes) coarsely assign them to "wall" and "floor".

One-Sentence Summary

FreeOcc is the first training-free framework for open-vocabulary occupancy prediction: it chains SLAM poses, geometrically anchored Gaussian updates, off-the-shelf vision-language semantics, and a probabilistic Gaussian-to-occupancy projection, requires no voxel annotations, no pose ground truth, and no learning stage of any kind, and reaches 31.29/13.86 IoU/mIoU monocular and 34.40/15.84 with RGB-D on EmbodiedOcc-ScanNet, more than double the self-supervised baselines, while transferring zero-shot to the new ReplicaOcc benchmark where every learned baseline collapses to near-zero per-class scores.

Background and Motivation

An embodied agent that has to act in a room cannot live on point clouds alone. Points arrive from depth sensors, structure-from-motion, and SLAM, and they have served robotic perception for decades, but their unstructured, irregularly sampled nature makes downstream reasoning awkward: there is no explicit statement of where space is empty, no stable notion of an object boundary, and no cheap way to ask whether a proposed motion is collision-free. Occupancy maps answer all three. They discretize space into free, occupied, and unknown regions, they give explicit geometric boundaries that collision checking and motion planning consume directly, and they support incremental updates as the agent explores. That is why the field converged on embodied occupancy prediction: estimating a semantic occupancy volume from Gaussians built out of egocentric observations, so that the continuous, render-friendly 3DGS representation and the discrete, planner-friendly voxel representation are two views of the same scene.

The problem is what it costs to get there. Fully supervised occupancy prediction needs large-scale 3D reconstruction followed by manual or semi-automatic voxel annotation, and the resulting models tend to perform well only inside the training distribution. The paper's own numbers make the second half of that sentence concrete rather than rhetorical: when EmbodiedOcc, a method trained on voxel labels, is moved to ReplicaOcc, its per-class scores land at 0.00-0.01, and when GaussianOcc and GaussTR, both self-supervised but still pose-dependent, are moved to the same benchmark, their mIoU is 0.00 and 0.01 respectively. These are not degradations; they are total failures on eight indoor scenes that a human would recognize immediately.

Self-supervised methods lighten the annotation burden and, in recent work, borrow open-vocabulary semantics from vision-language models. But they still assume that camera poses are known during both training and inference, and as learning-based systems they overfit to the specific scenes, viewpoints, intrinsics, and metric scale of their training set. The domain gap has two components that the paper separates carefully: appearance and geometry differ between Replica and ScanNet, and learned models have memorized the training-set camera intrinsics and metric scale. The second component is the more damaging one for a benchmark transfer, because it cannot be fixed by more data of the same kind.

The second motivation is internal to 3DGS-SLAM rather than to occupancy. Systems such as Photo-SLAM, built on ORB-SLAM, and DROID-Splat, built on DROID-SLAM, inherit the localization accuracy and efficiency of classical SLAM pipelines, but they share a structural property: the Gaussian scene representation is optimized independently of the point-based SLAM map. Gaussian parameters are updated to maintain rendering consistency while the SLAM backend enforces geometric consistency, and the two objectives are only loosely coupled. The paper's argument for why this decoupling is expensive is a first-moment argument, developed rigorously in the preliminaries below: volumetric rendering along a pixel ray constrains only the first moment of the color and depth distributions, so many distinct depth-weight configurations produce identical observations. The minimizer of the rendering loss is therefore not an isolated point but a near-manifold in parameter space, and unconstrained Gaussian updates drift along that manifold, progressively eroding the geometric consistency that SLAM had supplied.

FreeOcc's answer is to delete the learning stage entirely. The four-layer pipeline connects SLAM geometry, a continuous Gaussian representation, and vision-language semantics so that occupancy prediction becomes pure inference: no voxel labels, no pose ground truth, no gradient step that serves the occupancy task. This single design choice addresses the supervision cost and the cross-domain generalization problem at once, because there is no fitted parameter left to overfit. The price is equally clear and the authors do not hide it: the performance ceiling is now bound to the quality of the SLAM backbone and the VLM, and the paper prices that dependency component by component in its exploratory experiments.

Positioning the work against the literature makes the novelty sharper. Fully supervised occupancy prediction has strong absolute numbers but an annotation supply chain that does not scale. Weakly supervised and self-supervised variants reduce labels but keep the pose assumption and the learned prior. 3DGS-SLAM gives excellent geometry for rendering yet was never designed to output voxels. FreeOcc sits at the intersection and takes one part from each column without inheriting any column's training requirement, which is why the paper can claim a first: the first solution that deploys open-vocabulary occupancy prediction in real-world settings, where neither pre-recorded trajectories nor ground-truth poses exist.

Preliminaries

The task is embodied semantic occupancy prediction. Given a stream of RGB observations $\mathcal{I}_{1:T}$, the agent must estimate online a global 3D semantic occupancy field $\mathcal{O}_{T}\in\mathbb{R}^{X\times Y\times Z\times C}$, where $(X,Y,Z)$ is the scene volume resolution and $C$ the number of semantic categories, so that every voxel simultaneously encodes geometric occupancy and semantic evidence. This differs from scene completion, which infers occupancy from a single RGB frame: the embodied setting requires the agent to build a globally consistent semantic map incrementally while it actively explores, and consistency across time is what makes the problem hard.

The scene is represented as a set of language-embedded Gaussian primitives $G_{i}=(\bm{\mu}_{i},\mathbf{s}_{i},\mathbf{r}_{i},o_{i},\mathbf{c}_{i},\mathbf{f}_{i})$: the 3D mean $\bm{\mu}_{i}\in\mathbb{R}^{3}$, anisotropic scale $\mathbf{s}_{i}\in\mathbb{R}_{+}^{3}$, rotation $\mathbf{r}_{i}$, opacity $o_{i}$, color $\mathbf{c}_{i}$, and a language-aligned open-vocabulary feature $\mathbf{f}_{i}$. Given intrinsics $K_{1:T}$ and globally consistent poses $\mathcal{T}_{1:T}$, a differentiable rendering operator $F$ produces the rendered image and depth map

$$\big(\hat{I}_{t},\hat{D}_{t}\big)\;=\;F\big(\mathcal{G},K_{t},\mathcal{T}_{t}\big),\qquad t=1,\ldots,T$$

and, with $\theta$ collecting all Gaussian parameters, standard 3DGS-based mapping solves the photometric-plus-depth objective

$$\min_{\theta}\;\sum_{t=1}^{T}\Big(\big\|\hat{I}_{t}-I_{t}\big\|_{2}^{2}+\beta\,\big\|\hat{D}_{t}-D_{t}\big\|_{2}^{2}\Big)$$

The ambiguity argument runs along a single pixel ray $\mathbf{u}$. Under volumetric alpha compositing the rendered quantities are first moments of the compositing weights:

$$\hat{I}(\mathbf{u})=\sum_{k}w_{k}\,\mathbf{c}_{k},\qquad \hat{D}(\mathbf{u})=\sum_{k}w_{k}\,z_{k},\qquad w_{k}\geq 0$$

where the weights $w_{k}$ depend on the ray and on the Gaussian parameters. Because only the first moment is constrained, many different $(w_{k},z_{k})$ configurations yield the same $(\hat{I},\hat{D})$ pair. Linearizing the rendering map around a solution exposes a non-zero perturbation $\delta\theta$ with $J\,\delta\theta=0$, so the solution set is a near-manifold rather than an isolated optimum. This is the starting point for the entire method design: a geometric prior is required to pull the solution off the manifold, and no amount of rendering supervision can do it, because every point on the manifold fits the images equally well.

The evaluation protocol deserves attention because it determines how much of the reported gap is real. EmbodiedOcc-ScanNet follows EmbodiedOcc in reporting global scene IoU and per-class mIoU. Since Occ-ScanNet truncates each scene to 100 frames, which is not enough for a SLAM front end, the corresponding scenes instead use the original ScanNet monocular or RGB-D sequences as SLAM input. Between the SLAM reconstruction and the ground-truth coordinate frame, a Sim(3) or SE(3) transform is estimated from camera centers as correspondences with the Umeyama closed-form solution:

$$\min_{s,\mathbf{R},\mathbf{t}}\;\sum_{i}\big\lVert\mathbf{c}_{i}^{\mathrm{gt}}-\big(s\,\mathbf{R}\,\mathbf{c}_{i}^{\mathrm{slam}}+\mathbf{t}\big)\big\rVert_{2}^{2}$$

The same transform is then applied consistently to Gaussian means $\mathbf{x}^{\prime}=s\mathbf{R}\mathbf{x}+\mathbf{t}$, scales $\bm{\sigma}^{\prime}=s\bm{\sigma}$, and orientations $\mathbf{R}_{g}^{\prime}=\mathbf{R}\mathbf{R}_{g}$. This step only removes the global gauge freedom that any monocular system has; it does not improve or damage the reconstruction itself, and it is applied identically to every baseline in the geometric comparison, which is what makes Table IV a fair measurement.

Method

4.1 Layer 1: SLAM Backbone and Globally Consistent Geometry

FreeOcc keeps the interface to the SLAM system deliberately open, since in principle the framework is compatible with arbitrary SLAM backends. The paper adopts DROID-SLAM for the default configuration because of its global geometric consistency and robustness under monocular input. Unlike feed-forward, model-based SLAM approaches such as MASt3R-SLAM or VGGT-SLAM, DROID-SLAM does not rely on explicit 3D supervision from structure-from-motion pipelines when training its optical-flow network. By jointly optimizing over long temporal windows it produces globally consistent camera poses $\mathcal{T}_{1:T}=\{\mathbf{T}_{1},\dots,\mathbf{T}_{T}\}$ and an accumulated point set $\mathcal{P}_{1:T}=\{\mathbf{p}_{i}\in\mathbb{R}^{3}\}_{i=1}^{N_{T}}$, which together form a stable spatial reference for everything downstream. That global consistency matters more than raw accuracy here: every subsequent module operates in one unified coordinate frame, and drift in layer 1 propagates multiplicatively into layers 2 through 4.

4.2 Layer 2: Geometrically Anchored Gaussian Updates and Ray-Aligned Initialization

To attack the decoupling ambiguity, the paper proposes a geometrically consistent Gaussian update strategy with two components. The first is ray-aligned anisotropic initialization. For a pixel $\mathbf{u}$ in frame $t$, the normalized viewing direction $\mathbf{d}_{t,\mathbf{u}}$ is computed from the intrinsics $K_{t}$, a local rotation $R_{t,\mathbf{u}}$ is defined so that its +Z axis aligns with that direction, and the initial scale is set to

$$\mathbf{s}_{t,\mathbf{u}}=\big(s_{\perp},\,s_{\perp},\,s_{\parallel}\big),\qquad s_{\parallel}=\gamma\,s_{\perp}$$

Every Gaussian is therefore modeled as a thin ellipsoid elongated along the sensor ray, with $\gamma$ a user-controlled elongation ratio exposed in the repository as frame_gaussians_ray_aspect and recommended in the 3-10 range. The prior is cheap but pointed: it compresses exactly the degree of freedom that the first-moment argument showed to be unconstrained, namely extent along the ray. The second component is the geometrically anchored update. Gaussian centers are fixed to the SLAM points $\mathcal{P}_{t}$, turning the mapping problem into a constrained one:

$$\min_{\theta}\;\sum_{t=1}^{T}\Big(\big\|\hat{I}_{t}-I_{t}\big\|_{2}^{2}+\beta\,\big\|\hat{D}_{t}-D_{t}\big\|_{2}^{2}\Big),\quad \text{s.t.}\ \ \bm{\mu}_{t}=\mathcal{P}_{t}$$

Centers no longer participate in the optimization. The rendering loss can only adjust scale, rotation, opacity, and color, and geometric consistency becomes the exclusive responsibility of the SLAM backend. This is a division of labor rather than a compromise: each subsystem optimizes the quantity it was designed to estimate. The two components are abbreviated GAGU (geometrically anchored Gaussian updates) and G-ini (geometry-aware initialization), and the ablation shows them to be the single largest source of gain in the whole paper, improving accuracy and throughput simultaneously because removing center variables from the optimizer shrinks the iteration space.

4.3 Layer 3: Open-Vocabulary Semantic Association

The semantic layer refuses to commit to a fixed label space. A pre-trained open-vocabulary segmentation model extracts per-pixel language-aligned embeddings $\mathbf{z}_{t}(\mathbf{u})\in\mathbb{R}^{D}$ for every frame, and the default implementation is Trident, which combines SAM ViT-B for mask refinement, CLIP ViT-B/16 for text alignment, and DINO as the visual foundation model. Each per-pixel embedding is lifted into 3D using SLAM depth; every lifted point then finds its geometrically anchored counterpart in the current Gaussian map and attaches the language feature to it, producing language-embedded Gaussians (LE-Gaussians). Because the features live on primitives whose centers were fixed by SLAM, the semantics inherit the geometric grounding of layer 2 rather than floating free. At query time an arbitrary text prompt is encoded by the CLIP text encoder and compared against Gaussian features, so localization needs no voxel-level supervision and no vocabulary agreed in advance.

4.4 Layer 4: Probabilistic Gaussian-to-Occupancy Projection

The projection paradigm follows GaussianFormer-2, but semantic aggregation is changed from closed-set category probabilities to language-embedded features. For a query location $\mathbf{x}$, the neighboring LE-Gaussians $\mathcal{H}(\mathbf{x})=\{G_{k}\}_{k=1}^{P(\mathbf{x})}$ are retrieved, each inducing a covariance $\bm{\Sigma}_{k}=R(\mathbf{r}_{k})\,\mathrm{diag}(\mathbf{s}_{k}^{2})\,R(\mathbf{r}_{k})^{\top}$, and each neighbor contributes a spatial support

$$\alpha_{k}(\mathbf{x})=\exp\!\left(-\frac{1}{2}\big(\mathbf{x}-\bm{\mu}_{k}\big)^{\top}\bm{\Sigma}_{k}^{-1}\big(\mathbf{x}-\bm{\mu}_{k}\big)\right)$$

These supports are composed with the probabilistic exclusion rule rather than summed, which is what prevents overlapping Gaussians from inflating occupancy beyond physical plausibility:

$$\alpha(\mathbf{x})=1-\prod_{G_{k}\in\mathcal{H}(\mathbf{x})}\big(1-\alpha_{k}(\mathbf{x})\big)$$

Semantic propagation is formulated as posterior responsibility assignment under a local Gaussian mixture whose mixing weights are the opacities, $\pi_{k}=o_{k}$:

$$p\big(G_{k}\mid\mathbf{x}\big)=\frac{p\big(\mathbf{x}\mid G_{k}\big)\,\pi_{k}}{\sum_{G_{j}\in\mathcal{H}(\mathbf{x})}p\big(\mathbf{x}\mid G_{j}\big)\,\pi_{j}},\qquad p\big(\mathbf{x}\mid G_{k}\big)=\mathcal{N}\big(\mathbf{x};\bm{\mu}_{k},\bm{\Sigma}_{k}\big)$$

Language features are then propagated in expectation and normalized, $\mathbf{f}(\mathbf{x})=\sum_{k}p(G_{k}\mid\mathbf{x})\,\mathbf{f}_{k}$ and $\hat{\mathbf{f}}(\mathbf{x})=\mathbf{f}(\mathbf{x})/\lVert\mathbf{f}(\mathbf{x})\rVert_{2}$. Given a query category set $\mathcal{C}$ whose text embeddings are normalized as $\hat{\mathbf{t}}_{c}=\mathbf{t}_{c}/\lVert\mathbf{t}_{c}\rVert_{2}$, the voxel-text similarity is the open-vocabulary semantic score

$$s(\mathbf{x},c)=\hat{\mathbf{f}}(\mathbf{x})^{\top}\hat{\mathbf{t}}_{c},\qquad c\in\mathcal{C}$$

The module outputs the occupancy probability $\alpha(\mathbf{x})$ together with $s(\mathbf{x},c)$, and semantics are reported only for occupied voxels, which avoids the well-known failure mode of labeling empty space. In the repository this stage is implemented by gaussians_to_occ() and the localagg_prob CUDA kernels; the custom aggregation is what keeps the online rate near 25 FPS instead of falling to the 10.7 FPS measured for the unanchored variant.

flowchart TB
  subgraph L1["Layer 1: SLAM backbone"]
    A[Monocular or RGB-D stream] --> B[DROID-SLAM
globally consistent poses + sparse points]
  end
  subgraph L2["Layer 2: geometrically consistent 3DGS"]
    B --> C[Ray-aligned anisotropic init
s_parallel = gamma times s_perp]
    C --> D[Geometrically anchored update
Gaussian centers fixed to SLAM points]
  end
  subgraph L3["Layer 3: open-vocabulary semantics"]
    E[Trident per-pixel language embeddings
SAM + CLIP + DINO] --> F[Lift with SLAM depth
attach to anchored Gaussian = LE-Gaussian]
    D --> F
  end
  subgraph L4["Layer 4: probabilistic projection"]
    F --> G[Neighbor support alpha_k
probabilistic exclusion composition]
    G --> H[Posterior responsibility pi_k = opacity
expectation-propagate language features]
    H --> I[CLIP text similarity
online open-vocabulary queries]
  end

Flow diagram: the four-layer incremental pipeline reconstructed from Sections IV and V. Only four kinds of object cross layer boundaries, namely poses, points, Gaussians, and features, and every stage updates in streaming fashion with no offline optimization pass.

Experimental Results

The main comparison runs on EmbodiedOcc-ScanNet (Table 1). The self-supervised baselines GaussianOcc and GaussTR receive ground-truth poses and still manage only 10.17/4.34 and 15.63/4.95 IoU/mIoU. FreeOcc reaches 31.29/13.86 monocular and 34.40/15.84 with RGB-D, more than double every baseline metric, with no task-specific training at all. Fully supervised methods remain ahead, RoboOcc at 53.3/44.05 and EmbodiedOcc++ at 52.2/43.60, and the paper is careful not to claim otherwise. It does raise two measurement caveats. First, EmbodiedOcc-ScanNet merges a large number of object categories into coarse labels such as "objects" and "furniture", which systematically disadvantages an open-vocabulary model that was never trained on that taxonomy. Second, the ground-truth labels themselves can depart from the visual evidence: in Figure 1 a window is labeled wall and a chair is labeled floor, so a prediction that better matches what the camera actually sees is penalized for it.

Method	Supervision	IoU	floor	wall	chair	sofa	mIoU
EmbodiedOcc++	voxel labels	52.2	27.9	43.9	49.0	59.2	43.60
RoboOcc	voxel labels	53.3	21.94	44.57	51.28	63.09	44.05
GaussianOcc	poses	10.17	3.81	2.53	3.84	9.90	4.34
GaussTR	poses	15.63	1.20	4.29	4.52	10.95	4.95
FreeOcc (mono)	none	31.29	3.16	16.14	19.66	23.43	13.86
FreeOcc (RGB-D)	none	34.40	6.56	21.69	21.02	23.61	15.84

Table 1: Main results on EmbodiedOcc-ScanNet, excerpted from paper Table II. IoU and per-class mIoU are percentages. Neither FreeOcc configuration uses annotations or pose ground truth.

Zero-shot generalization is tested on ReplicaOcc, the benchmark this paper introduces (Table 2). ReplicaOcc is built from the Replica sequences released with NICE-SLAM, uses 0.08 m voxels, carries 44 semantic categories per scene to preserve semantic diversity, covers 8 scenes, and is test-only. Its mIoU is reported over the 8 categories shared with EmbodiedOcc-ScanNet so the two benchmarks stay comparable. Construction follows prior embodied occupancy benchmarks in three stages: back-projecting valid depth pixels with a fixed pixel stride of 4 into world points that inherit per-pixel labels, quantizing them into 0.08 m voxels whose label is decided by majority voting, then densifying into an axis-aligned global grid where each cell takes the label of its nearest sparse voxel within one voxel distance. Observability is computed separately by fusing depth-frustum consistency over frames sampled with stride 2; voxels outside the mask are excluded as unknown with label 255, and observable but unlabeled voxels count as known free space with label 0. Depth is truncated at a maximum range of 10 m.

Against that ground truth the learned methods collapse completely. EmbodiedOcc scores 0.00-0.01 on every class, GaussianOcc is zero everywhere, and GaussTR manages 0.10 on wall alone. FreeOcc transfers without any adaptation to 46.81/16.93 monocular and 55.65/20.90 with RGB-D. The paper attributes the collapse to the two domain gaps identified earlier, appearance and geometry on one side, memorized intrinsics and metric scale on the other, and the second is the one no amount of additional ScanNet training would repair.

Method	Supervision	IoU	ceiling	wall	chair	sofa	table	mIoU
EmbodiedOcc	voxel labels	22.91	0.00	0.00	0.00	0.01	0.01	0.00
GaussianOcc	poses	8.71	0.00	0.00	0.00	0.00	0.00	0.00
GaussTR	poses	15.01	0.00	0.10	0.00	0.00	0.00	0.01
FreeOcc (mono)	none	46.81	19.38	38.23	15.11	25.07	16.46	16.93
FreeOcc (RGB-D)	none	55.65	17.80	44.33	20.76	34.90	24.48	20.90

Table 2: Zero-shot generalization on ReplicaOcc (paper Table III). Per-class scores of learned baselines collapse to near zero, while FreeOcc transfers with no adaptation.

Figure 5: Representative scenes with similar geometric layout from EmbodiedOcc-ScanNet and ReplicaOcc (paper Figure 5). EmbodiedOcc-ScanNet contains 11 semantic categories while ReplicaOcc provides 44 categories per scene, which is what makes the transfer a genuine test of open-vocabulary coverage rather than of label-set overlap.

Geometric consistency is measured separately, and this is the cleanest evidence for the layer-2 claim. Six 3DGS-SLAM systems and FreeOcc are put on the same ruler: every method's Gaussian map goes through the identical alignment procedure and the identical occupancy conversion before geometric IoU is compared. Monocular FreeOcc averages 39.34 against 22.34 for the next-best DROID-Splat, an improvement of 76.1%; with RGB-D it is 45.24 against 29.37, an improvement of 54.0%. The margin is not confined to one scene type either: on Replica FreeOcc scores 46.81 monocular where Photo-SLAM reaches 25.03, and on ScanNet-mini 31.87 where MonoGS reaches 14.83. Since anchoring the centers is the only structural difference being tested, the result isolates the ambiguity argument from everything else in the pipeline.

Setting	Method	Replica	ScanNet-mini	Average
Monocular	Photo-SLAM	25.03	15.29	20.16
	MonoGS	29.50	14.83	22.17
	DROID-Splat	26.27	18.41	22.34
	FreeOcc (mono)	46.81	31.87	39.34
RGB-D	SplaTAM	31.11	17.91	24.51
	GS-ICP	28.95	21.22	25.09
	Photo-SLAM	36.97	16.29	26.63
	RTG-SLAM	35.84	18.46	27.15
	MonoGS	38.97	19.58	29.28
	DROID-Splat	34.48	24.26	29.37
	FreeOcc (RGB-D)	55.65	34.82	45.24

Table 3: Geometric IoU of 3DGS-based SLAM backbones for occupancy prediction on ReplicaOcc and EmbodiedOcc-ScanNet-mini (paper Table IV). All Gaussian maps are aligned and converted with identical procedures before comparison.

The component ablation splits the layer-2 gain further. Removing both GAGU and G-ini leaves 19.88/10.53 monocular at 10.7 FPS. Adding GAGU alone lifts this to 31.20/12.06 and raises the frame rate by a factor of 2.5, which is the counter-intuitive part of the story: constraining the optimization makes it faster because center variables leave the iteration. Adding G-ini on top reaches 39.05/15.40 with almost no further throughput cost. The RGB-D row behaves identically, 27.98/11.20 at 8.8 FPS, then 40.18/16.03 at 25.0, then 45.03/18.37 at 24.6. Precision and efficiency move together here rather than trading off, which is unusual enough to be worth stating plainly.

Ablation setting	Monocular IoU / mIoU / FPS	RGB-D IoU / mIoU / FPS
without GAGU, without G-ini	19.88 / 10.53 / 10.7	27.98 / 11.20 / 8.8
GAGU only	31.20 / 12.06 / 26.8	40.18 / 16.03 / 25.0
GAGU + G-ini (full)	39.05 / 15.40 / 25.3	45.03 / 18.37 / 24.6

Table 4: Component ablation (paper Table V), averaged over ReplicaOcc and EmbodiedOcc-ScanNet. GAGU denotes geometrically anchored Gaussian updates, G-ini geometry-aware initialization.

The gap analysis in paper Table VII is the most honest part of the experimental section. Under the RGB-D setting, replacing estimated poses with ground-truth poses raises IoU from 34.40 to 45.06 and mIoU from 15.84 to 21.34, which quantifies how much of the remaining distance to supervised methods is attributable to pose alignment. Replacing the open-vocabulary semantic module with a closed-set segmentation model, DVEFormer whose 40 classes are manually mapped onto the 11 occupancy categories of EmbodiedOcc-ScanNet, keeps IoU at 34.39 but lifts mIoU to 20.42. Combining ground-truth poses with closed-set semantics reaches 45.03/27.39. The conclusion the authors draw is specific: the gap comes primarily from pose alignment and semantic category assignment, not from the occupancy geometry itself, and per-class IoU still trails supervised methods on sofa, furniture, and other objects, so semantic occupancy supervision remains valuable for maximizing mIoU on a fixed-label benchmark.

Component substitutability is probed in paper Table VI under the monocular setting, and the results split cleanly by layer. Swapping the SLAM backbone for recent end-to-end systems improves both metrics: MASt3R-SLAM reaches 33.80/15.66 at 18.1 FPS, and VGGT-SLAM 33.09/15.90 at 45.17 FPS, both above the default DROID-SLAM's 31.29/13.86 at 25.30. The framework therefore benefits from stronger geometric estimation with no change to the occupancy pipeline. Swapping the semantic module is a different story: SEEM holds IoU at 31.18 but drops mIoU to 8.35, and DINOv2 gives 31.59 with mIoU 8.18. Geometric occupancy estimation is stable under substitution while semantic quality is highly sensitive to the open-vocabulary module, which tells a practitioner exactly where to spend integration effort.

Variant	IoU	mIoU	FPS
Default (DROID-SLAM + Trident), mono	31.29	13.86	25.30
MASt3R-SLAM backbone	33.80	15.66	18.10
VGGT-SLAM backbone	33.09	15.90	45.17
SEEM semantic module	31.18	8.35	30.26
DINOv2 semantic module	31.59	8.18	24.93

Table 5: Influence of individual components on EmbodiedOcc-ScanNet under the monocular setting (paper Table VI). Backbone swaps move geometry and semantics together; semantic-module swaps move semantics only.

Open-vocabulary coverage is validated quantitatively in paper Table VIII by sorting all ReplicaOcc categories by occurrence frequency and reporting mIoU over the top-$K$ of them. Top-10 gives 31.06, and expanding the vocabulary to top-20, top-30, and top-40 yields 23.02, 16.57, and 12.01. The gradual decay is what one should expect: low-frequency categories correspond to smaller objects, partial observations, and visually ambiguous regions, all of which are harder without task-specific semantic occupancy supervision. Reporting the decay curve rather than a single number is the right call, because a lone top-10 figure would overstate what the system can do on a 44-class scene.

Figure 2: Qualitative occupancy comparison (paper Figure 2). Panel (A) compares against learned methods on scene0470 and room2, where the baselines produce fragmented or nearly empty maps on ReplicaOcc. Panel (B) compares against the two most geometrically accurate 3DGS-SLAM systems on scene0006 and office0, where anchored updates give more complete and consistent structure at object boundaries and thin elements.

Figure 3: Open-vocabulary queries on ReplicaOcc (paper Figure 3). Small objects such as "basket" and "clock", low-light categories such as "indoor plant", and semantically ambiguous ones such as "picture" are all localized directly from the occupancy map by text query.

Figure 6: Visualization of all eight ReplicaOcc scenes (paper Figure 6). Because the true Replica scenes nearly touch the ceiling, transparency is reduced to 0.3 throughout these visualizations.

Real-world deployment is reported in Appendix VIII and is where the training-free claim is actually cashed in. An Intel RealSense D435i captures synchronized RGB and depth at 1920x1080, with depth registered to the RGB frame and raw sensor units converted to metric scale using the device-reported depth factor $10^{-3}$. Since no ground-truth labels exist in deployment, semantic cues come from a pre-trained Qwen3-VL model that generates all visible object categories per incoming RGB frame; these per-frame predictions are aggregated across frames into a scene-level open-vocabulary space, which removes the need for manual annotation or a closed label set. The current RGB frame, the aligned depth map, and the labels feed straight into the pipeline, poses are estimated incrementally, and the global representation updates online with no pre-recorded sequence and no offline preprocessing. A short warm-up period at the start of capture lets auto-exposure converge. Everything runs on an Intel Core i9-14900KF with a single NVIDIA GeForce RTX 5090.

The red-and-yellow cup experiment (Figure 4) probes fine-grained discrimination: two visually similar objects are correctly separated and localized by open-vocabulary text queries. The same experiment exposes a resolution wall, and the authors state it without hedging. The standard 0.08 m grid used across embodied occupancy prediction is far too coarse for desktop objects, and good results on the cups appeared only after the individual voxel side length was reduced to 0.005 m, a factor of sixteen. Adaptive dynamic resolution for occupancy grids is named as a highly promising research direction, which is a fair characterization: a uniform grid cannot simultaneously cover a room and resolve a cup.

Figure 4: Real-world red-and-yellow cup experiment (paper Figure 4). FreeOcc correctly localizes and distinguishes visually similar objects from open-vocabulary text queries; the experiment also exposes the insufficiency of 0.08 m voxels for small objects.

The incremental visualizations in Appendix XI add a qualitative argument that the numbers cannot make. On scene0000 in ScanNet the four layers are shown building up together: the point-cloud layer supplies sparse but reliable geometric anchors, the Gaussian layer densifies observed regions and preserves surface appearance, the semantic layer attaches language-aligned features to reconstructed structure, and the occupancy layer converts accumulated geometry and semantics into voxels. Semantic and occupancy maps stay aligned with the underlying point cloud and Gaussian maps throughout, which supports the claim that open-vocabulary predictions are geometrically grounded rather than inferred independently from single 2D frames. An outdoor real-world sequence with irregular geometry, larger depth variation, and more complex appearance still yields coherent multi-layer maps, suggesting the pipeline is not tuned to indoor corridors and living rooms alone.

Limitations

The authors state two limitations. First, occupancy map quality is fundamentally bound to the robustness of the SLAM backbone: accumulated drift or imperfect data association erodes long-term geometric and semantic consistency, and because layer 1 defines the coordinate frame for layers 2 through 4, its errors cannot be corrected downstream. The improvement direction they identify is to feed geometry and semantic cues derived from the occupancy representation back into the SLAM factor graph as optimization targets, closing a loop that is currently strictly feed-forward. Second, current VLMs produce temporally inconsistent semantic predictions, particularly between consecutive frames with high co-visibility, and that inconsistency enters the semantic association of Gaussian primitives as noise. Confidence-aware feature filtering or explicit temporal consistency constraints are left to future work; neither is present in the released pipeline.

We add three independent judgments. First, ReplicaOcc consists of 8 scenes drawn from the visually synthetic Replica renders, which limits its diversity as evidence of cross-domain generalization. The all-zero performance of learned baselines is striking, but part of it stems from intrinsics and metric-scale shift, an engineering-fixable factor; the paper does not attempt scale recalibration of the baselines before comparison, so the strength of the conclusion that learned methods cannot generalize slightly exceeds what the experimental design supports. A recalibrated GaussianOcc would be the informative control, and its absence is the main gap in an otherwise thorough evaluation. Second, absolute mIoU remains far below fully supervised methods, 15.84 against 44.05, and the gap analysis shows that closed-set semantics alone lift mIoU to 20.42. The cost of open vocabulary is thus quantified clearly, but the headline capability and the benchmark metric measure different things: on a fixed taxonomy an open-vocabulary system is structurally penalized, and the two readings should not be blended. Third, the 25 FPS online rate depends on a single RTX 5090 and per-frame Trident inference, with the semantic module as the dominant bottleneck, since the VGGT-SLAM variant reaches 45.17 FPS on the geometry side alone. Real-time behavior of the full four-layer pipeline on compute-constrained mobile platforms remains unverified, and that is precisely where an embodied agent would run it.

Conclusion and Outlook

The value of FreeOcc lies in removing occupancy prediction from the supervision-for-accuracy trade-off curve. No learnable parameter anywhere in the four layers serves the occupancy task: SLAM supplies poses, anchored updates supply geometry, the VLM supplies semantics, and probabilistic projection supplies voxels. Each layer can be replaced independently, and the exploratory experiments price every replacement, so a practitioner knows that swapping in VGGT-SLAM buys 1.8x throughput and 2 points of mIoU while swapping Trident for SEEM costs 5.5 points of mIoU. Doubling the self-supervised baselines on EmbodiedOcc-ScanNet and zeroing out learned competitors on ReplicaOcc together make the case that training-free is not a performance concession but a generalization dividend.

Open problems cluster in three places: closing the loop between SLAM drift and occupancy feedback, for which the authors already sketch a factor-graph formulation; filtering VLM semantics for temporal consistency before they are fused into primitives; and breaking the 0.08 m resolution wall for small objects through adaptive dynamic voxel resolution. A fourth is implied by the deployment section rather than stated: making the semantic layer cheap enough for onboard compute. If these are addressed, training-free occupancy prediction is a plausible default perception substrate for embodied agents, one that does not require retraining for every new environment and starts understanding space the moment a camera is plugged in.

Golden Quote

Once occupancy prediction no longer needs a single annotated voxel, generalization stops being a capability that must be learned and becomes a property the pipeline carries by construction.