Visual Relocalization from Sparse Views in Aliased and Low-Texture Environments via Novel View Synthesis

Figure 1 — Relocalization challenges in extreme planetary-like environments. The bottom three images overlap, but large viewpoint change and weakly aliased textures pose substantial challenges for 6-DoF pose estimation within the map shown at top.

1. Background and Motivation

Reliable visual relocalization is key for long-term robotic autonomy in GNSS-denied environments (e.g., planetary scenarios), enabling loop closure and mitigating odometry drift. But perceptual aliasing and harsh illumination severely degrade visual place recognition and pose estimation. Planetary rovers' predominantly forward motion limits viewpoint diversity, parallax, and causes perspective-induced appearance changes.

3DGS provides an explicit differentiable representation enabling render-based alignment and robustness to viewpoint changes. But standard photometric 3DGS training is ill-suited for rover outdoor sequences: forward motion → weak multi-view constraints → high reconstruction error; sparse viewpoints limit generalization; wide-baseline revisits challenge naive photometric losses. This work first systematically combines MVS and LiDAR geometric supervision, showing they are complementary and jointly better than either alone.

2. Core Method

2.1 3DGS Submaps

Scene representation $\mathcal{G}=\{\mathcal{G}_1,\ldots,\mathcal{G}_m\}$ of $m$ submaps, each $\mathcal{G}_s=f(\mathcal{P}_s,\mathcal{I}_s,\mathcal{T}_s)$ a set of Gaussians from aggregated LiDAR $\mathcal{P}_s$, $m$ RGB images $\mathcal{I}_s$, camera poses $\mathcal{T}_s\in SE(3)$. Given query $\mathbf{I}_q$, first retrieve candidate submaps by visual place recognition, then estimate 6-DoF pose $\mathbf{T}_q^s$ directly against the Gaussian representation.

Figure 2 — Relocalization pipeline based on structure-preserving 3DGS. RGB + LiDAR grouped into submaps; trained with photometric + strong geometric supervision. Inference: query retrieves top-k submaps, pose estimated by minimizing rendering error against 3DGS.

2.2 Structure-Preserving 3DGS Fitting

Photometric supervision:

$$\mathcal{L}_{\text{photo}}=(1-\lambda_{\text{SSIM}})\mathcal{L}_1+\lambda_{\text{SSIM}}\mathcal{L}_{\text{SSIM}}$$

with $\lambda_{\text{SSIM}}=0.2$.

MVS supervision: MVSAnywhere (MVSA) predicts depth maps. Depth alignment $\mathcal{L}_1$ loss:

$$\mathcal{L}_{\text{MVSD}}=\frac{1}{|\Omega|}\sum_{i\in\Omega}|d_i^{\text{GS}}-d_i^{\text{MVS}}|$$

Normal consistency cosine loss:

$$\mathcal{L}_{\text{MVSN}}=1-\frac{1}{|\Omega|}\sum_{i\in\Omega}{\mathbf{n}_i^{\text{GS}}}^\top\cdot\mathbf{n}_i^{\text{MVS}}$$

Normals from differentiating MVS depth maps. Enhances local surface coherence. But MVS depends on parallax, less reliable under sparse viewpoints/linear trajectories.

LiDAR supervision: symmetric Chamfer loss aligning reconstructed Gaussians with LiDAR:

$$\mathcal{L}_{\text{Ch}}=\mathcal{L}_{\text{acc}}+\lambda_{\text{comp}}\mathcal{L}_{\text{comp}}$$

with accuracy and completeness:

$$\mathcal{L}_{\text{acc}}=\frac{1}{|\mathcal{P}_{\text{GS}}|}\sum_{\mathbf{p}\in\mathcal{P}_{\text{GS}}}\min_{\mathbf{q}\in\mathcal{P}_s}\|\mathbf{p}-\mathbf{q}\|_2,\quad \mathcal{L}_{\text{comp}}=\frac{1}{|\mathcal{P}_s|}\sum_{\mathbf{q}\in\mathcal{P}_s}\min_{\mathbf{p}\in\mathcal{P}_{\text{GS}}}\|\mathbf{q}-\mathbf{p}\|_2$$

Tolerates large density differences, needs no explicit correspondences, fully differentiable. Anchors Gaussians to metrically accurate LiDAR, resolving low-parallax depth. Total loss:

$$\mathcal{L}=\mathcal{L}_{\text{photo}}+\lambda_{\text{MVSD}}\mathcal{L}_{\text{MVSD}}+\lambda_{\text{MVSN}}\mathcal{L}_{\text{MVSN}}+\lambda_{\text{Ch}}\mathcal{L}_{\text{Ch}}$$

flowchart TB
    RGB["RGB images"] --> SUB["submap G_s"]
    LIDAR["LiDAR point cloud P_s"] --> SUB
    POSE["camera poses T_s"] --> SUB
    SUB --> PHOTO["photometric loss L_photo"]
    SUB --> MVSA["MVSAnywhere depth+normals"]
    MVSA --> LMVS["L_MVSD + L_MVSN
(local surface coherence)"]
    SUB --> CHAM["symmetric Chamfer L_Ch
(global metric alignment)"]
    LIDAR --> CHAM
    PHOTO --> TOTAL["total loss L"]
    LMVS --> TOTAL
    CHAM --> TOTAL
    TOTAL --> GS["geometry-aware 3DGS"]
    GS --> REL["6DGS 6-DoF relocalization"]
    style CHAM fill:#dbeafe,stroke:#2563eb
    style LMVS fill:#fef9c3,stroke:#ca8a04
    style REL fill:#dcfce7,stroke:#16a34a

3. Experiments

Vulcano island (Sicily) planetary-analog moon_lake sequence, ~25 min, 1.5 km, synchronized RGB+LiDAR+dGNSS. Nerbstudio; Photo-3DGS 30,000 iter Adam; geometry-aware 20,000 iter, $\lambda_{\text{MVSD}}=0.05$, $\lambda_{\text{MVSN}}=0.1$, Chamfer activated after 2,000 iter linearly to 8,000 reaching $\lambda_{\text{Ch}}=5\times10^{-5}$.

3.1 Reconstruction Quality

LiDAR initialization cuts geometric error 61%; adding Chamfer ~74%. MVS+LiDAR jointly more accurate than either alone. Geometric gains come without compromising photometric fidelity—metric consistency, not photometric quality alone, is key.

3.2 Relocalization Recall

Under relaxed threshold, PnP/3DGS/3DGS+LiDAR recall is tiny (0.16%/6.25%/2.10%); ours reaches 43.2%. Strict threshold: baselines all 0, ours 6.80%. PnP has large translation errors from few reliable correspondences; photometric 3DGS has large rotation errors (visually similar candidates → wrong spatial alignment); ours cuts rotation errors (median <25°) while keeping competitive translation.

Figure 5 — Rendering quality (vertical) vs geometric accuracy (horizontal). Geometric supervision improves metric consistency while preserving photometric fidelity.

4. Main Contributions

• Structure-preserving 3DGS: first to combine MVS depth/normals with LiDAR symmetric Chamfer loss for 3DGS geometry, showing complementarity and joint superiority.
• Planetary relocalization pipeline: visual place recognition retrieves submaps + 6DGS estimates 6-DoF directly against the Gaussian map.
• Metric consistency is key: geometric gains (Chamfer −74%) come without photometric loss; metric consistency, not photometric quality alone, drives quality.
• Large recall gains: (10m,15°) 6.25%→43.2%, rotation median <25°.

5. Limitations and Future Work

Author-stated: 6DGS estimates poses via sparse geometric ray correspondences, not photometric alignment, so renders may show photometric artifacts; local 3DGS inaccuracies near submap boundaries mean ground-truth-pose renders don't perfectly match query images.

Our analysis: only moon_lake single sequence evaluated; generalization to other planetary-analog/real scenes unverified. Strict (2m,10°) recall only 6.80%, so the task remains very hard. MVS supervision relies on MVSAnywhere cross-domain generalization, whose quality on other terrains is unknown. Chamfer weight and activation schedule are empirical. 6DGS's no-photometric-refinement is a design choice; combining iNeRF-style render-and-compare could improve accuracy but adds complexity.

6. Conclusion

For visual relocalization in planetary-like terrains (low texture, aliasing, forward-motion sparse views), this work proposes structure-preserving 3DGS: beyond photometric loss, it first combines MVS depth/normal consistency with a LiDAR symmetric Chamfer loss, systematically showing they are complementary and jointly better—Chamfer error −74%, PSNR up to 25.27, without photometric loss. On Vulcano moon_lake, single-image 6-DoF recall at (10m,15°) lifts from photometric 3DGS's 6.25% to 43.2%, strict (2m,10°) from 0% to 6.80%, rotation median <25°. The core conclusion: metric consistency, not photometric reconstruction quality alone, is the key to high-quality representations and robust relocalization; geometric supervision lets the pose optimizer refine moderately misaligned retrieval candidates.