Bake It Till You Make It: Ultrafast Spatial Texture-Atlas Splatting

Figure 1 — Textured surfel representation disentangles high-frequency texture details from view-dependent geometry, enabling low primitive counts and high-speed rendering.

1. Background & Motivation

Novel view synthesis (NVS) has shifted toward explicit scene representations for fast, high-quality rendering, falling into two categories:

• Grid-based methods (NeRF, Instant NGP): model scenes using voxel and multiresolution hash grids, sampled via ray-marching.
• Primitive-based methods (3DGS): use sparse volumetric Gaussian splats, efficiently rasterized on GPU.

The persistent limitation of 3DGS: inability to model high-frequency texture variations without a prohibitive number of primitives. Recent work (NeST Splatting, Nexels, Hybrid Latents) introduced spatial hash grids to decouple geometry from appearance. While effective at reducing primitive counts, these methods suffer from slow inference due to computational overhead of hash grid queries in fragment shaders.

World models can now generate 3DGS scenes directly from sparse images, but without an efficient renderer, generated models remain impractical for edge devices or latency-sensitive applications. BITYMI targets this rendering efficiency bottleneck.

2. Core Method

BITYMI's core innovation is a decoupled radiance representation splitting the rendering into two complementary parts:

Low-frequency geometry and view-dependent appearance are modeled with 2D surfels; high-frequency, view-independent textures are represented via a spatial hash grid that is baked into a compact texture atlas.

Figure 2 — Method overview. During training, spherical color models capture per-surfel view-dependent appearance, while the spatial hash grid captures high-frequency texture residual.

2.1 Decoupled Radiance Representation

Each surfel's final color has three components: base color, view-dependent color (Spherical Voronoi), and texture residual. The total rendered color:

$$\mathbf{C}(\mathbf{x}) = \mathbf{f}_{SV}(\mathbf{d}) + \boldsymbol{\rho}(\mathbf{x})$$

where $\mathbf{f}_{SV}$ is the Spherical Voronoi (SV) term encoding view-dependent low-frequency appearance, and $\boldsymbol{\rho}(\mathbf{x})$ is the texture residual produced by the spatial hash grid, encoding high-frequency, view-independent color detail.

Figure 3 — Color decomposition (train scene). Left to right: composited final image, SV term $\mathbf{f}_{SV}$, positive part of texture residual $\boldsymbol{\rho}$, and negative part. The two residuals add high-frequency detail that per-primitive SV cannot represent.

2.2 View-independent Texture Grid

During training, a multiresolution hash grid $E_\theta$ with a compact MLP decoder $f_\phi$ is used. At each fragment location $\mathbf{x}$, features are trilinearly interpolated from each hash grid level, concatenated, and passed through the MLP. Residual values are unbounded (allowing positive and negative offsets), increasing representational capacity and reducing the burden on surfels.

Hash grid config: 4 levels, 4D features per level, $\log_2$ resolutions 8–11; decoder is a 2-hidden-layer MLP with width 16.

2.3 Texture Baking

Evaluating the hash grid and MLP during rendering introduces a computational bottleneck. After training, the view-independent hash grid output is evaluated across each surfel and baked into a global RGB texture atlas for fast inference.

Each surfel is assigned an anisotropic UV-grid resolution $\mathbf{r}_g = (r_x, r_y)$ matching the hash grid's Nyquist rate:

$$r_{g,a} = \mathrm{clamp}\!\left(2^{\lceil \log_2(4es_{g,a}/\delta) \rceil},\; r_{\min},\; r_{\max}\right)$$

where $s_{g,a}$ is the surfel scale on tangent axis $a$, $e = 4\sigma$ is the UV extent, $\delta$ is the finest hash voxel size. Resolution is bounded between $r_{\min}=4$ and $r_{\max}=64$. Anisotropic dimensions save ~30% atlas area on long, thin surfels.

The atlas is packed using a Shelf-First-Fit-Decreasing algorithm. Texel-center coordinates prevent boundary seams:

$$u = \frac{i+0.5}{r_x} \cdot 2e - e, \quad v = \frac{j+0.5}{r_y} \cdot 2e - e$$

2.4 Quantization

Instead of dense FP16, BC7 Block Compression reduces memory to 1 byte/texel. A scalar range $[\mu - 6\sigma, \mu + 6\sigma]$ is computed via bootstrap sampling; a single global (offset, scale) pair is applied as multiply-add at inference. No perceivable quality loss.

2.5 Sparsity Optimization

Two regularizers further reduce primitive count:

• Semi-transparency penalty: penalizes semi-transparent surfels, pruning insignificant ones.
• Falloff regularizer: pushes per-primitive Beta parameter $\beta_i$ toward smaller, flatter values, self-activated by local photometric error:

$$\omega(p) = \exp\!\big(-\gamma\, \|\mathbf{C}(p) - \hat{\mathbf{C}}(p)\|_1\big)$$

Figure 4 — Per-pixel overdraw heatmap. As the falloff regularizer strengthens, kernels flatten and fragment overdraw drops significantly.

flowchart TB
    subgraph Train["Training Pipeline"]
        A[Multi-view Images] --> B[2D Surfels
geometry + view-dep SV]
        B --> C[Spatial Hash Grid
high-freq texture residual]
        C --> D[MLP Decoder f_phi]
        D --> E[Loss + Sparsity
+ Falloff Regularizer]
        E --> B
    end
    subgraph Bake["Baking (one-time)"]
        E --> F[Evaluate Hash Grid
per surfel texel]
        F --> G[BC7 Texture Atlas
1 byte/texel]
    end
    subgraph Infer["Inference (real-time)"]
        G --> H[GPU Texture Mapping]
        B --> H
        H --> I[4K 60fps Render]
    end
    style C fill:#e0e7ff,stroke:#2563eb
    style G fill:#fef3c7,stroke:#d97706
    style I fill:#dcfce7,stroke:#16a34a

3. Experimental Results

Evaluated on Tanks and Temples and Mip-NeRF 360, compared against 3DGS, 2DGS, Beta-Splatting, NeST-Splatting, Hybrid Latents, BBSplat, Nexels, and FastGS.

* Methods benchmarked on RTX 4090 with cuda.Event timing (10 warm-up + 200 timed frames).

Key findings:

• Fewest primitives: BITYMI (falloff) uses only 0.11M points on T&T — 13.6× fewer than 3DGS, 2.2× fewer than FastGS.
• Best LPIPS: 0.157, the lowest among all methods, indicating highest perceptual quality.
• Speed-quality balance: Ours (falloff) achieves 1094 FPS, approaching FastGS's 1173 FPS, but with LPIPS reduced from 0.210 to 0.164.

On Mip-NeRF 360, Ours (falloff) uses 0.14M points at 664 FPS vs 3DGS's 2.7M points at 134 FPS — 19× fewer primitives, 5× faster.

Cross-Platform Performance

Ablation Study

View-dependent feature ablation (bonsai): replacing Spherical Voronoi (SV) with Spherical Harmonics (SH deg 3) and Spherical Betas (SB, $K=2$):

4. Main Contributions

• Decoupled radiance representation: First to fully separate high-frequency texture from view-dependent geometry in a surfel framework, using spatial hash grid for the former and 2D surfels for the latter.
• Baked texture atlas: One-time baking of hash grid + MLP output into BC7-compressed texture atlas, enabling 1 byte/texel extreme compression with GPU hardware texture mapping at inference.
• Sparsity optimization: Semi-transparency penalty + error-gated falloff regularizer reduce primitives to 0.11M (T&T), an order of magnitude fewer than 3DGS.
• Cross-platform real-time rendering: Same atlas bundle achieves 800+/250+/90+ FPS on RTX 4090, MBP M3, and Galaxy S24 respectively.

5.

体渲染颜色积分

$$ C=\sum_{i=1}^{N}c_{i}(x)\sigma_{i}\prod_{j=1}^{i-1}(1-\sigma_{j}) $$

Limitations & Future Work

Author-stated limitations:

• Periodic hashMLP training strategy accelerates training, but convergence analysis and performance optimization remain open.
• Per-fragment hash-grid forward/backward passes become a bottleneck under high overdraw, affecting training time competitiveness.
• Compressed texture atlas still demands substantial memory — future work will explore generating smaller atlases via learned mapping to fixed-size atlases.

Analysis:

The method achieves a breakthrough in inference speed, but training-phase hash grid query overhead remains unresolved. BC7 compression reduces memory to 1 byte/texel, but total atlas size grows with scene complexity, potentially problematic for very large scenes. Future directions include learned fixed-size atlas mapping, more efficient texture coordinate computation, and explicit overdraw reduction loss informed by per-pixel loss.

6. Conclusion

The core idea of BITYMI is to use a neural network (spatial hash grid + MLP) to capture high-frequency texture details during training, but at inference time, not run the neural network at all — instead, the trained output is baked into a GPU-native BC7-compressed texture atlas. This returns rendering to the GPU hardware texture mapping fast path, eliminating per-fragment hash query overhead.

Experiments show that on Tanks and Temples, the method uses only 0.11M primitives (13.6× fewer than 3DGS) with the best LPIPS of 0.157, while achieving 1094 FPS — approaching the fastest method FastGS. On Mip-NeRF 360, it achieves 19× primitive reduction and 5× speedup. The same atlas bundle renders in real-time on RTX 4090, MacBook Pro M3, and Galaxy S24.