PAPER DEEP DIVE
GenSplatCodec: Feed-Forward Gaussian Splatting Compression via One-Step Diffusion
Feed-forward 3D Gaussian Splatting (3DGS) enables scalable scene reconstruction without per-scene optimization, yet produces dense Gaussians that are costly to store and transmit. Existing feed-forward Gaussian compression methods formulate decoding as deterministic representation recovery, which becomes inadequate at low bitrates when high-frequency textures and view-dependent appearance are discarded. Although generative models offer a promising alternative, using them as standalone post-processing decouples generation from the transmitted scene structure, thereby compromising cross-view consistency. To address these limitations, we propose GenSplatCodec, a unified feed-forward Gaussian codec that reformulates low-bitrate Gaussian compression as geometry-guided generative decoding. We present a detail-aware feed-forward Gaussian coding scheme within a dual-stream formulation, where the resulting compact Gaussian structural stream is complemented by a lightweight reference appearance stream. We further introduce a geometry-guided one-step generative decoding approach that jointly exploits decoded structural and appearance cues through hierarchical geometry control to reconstruct high-fidelity and view-consistent novel views. Finally, we develop a three-stage optimization strategy that stabilizes the learning of the unified codec and adapts the generative decoder to codec-derived structural and appearance cues. Extensive experiments across multiple datasets demonstrate that GenSplatCodec consistently achieves superior rate-distortion (RD) performance over existing methods.
GenSplatCodec: Feed-Forward Gaussian Splatting Compression via One-Step Diffusion
Paper: GenSplatCodec: Feed-Forward Gaussian Splatting Compression via One-Step Diffusion
Authors: Qiang Hu, Zhenlong Wu, Lei Huang, Zihan Zheng, Xiaoyun Zhang, Wenjun Zhang
Affiliations: Shanghai Jiao Tong University
Links: arXiv:2607.24403
One-Sentence Summary
GenSplatCodec reformulates low-bitrate feed-forward 3DGS compression as geometry-guided generative decoding, using a dual-stream codec (compact Gaussian structural stream + lightweight reference appearance stream) and a one-step geometry-guided generative decoder to achieve comparable reconstruction quality to YoNoSplat's 6.508 MB with only 0.376 MB per scene on DL3DV, a ~17x storage reduction.
Background and Motivation
Immersive multimedia applications (free-viewpoint video, AR/VR, telepresence, digital twins) drive the transition from 2D media to interactive 3D content, requiring photorealistic novel-view rendering under stringent bandwidth, storage, and latency constraints. 3D Gaussian Splatting (3DGS) is attractive for immersive media with explicit representation and real-time rendering. However, conventional neural rendering relies on costly per-scene optimization, limiting scalability.
Feed-forward 3DGS methods (pixelSplat, MVSplat, DepthSplat, AnySplat, YoNoSplat) directly predict renderable Gaussian representations from sparse views without test-time optimization, substantially improving efficiency. However, their primary objective is rendering quality rather than compact transmission—predicted representations are dense and high-dimensional. YoNoSplat requires ~6.5 MB per scene, challenging for bandwidth-constrained scenarios.
Existing compression methods (SOGS, FCGS) reduce redundancy or entropy-code latent features but rely on deterministic reconstruction—high-frequency textures, view-dependent appearance, and weakly observed details are discarded at low bitrates. FCGS reduces DepthSplat from 20.5 MB to 1.76 MB but PSNR drops from 19.35 to 18.23 dB and LPIPS degrades from 0.272 to 0.415—deterministic reconstruction quality degrades severely at low bitrates. A straightforward remedy is generative enhancement as post-processing, but this is decoupled from the 3D coding system—generation doesn't participate in representation design, bit allocation, or structure-aware decoding, and multi-step iterative diffusion is impractical for real-time multimedia.
From the 3DGS compression research perspective, existing methods (HAC, ContextGS) reduce redundancy through pruning, SH reduction, quantization, and context modeling. However, they assume high-quality optimized Gaussians and don't address feed-forward predicted ones. They are all deterministic, unable to recover discarded high-frequency information at ultra-low bitrates. GenSplatCodec therefore reformulates feed-forward Gaussian compression: low-bitrate coding should preserve reliable scene structure and compact appearance evidence, while recovering missing details through geometry-guided generation—shifting from deterministic recovery to geometry-guided generative decoding, with compact coding and generative reconstruction unified within the codec.
Method Details
GenSplatCodec adopts a dual-stream coding architecture: from sparse input views $\mathcal{I}$, producing a compact Gaussian structural stream $B_G$ and a lightweight reference appearance stream $B_R$, $(B_G, B_R)=\mathcal{E}(\mathcal{I})$. The Gaussian stream carries compact 3D structure, visibility, and coarse appearance. The reference stream encodes a selected reference view in VAE latent domain. The decoder uses a geometry-guided one-step generative decoder $\tilde{I}_q=\mathcal{D}(B_G, B_R, \pi_q, \pi_r)$. The key idea: separate structural coding from detail synthesis.
Figure 0: GenSplatCodec overview. Low-bitrate coding preserves reliable scene structure; generative decoder recovers high-frequency details under geometric constraints.
Figure 1: Dual-stream coding architecture. (a) Gaussian structural stream: RD-guided pruning + geometry-attribute decoupled coding producing $B_G$. (b) Reference appearance stream: VAE latent domain quantization and entropy coding producing $B_R$.
Detail-Aware Feed-Forward Gaussian Coding. The prediction network $\Phi_\theta$ infers renderable Gaussians $G=\{g_j\}_{j=1}^M$ with each $g_j=\{\boldsymbol{\mu}_j, \mathbf{s}_j, \mathbf{q}_j, \alpha_j, \mathbf{c}_j\}$. Two complementary feature pathways: the structural pathway aggregates local-global context for center prediction; the detail pathway extracts fine-grained appearance from pretrained DINO via lightweight MLP, injected only into attribute prediction heads. DINO features contain rich local texture and high-frequency appearance, injected directly to avoid being smoothed through multiple attention layers. Asymmetric fusion enhances fine-grained attribute prediction without disturbing cross-view center estimation. Zeroth-order SH for compact RGB. Attributes transformed to coding-friendly domain: $\mathbf{a}_j=[\mathbf{c}_j, \mathrm{logit}(\alpha_j), \log\mathbf{s}_j, \mathbf{q}_j]\in\mathbb{R}^{11}$.
RD-Guided Gaussian Compaction. Dense primitives are inefficient to encode. For the $j$-th Gaussian, visual contribution $v_j$ (opacity and spatial coverage) and attribute coding cost $b_j^a$ (learned entropy model) yield retention priority:
$$p_j^{\mathrm{RD}}=\frac{v_j}{b_j^a+\epsilon}$$
Top-ranked candidates are retained. Centers are quantized and compressed via G-PCC exploiting spatial occupancy structure ($B_\mu$); attributes are Morton-ordered, quantized, and hierarchically entropy-coded ($B_m$ + hyperprior $B_h$). The reference appearance stream encodes a selected reference view to VAE latent domain, quantized and entropy-coded as $B_R$. Reference view selection is based on viewpoint overlap with the query. $B_R$'s bitrate is explicitly included in total rate with no unaccounted side information. Total rate: $R_{total}=R_G+R_R$.
Geometry-Guided One-Step Generative Decoding. Decoding recovers $\hat{G}=Dec_G(B_G)$ and $\hat{\boldsymbol{z}}_r=ED_R(B_R)$. From $\hat{G}$, query RGB anchor and geometry cues: $(\hat{I}_q^g, \hat{D}_q^g)=\mathcal{R}(\hat{G}, \pi_q)$, reference geometry $\hat{D}_r^g=\mathcal{R}_d(\hat{G}, \pi_r)$. Query branch initialized $\boldsymbol{z}_q=E_{vae}(\hat{I}_q^g)$, reference branch uses $\hat{\boldsymbol{z}}_r$. Hierarchical geometry control converts rendered depth to latent-aligned control signals:
$$\mathbf{g}_q=\rho(\nu(\hat{D}_q^g)),\quad \{\Delta\mathbf{H}_q^\ell\}_{\ell=1}^L=C_\phi(\mathbf{g}_q)$$
Injected at each latent stage: $\mathbf{H}_v^\ell=U_\omega^\ell(\mathbf{H}_v^{\ell-1})+\Delta\mathbf{H}_v^\ell$. Query and reference branches processed through shared one-step multi-view generative U-Net; attention layers concatenate tokens for cross-view latent interaction—query aggregates complementary textures from reference, geometry controls regulate interpretation. Final $\tilde{\boldsymbol{z}}_q$ VAE-decoded to $\tilde{I}_q$. Single step, no iterative diffusion sampling.
graph LR A[Sparse Input Views] --> B[Dual-branch Gaussian Pred
Structural + DINO Detail] B --> C[RD-guided Compaction
Retention Priority Pruning] C --> D[Geometry-Attribute Decoupled Coding
G-PCC + Hierarchical Entropy] D --> E[Gaussian Structural Stream B_G] A --> F[Reference Appearance Stream B_R
VAE Latent Quantization + Entropy] E --> G[Decode Gaussians + Render Anchor/Geometry] F --> H[Decode Reference Latent] G --> I[One-Step Generative Decoding
Hierarchical Geometry Control + Cross-view] H --> I I --> J[High-Fidelity Novel View] style C fill:#fff3cd,stroke:#856404,stroke-width:2px style I fill:#e3f2fd,stroke:#1565c0,stroke-width:2px
Figure 2: GenSplatCodec workflow. After dual-stream coding, one-step generative decoding (blue) jointly exploits Gaussian structure and reference appearance.
Three-Stage Optimization. Stage 1: train detail-aware Gaussian prediction without coding constraints ($\mathcal{L}_{ff}=\mathcal{L}_{mse}+\lambda_{ssim}(1-\mathrm{SSIM})$). Stage 2: freeze predictor, optimize compression ($\mathcal{L}_{fid}+\mathcal{L}_{rate}$, $\mathcal{L}_{rate}=\frac{R_m+R_h}{M_r D_a}$). Stage 3: fix coding path, train generative decoder ($\mathcal{L}_{gen}=\lambda_2\|\tilde{I}_q-I_q\|_2^2+\lambda_{lpips}\mathcal{L}_{LPIPS}$). Stage-wise optimization avoids unstable joint optimization.
Experimental Results
Evaluated on DL3DV and RealEstate10K at 224x224, compared with DepthSplat, AnySplat, YoNoSplat and cascaded variants (+SOGS, +FCGS).
Quantitative Comparison. On DL3DV, Ours-Low achieves 19.21 dB with 0.206 MB vs DepthSplat's 19.35 dB with 20.5 MB (~1% storage). Ours-Mid achieves 20.02 dB with 0.376 MB vs YoNoSplat's 20.09 dB with 6.508 MB (17.3x reduction). Ours-High achieves best across all six metrics (20.76 dB, 0.638 SSIM, 0.184 LPIPS, 50.51 FID) with 0.830 MB. On RealEstate10K, Ours-High achieves highest 23.33 dB PSNR, lowest 0.125 LPIPS and 32.45 FID, reducing storage from 6.508 to 0.748 MB (88.5% reduction). Cascaded compression degrades severely—YoNoSplat+FCGS LPIPS from 0.201 to 0.360, FID from 54.51 to 87.89. In contrast, Ours-Mid at lower storage (0.376 vs 1.199 MB) achieves better LPIPS (0.209 vs 0.213) and FID (53.20 vs 58.03), demonstrating generative decoding effectively recovers high-frequency details at low bitrates.
| Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | FID ↓ | Size(MB) ↓ |
|---|---|---|---|---|---|
| DepthSplat | 19.35 | 0.599 | 0.272 | 57.85 | 20.500 |
| YoNoSplat | 20.09 | 0.617 | 0.201 | 54.51 | 6.508 |
| YoNoSplat+SOGS | 20.04 | 0.614 | 0.213 | 58.03 | 1.199 |
| YoNoSplat+FCGS | 18.55 | 0.583 | 0.360 | 87.89 | 1.457 |
| Ours-Low | 19.21 | 0.562 | 0.260 | 65.30 | 0.206 |
| Ours-Mid | 20.02 | 0.619 | 0.209 | 53.20 | 0.376 |
| Ours-High | 20.76 | 0.638 | 0.184 | 50.51 | 0.830 |
Table 1: DL3DV 224x224 quantitative comparison. Ours-Mid achieves comparable quality to YoNoSplat with 17.3x less storage.
Input Sparsity Robustness. Across 2/4/6 context views, GenSplatCodec consistently achieves smallest bitstream with best or near-best quality. 4 views: 0.989 MB / 24.26 dB vs YoNoSplat+SOGS 1.941 MB / 24.24 dB. 6 views: 1.089 MB / 23.83 dB vs 2.830 MB / 23.76 dB. 2 views: 0.376 MB / 20.02 dB vs 1.199 MB / 20.04 dB (31% storage).
| Views | Method | PSNR ↑ | LPIPS ↓ | Size(MB) ↓ |
|---|---|---|---|---|
| 2 | YoNoSplat+SOGS | 20.04 | 0.213 | 1.199 |
| 2 | Ours | 20.02 | 0.209 | 0.376 |
| 4 | YoNoSplat+SOGS | 24.24 | 0.111 | 1.941 |
| 4 | Ours | 24.26 | 0.103 | 0.989 |
| 6 | YoNoSplat+SOGS | 23.76 | 0.113 | 2.830 |
| 6 | Ours | 23.83 | 0.108 | 1.089 |
Table 2: Compression performance under different input view counts.
Figure 3: Rate-distortion curves. GenSplatCodec consistently outperforms cascaded compression, especially under tight storage budgets.
Multi-View Consistency. Compared with MVSplat360 and LatentSplat, GenSplatCodec achieves best SIFT (90.88 vs 78.90/74.11), CLIP (0.941), LPIPS (0.332), MS (0.897), BC (0.935), SC (0.878), confirming cross-view consistent novel views—validating geometry-guided generation over unconstrained post-processing.
高斯参数化
$$ G=\Phi_{\theta}(\mathcal{I})=\{g_{j}\}_{j=1}^{M},\qquad g_{j}=\{\boldsymbol{\mu}_{j},\mathbf{s}_{j},\mathbf{q}_{j},\alpha_{j},\mathbf{c}_{j}\} $$
高斯属性向量
$$ \mathbf{a}_{j}=[\mathbf{c}_{j},\operatorname{logit}(\alpha_{j}),\log\mathbf{s}_{j},\mathbf{q}_{j}]\in\mathbb{R}^{11} $$
编码器输出
$$ (B_{G},B_{R})=\mathcal{E}(\mathcal{I}) $$
LimitationsFirst, the generative decoder depends on pretrained one-step generation prior quality. Insufficient priors for specific scene types (extreme lighting, rare materials, complex transparent objects) limit decoding quality. One-step generation is fast but may be less refined than multi-step diffusion; at very high bitrates, deterministic reconstruction may be superior—GenSplatCodec's advantage is primarily at low bitrates. The prior's generalization determines codec performance on novel scene types.
Second, the reference appearance stream encodes only a single reference view, potentially insufficient for complex scenes. Zeroth-order SH cannot represent view-dependent appearance, relying on generative compensation. Three-stage optimization is stable but lengthy; stage-wise freezing may limit end-to-end optimization potential. Non-differentiable G-PCC geometry coding cannot be directly included in end-to-end rate optimization. Evaluation is primarily at 224x224; higher resolution scaling needs validation. RD-guided pruning retention ratio requires manual tuning.
Conclusion and Future Work
GenSplatCodec reformulates low-bitrate feed-forward Gaussian compression from deterministic recovery to geometry-guided generative decoding, achieving 17.3x compression on DL3DV with comparable quality to YoNoSplat, outperforming cascaded compression across all RD operating points. Multi-view consistency experiments confirm cross-view consistent novel views.
This work demonstrates that unified design of compact structural coding and generative detail recovery outperforms cascaded approaches, providing a practical path for bandwidth-constrained immersive media transmission. Future work can explore multi-reference view encoding, higher-order SH support, differentiable geometry coding for end-to-end rate optimization, more efficient generation priors, higher resolution extension, spatiotemporal compression for video sequences, comparison with V-PCC standards, and real-time inference optimization.
Golden Quotes
Low-bitrate coding should not attempt to retain every detail, but preserve reliable structure and let generation recover the rest—a paradigm shift from deterministic recovery to geometry-guided generation.
The generative decoder is not a post-processing patch but an integral part of the codec—participating in bit allocation and explicitly constrained by transmitted structure.
SOURCE LINKS
