PAPER DEEP DIVE
ATSplat: Compact Feed-forward 3D Gaussian Splatting with Adaptive Token Expansion
3D Gaussian Splatting (3DGS) achieves high-quality novel-view synthesis by optimizing freely placed primitives in 3D and adaptively densifying them in under-reconstructed regions. However, this scene-adaptive capacity allocation is largely lost in existing feed-forward 3DGS methods, which commonly regress Gaussians at input pixels and lift them along camera rays. Such pixel-aligned formulations make the number and placement of primitives depend on image resolution and input viewpoints rather than scene complexity, resulting in dense and often redundant Gaussian sets. We present ATSplat, a feed-forward 3DGS framework that restores the adaptive allocation capability of 3DGS optimization through Adaptive 3D Tokens. ATSplat first lifts coarse patch-level depth and camera cues into sparse 3D anchor tokens, forming a compact scaffold of the scene. Each token is then regressed into local Gaussians with learnable 3D offsets, decoupling primitive placement from input image grids. An Adaptive Token Expansion module predicts a token-level uncertainty score, supervised by rendering error maps, and selectively expands high-uncertainty tokens through learnable expansion layers. This sparse-to-adaptive formulation enables ATSplat to concentrate primitives in challenging regions while maintaining a compact representation. Experiments on two representative datasets, RealEstate10K and DL3DV, show that ATSplat achieves state-of-the-art rendering quality while reducing the number of Gaussians by more than $5.7\times$ compared with dense feed-forward 3DGS methods. From 12 input images at $512 \times 960$ resolution, ATSplat completes reconstruction in less than a second using a single commercial GPU, and renders high-quality novel views at 1136 FPS ($512 \times 960$) with only 311K Gaussians.
ATSplat: Compact Feed-forward 3D Gaussian Splatting with Adaptive Token Expansion
Paper: ATSplat: Compact Feed-forward 3D Gaussian Splatting with Adaptive Token Expansion
Authors: In Cho, Jeonghwan Cho, Mijin Yoo, Gim Hee Lee, Seon Joo Kim
Affiliations: Yonsei University (South Korea), National University of Singapore
Links: arXiv:2607.20417 | Project Page
Code: Not yet released
One-Sentence Summary
ATSplat reintroduces the three principles of 3DGS optimization—sparse initialization, free placement, and adaptive densification—into the feed-forward framework, replacing per-pixel Gaussians with sparse 3D anchor tokens and selectively expanding tokens in difficult regions via learned uncertainty scores, achieving SOTA rendering quality while reducing Gaussian count by over 5.7x.
Background and Motivation
3D Gaussian Splatting (3DGS) achieves high-quality novel-view synthesis by placing anisotropic Gaussian primitives freely in 3D space and optimizing them via differentiable rasterization. Its core strength lies in scene-adaptive capacity allocation: starting from a sparse point cloud, adaptive densification redirects representational capacity toward under-reconstructed regions. This ability to allocate primitives based on scene complexity rather than camera sampling density is what distinguishes 3DGS from implicit representations like NeRF. However, this advantage comes at the cost of expensive per-scene optimization, typically requiring tens of thousands of iterations to converge, limiting deployment efficiency in practical applications.
Feed-forward 3DGS methods emerged to bypass this cost by learning priors that predict Gaussian primitives directly from multi-view images in a single forward pass. The dominant design is the pixel-aligned paradigm: one Gaussian is regressed at every input pixel and lifted along its camera ray. pixelSplat pioneered this approach, MVSplat improved multi-view feature fusion with plane-sweep cost volumes, and DepthSplat further connected multi-view depth estimation with Gaussian prediction using depth priors. Subsequent works extended this to 360-degree scenes, unposed settings, and unconstrained environments. Large Reconstruction Models (LRM) scaled the pixel-aligned paradigm with large transformer backbones. However, all these methods produce primitive counts that grow linearly with image resolution and view count, generating redundant Gaussians in simple regions.
The pixel-aligned paradigm suffers from a fundamental two-fold problem. First, simple flat regions (e.g., white walls, sky) produce large numbers of redundant Gaussians, causing storage and rendering overhead. At 512x960 resolution with 12 views, DepthSplat generates nearly 5.9 million Gaussians, most concentrated in regions that need no fine representation. Second, complex regions (e.g., thin structures, high-frequency textures, occlusion boundaries) cannot receive additional capacity because primitive density is uniformly determined by the pixel grid, leading to significant quality degradation in these areas. In other words, feed-forward methods lost the most critical capability of 3DGS optimization: adaptively allocating representational capacity based on scene complexity.
Several works attempted to compress feed-forward Gaussian counts. FreeSplat reduces primitives through cross-view fusion and redundancy removal for indoor scene reconstruction; Gaussian Graph Network models cross-view relations among Gaussian groups and pools them into a more efficient representation; Fuse-and-Refine aggregates pixel-aligned primitives into canonical 3D space before refinement. These methods mostly perform post-hoc processing on already-reconstructed dense Gaussians. Generative Densification takes a complementary direction by learning a feed-forward densification module. iLRM decouples viewpoint tokens from input images and predicts Gaussians at a lower-resolution grid, achieving 4x compression. However, this is uniform downsampling—while reducing total primitives, it also weakens representation density in complex regions. The essential problem is not density but allocation: the feed-forward framework needs a mechanism to allocate capacity by scene complexity rather than image structure, as 3DGS optimization does.
ATSplat is designed precisely for this purpose. Unlike the above methods, it does not start from dense pixel-aligned Gaussians but from sparse 3D anchor tokens, selectively expanding tokens in difficult regions via learned uncertainty scores to achieve scene-adaptive capacity allocation in the feed-forward framework. Compared to concurrent directions like VolSplat (voxel-aligned), SparseSplat (information-richness sampling), and AnchorSplat (3D anchored feature volumes), ATSplat uniquely constructs a scene-conditioned scaffold of sparse anchors from coarse depth and camera information, and progressively increases capacity in difficult regions through learnable expansion layers. The core idea: high-quality feed-forward 3DGS depends on how representational capacity is allocated, not on how densely it is sampled.
Problem Formulation
Given $V$ posed multi-view images $\{\mathbf{I}_v, \boldsymbol{\pi}_v\}_{v=1}^V$, a feed-forward 3DGS model predicts a set of Gaussian attributes in a single forward pass:
$$f_\theta: \{\mathbf{I}_v, \boldsymbol{\pi}_v\}_{v=1}^V \longmapsto \{(\boldsymbol{\mu}_g, \mathbf{q}_g, \mathbf{s}_g, \alpha_g, \mathbf{SH}_g)\}_{g=1}^G$$
where each Gaussian is parameterized by center $\boldsymbol{\mu}_g$, rotation quaternion $\mathbf{q}_g$, scale $\mathbf{s}_g$, opacity $\alpha_g$, and spherical harmonics $\mathbf{SH}_g$. Pixel-aligned methods determine the center as a predicted depth projected along the camera ray:
$$\boldsymbol{\mu}_v(\mathbf{x}) = \mathbf{o}_v + d_v(\mathbf{x})\,\mathbf{r}_v(\mathbf{x})$$
where $\mathbf{o}_v \in \mathbb{R}^3$ and $\mathbf{r}_v(\mathbf{x}) \in \mathbb{R}^3$ are the camera origin and ray direction at pixel $\mathbf{x}$. This simplifies center prediction to per-pixel depth estimation but ties each primitive to an input pixel, making the total Gaussian count and distribution determined by the input grid rather than the scene. ATSplat treats this as a formulation-level problem: feed-forward models should allocate representation by reconstruction difficulty, not pixel grids.
Method Details
The ATSplat framework recasts three guiding principles of 3DGS optimization—sparse initialization, free 3D placement, and adaptive capacity allocation—as feed-forward operations centered on 3D anchor tokens. The pipeline includes four stages: multi-view encoding, sparse anchor initialization, image-to-3D decoding (with adaptive token expansion), and anchor-to-local-Gaussian regression.
Figure 1: ATSplat framework overview. (a) Multi-view encoder extracts coarse patch features and estimates depth to build a sparse anchor scaffold; the decoder selectively expands tokens in difficult regions via ATE. (b) ATE predicts per-anchor uncertainty scores. (c) Each anchor is regressed into K local Gaussians with 3D offsets relative to the anchor.
Multi-view Image Encoder. The encoder extracts cross-view patch features at coarse resolution. Each view is tokenized into coarse patches using a frozen DINOv2-B backbone (768-dim). Choosing a frozen pre-trained DINOv2 rather than training from scratch leverages rich semantic priors learned from large-scale data while reducing trainable parameters. To inject camera geometry, Plucker raymap embeddings are added to patch tokens—Plucker coordinates parameterize rays in a way that encodes both camera origin and ray direction, providing more comprehensive geometric information than simple positional encoding. A multi-view transformer then flattens patch tokens across all views and applies 12 layers of global self-attention to produce cross-view image features. This design enables the network to aggregate multi-view information during encoding, laying the foundation for subsequent 3D reasoning.
Sparse Anchor Token Initialization. The core innovation is that instead of predicting Gaussians directly on the pixel grid, the framework first constructs a sparse 3D scaffold. For each patch feature $\mathbf{f}_i \in \mathbb{R}^C$, a lightweight MLP predicts patch-level depth $\hat{d}_i$ and unprojects it along the corresponding ray:
$$\mathbf{p}_i = \mathbf{o}_{v_i} + \hat{d}_i\,\mathbf{r}_{v_i}(\mathbf{x}_i)$$
The pair $(\mathbf{p}_i, \mathbf{f}_i)$ defines a 3D anchor token—anchored at 3D position $\mathbf{p}_i$ and carrying the encoded feature $\mathbf{f}_i$. Local 3D context is injected via k-nearest neighbor aggregation using a PointNet-style operator, making each anchor aware of its spatial neighborhood's structural information. This step achieves "sparse initialization": the number of anchors depends on patch count (coarse resolution) rather than pixel count, compressing the primitive scale at the source.
Image-to-3D Decoder. The decoder refines anchor tokens by injecting fine-grained image information through cross-attention. For each view, fine patch features are extracted at twice the coarse resolution, with Plucker raymap embeddings and two per-view self-attention layers. The decoder applies a stack of $L=4$ blocks, each containing an ATE module and cross-attention layers. The ATE module selectively expands anchors associated with under-reconstructed regions, progressively increasing representational capacity where needed. This design achieves "free 3D placement"—anchors are no longer tied to pixel grids but freely acquire information from images through cross-attention.
Anchor to Local Gaussians. Each refined anchor token $\hat{\mathbf{f}}_i$ is regressed into $K=16$ local Gaussian primitives. A lightweight Gaussian head (2-layer MLPs) maps the anchor feature to $K$ sets of Gaussian attributes:
$$\{(\Delta\boldsymbol{\mu}_{i,k},\,\mathbf{q}_{i,k},\,\mathbf{s}_{i,k},\,\alpha_{i,k},\,\mathbf{SH}_{i,k})\}_{k=1}^K = \mathrm{MLP}(\hat{\mathbf{f}}_i)$$
Each Gaussian center is placed relative to the anchor:
$$\boldsymbol{\mu}_{i,k} = \mathbf{p}_i + \Delta\boldsymbol{\mu}_{i,k}$$
This anchor-offset formulation decouples primitive positions from the input pixel grid—the $K$ Gaussians can be freely distributed in the 3D space around the anchor rather than arranged along rays. Ablation studies confirm that this relative offset design outperforms absolute center regression, as coarse anchors provide meaningful scene geometry priors while local offsets preserve the flexibility needed for accurate placement.
Adaptive Token Expansion (ATE). This is ATSplat's core innovation, responsible for "adaptive capacity allocation." The key challenge: in a single forward pass, there are no rendering errors to reference—how can under-reconstructed regions be identified? 3DGS optimization discovers high-error regions through iterative render-compare-densify loops, but the feed-forward framework has only one chance. ATE addresses this by learning a per-token uncertainty score as a proxy for reconstruction difficulty.
At the $l$-th decoder block, a lightweight MLP $g_\phi^{(l)}$ predicts a per-anchor uncertainty score $\hat{u}_i^{(l)}$. Simultaneously, images are rendered from the intermediate Gaussian set $\mathcal{G}^{(l)}$ and D-SSIM error maps $\mathbf{U}^{(l)}$ are computed as supervision signals. The uncertainty loss is:
$$\mathcal{L}_{\mathrm{unc}}^{(l)} = \bigl\|\,\hat{\mathbf{U}}^{(l)} - \mathrm{sg}(\mathbf{U}^{(l)})\,\bigr\|_1$$
where $\mathrm{sg}(\cdot)$ stops gradients so this loss updates only the uncertainty head $g_\phi^{(l)}$ without affecting Gaussian parameters—this is critical, otherwise Gaussian parameters would "cheat" by increasing errors to reduce the uncertainty loss. Based on predicted uncertainty, ATE selects the top-$\rho_l$ fraction of high-uncertainty anchors and expands each selected anchor into multiple tokens (expansion ratio 2) via learnable expansion layers. The selection ratios $\rho_l$ for three ATE modules are 0.5, 0.5, and 0.25. Expanded tokens continue to be refined through cross-attention in subsequent decoder blocks. This process directs both representation and computation toward difficult regions simultaneously.
graph LR A[Multi-view Images] --> B[DINOv2 Encoder
+ Plucker Embeddings] B --> C[Multi-view Transformer
12-layer Global Self-Attention] C --> D[Depth Prediction + Unprojection
Sparse Anchor Init] D --> E[Decoder Block 1
Cross-Attention] E --> F[ATE Module
Uncertainty Prediction + Expansion] F --> G[Decoder Blocks 2-4
Continued Refinement] G --> H[Gaussian Head
Each anchor → 16 Gaussians] H --> I[3D Gaussian Set
311K total] style F fill:#fff3cd,stroke:#856404,stroke-width:2px
Figure 2: ATSplat workflow from sparse anchors to adaptive Gaussians. The ATE module (yellow) is the core innovation, selectively expanding tokens in difficult regions via learned uncertainty.
Training Objective. The final rendering loss combines MSE with a perceptual term: $\mathcal{L}_{\mathrm{render}} = \mathcal{L}_{\mathrm{MSE}} + \lambda_p \cdot \mathcal{L}_{\mathrm{perceptual}}$, where $\lambda_p = 0.5$. Intermediate Gaussian sets are supervised with auxiliary rendering loss $\mathcal{L}_{\mathrm{interm}}^{(l)}$ (using D-SSIM instead of perceptual term for efficiency) and uncertainty loss $\mathcal{L}_{\mathrm{unc}}^{(l)}$. The full objective is:
$$\mathcal{L} = \mathcal{L}_{\mathrm{render}} + \sum_{l=1}^{L}\bigl(\lambda_{\mathrm{int}} \cdot \mathcal{L}_{\mathrm{interm}}^{(l)} + \lambda_{\mathrm{unc}} \cdot \mathcal{L}_{\mathrm{unc}}^{(l)}\bigr)$$
where $\lambda_{\mathrm{interm}} = 0.5$ and $\lambda_{\mathrm{unc}} = 0.1$. Training follows a progressive strategy: first train the base model on RealEstate10K with 2 views using 4 RTX 4090 GPUs for about two days, then train on DL3DV with 6 views using 4 H200 GPUs for two days, and finally fine-tune the high-resolution 10-view model using 8 H200 GPUs for less than two days.
Experimental Results
The paper evaluates on RealEstate10K and DL3DV, reporting PSNR, SSIM, LPIPS, Gaussian count, and inference runtime. RealEstate10K contains mostly indoor home scenes, while DL3DV covers larger-scale indoor and outdoor environments with more complex geometry. All runtimes are measured on a single RTX 3090 unless otherwise specified.
RealEstate10K. At 256x256 resolution with 2 input views, ATSplat achieves 28.46 dB PSNR with only 23K Gaussians, comparable to the state-of-the-art iLRM (28.65 dB, 131K Gaussians) but with 5.7x fewer Gaussians. TokenGS achieves slightly higher SSIM (0.903) but worse LPIPS (0.135 vs 0.118) and uses 262K Gaussians. Inference takes only 0.022 seconds, the fastest among all methods, 14% faster than iLRM. Notably, ATSplat's PSNR is slightly lower than iLRM (0.19 dB), likely because the 2-view setting provides fewer anchors (only 23K), while iLRM uses 131K Gaussians for denser coverage. However, ATSplat's advantage becomes more pronounced with more views and higher resolutions.
| Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | #Gauss. ↓ | Time(s) ↓ |
|---|---|---|---|---|---|
| pixelSplat | 25.89 | 0.858 | 0.142 | 131K | 0.127 |
| MVSplat | 26.39 | 0.869 | 0.128 | 131K | 0.044 |
| DepthSplat | 27.47 | 0.889 | 0.114 | 131K | 0.067 |
| iLRM | 28.65 | 0.900 | 0.110 | 131K | 0.025 |
| ATSplat (Ours) | 28.46 | 0.894 | 0.118 | 23K | 0.022 |
Table 1: RealEstate10K quantitative results at 256x256, 2 views. ATSplat achieves comparable quality with 5.7x fewer Gaussians.
DL3DV. At 256x448 resolution with 2, 4, and 6 input views, ATSplat outperforms prior methods across all settings while using substantially fewer Gaussians. At 6 views, ATSplat achieves 27.28 dB PSNR / 120K Gaussians, compared to DepthSplat's 24.19 dB / 688K Gaussians—a 3.09 dB PSNR improvement with 5.7x fewer Gaussians. iLRM reduces Gaussians via low-resolution prediction (172K) but achieves only 25.60 dB PSNR because uniform downsampling weakens representation in complex regions. SparseSplat uses 150K Gaussians but reaches only 24.20 dB. ATSplat leads in both PSNR and Gaussian count, validating the sparse-to-adaptive paradigm—Gaussian count grows moderately from 40K (2 views) to 80K (4 views) to 120K (6 views), always far below dense methods.
| Method (6 views) | PSNR ↑ | SSIM ↑ | LPIPS ↓ | #Gauss. ↓ |
|---|---|---|---|---|
| MVSplat | 22.93 | 0.775 | 0.193 | 688K |
| DepthSplat | 24.19 | 0.823 | 0.147 | 688K |
| iLRM | 25.60 | 0.830 | 0.168 | 172K |
| SparseSplat | 24.20 | 0.817 | 0.168 | 150K |
| ATSplat (Ours) | 27.28 | 0.868 | 0.138 | 120K |
Table 2: DL3DV quantitative results at 256x448, 6 views. ATSplat leads the runner-up iLRM by 1.68 dB PSNR.
High-Resolution Novel-View Synthesis. At 512x960 resolution with 12 input views, ATSplat achieves 24.85 dB PSNR / 311K Gaussians / 0.677s inference, outperforming optimization-based methods 3DGS (22.87 dB / 553K / >10min) and Mip-Splatting (22.52 dB / 676K / >10min) in both PSNR and SSIM, as well as feed-forward methods DepthSplat (21.33 dB / 5898K) and iLRM (24.35 dB / 1474K), with 19x and 4.7x fewer Gaussians respectively. Rendering speed reaches 1136 FPS. This result is particularly striking: in under 1 second of forward inference, ATSplat surpasses 3DGS which requires over 10 minutes of optimization, while using fewer Gaussians. This demonstrates that the feed-forward framework, with appropriate capacity allocation, can simultaneously achieve improvements in quality, speed, and compactness.
Figure 3: Visualization of expanded tokens. Each expanded token's coordinate is computed as the mean center of its decoded Gaussians, showing concentration in structurally complex regions.
Ablation Studies. For 3D anchor design, the anchor-offset formulation (28.46 dB) outperforms pixel-aligned ray-depth, learnable initial tokens (no geometric prior), and absolute center regression variants. The pixel-aligned variant constrains adaptive expansion flexibility, while the latter two lack scene-specific geometric priors. For ATE selection strategy, uncertainty-guided selection (28.46 dB) outperforms random selection (27.97 dB), farthest point sampling (27.98 dB), and straight-through estimator (27.93 dB), while removing expansion entirely causes a 1.44 dB drop (27.02 dB), proving the necessity of the expansion mechanism. This shows that expansion alone is insufficient—it must be directed to the right regions. Random expansion adds capacity but cannot concentrate it in difficult regions, yielding limited gains.
Figure 4: Predicted uncertainty and selected tokens. High predicted uncertainty aligns with large actual rendering errors, and selected tokens concentrate in these uncertain regions.
Limitations
First, ATSplat's token expansion is unidirectional—it can only add tokens but cannot prune those that become redundant during decoding. The pruning mechanism in 3DGS optimization is another key means of controlling primitive count and quality, and ATSplat has not yet incorporated it into the feed-forward framework. The authors explicitly acknowledge this in the conclusion, noting that a pruning mechanism analogous to 3DGS could be key for improving decoder efficiency and more effective capacity allocation. This limits the efficiency ceiling of capacity allocation—tokens that become unnecessary after expansion continue to consume computational resources.
Second, the current architecture faces scalability challenges when extending to larger-scale scenes, more input views, and higher resolutions. The paper notes the need for more scalable architectural designs for these extreme settings. Additionally, ATSplat currently requires known camera poses, and extending to unposed settings would be an important direction for broadening applicability. While unposed feed-forward methods like NoPoSplat exist, combining pose estimation with adaptive capacity allocation remains an open problem.
Conclusion and Future Work
ATSplat's core contribution lies in reintroducing the three principles of 3DGS optimization—sparse initialization, free 3D placement, and adaptive capacity allocation—into the feed-forward framework. Through sparse 3D anchor tokens, anchor-offset local Gaussian regression, and uncertainty-guided adaptive token expansion, ATSplat achieves SOTA rendering quality while reducing Gaussian count by over 5.7x. Its high-resolution model achieves 1136 FPS rendering at 512x960 resolution with 311K Gaussians, completing inference in under 1 second on a single GPU, surpassing 3DGS which requires over 10 minutes of optimization.
This work sends a clear signal: the quality bottleneck of feed-forward 3DGS lies not in sampling density but in capacity allocation. From dense pixel-alignment to sparse adaptive allocation, ATSplat represents an important paradigm shift in feed-forward 3DGS. Future work incorporating token pruning, scalable architectures, and unposed support could further unlock the potential of feed-forward 3DGS.
Golden Quotes
High-quality feed-forward 3DGS depends on how representational capacity is allocated, not on how densely it is sampled.
What the feed-forward framework lost was not 3DGS's rendering capability, but its ability to adaptively allocate representational capacity based on scene complexity—ATSplat aims to recover exactly this.
SOURCE LINKS



