AniGS: Bridging Rendering and Diffusion Prior for 3D Scene Animation

AniGS: Bridging Rendering and Diffusion Prior for 3D Scene Animation

Paper: AniGS: Bridging Rendering and Diffusion Prior for 3D Scene Animation
Authors: Yen-Chi Cheng, Chen Gao, Chuhan Chen, Tuotuo Li, Rajvi Shah, Ayush Saraf, Changil Kim, Liangyan Gui, Alexander Schwing, Johannes Kopf, Hung-Yu Tseng
Affiliations: UIUC, Waymo, CMU, Meta
Links: arXiv:2607.18539

One-Sentence Summary

AniGS distills motion priors from a pretrained video diffusion model into a 3DGS deformation field through an iterative dataset-model update strategy and composed video-to-video refinement, adding natural ambient dynamics (e.g., vegetation motion) to large-scale outdoor 3D reconstructed scenes while preserving rigid structures.

Background and Motivation

Outdoor environments are inherently dynamic—even without moving subjects, wind and subtle material movements cause foliage, grass, and surrounding clutter to move constantly. This ambient motion is critical for immersive viewing. Although 3D Gaussian Splatting (3DGS) advances enable fast reconstruction and photorealistic novel view rendering, most reconstructed scenes remain static, missing the ambient dynamics that make scenes feel alive. Imagine a virtual real estate tour—walls and floors are accurately reconstructed, but if leaves outside the window are completely still, the entire scene immediately loses realism.

Rapid progress has been made in animating static 3D assets, especially object-centric targets. Animate3D and Bringing Objects to Life use multi-view/video diffusion guidance to produce view-consistent dynamic renderings distilled into time-varying 3D representations. Gaussians2Life animates static 3DGS by generating diffusion guidance and lifting 2D motion into 3D Gaussian deformations. PhysDreamer distills diffusion-predicted motion into physically grounded dynamics by optimizing differentiable simulator (MPM) material parameters. PromptVFX explores text-driven fields for animating open-world 3D Gaussians. However, these methods typically focus on single subjects or small regions and cannot scale to large, cluttered, walkable 3D scenes where motion is distributed across many regions.

4D generation methods (e.g., 4Dfy, Vivid Dream, DreamGaussian4D) combine diffusion priors with explicit 3D representations to generate dynamic content from text/images/videos. Recent works emphasize multi-view and multi-frame consistency. Despite impressive results, these methods primarily target object-centric content or bounded scenes, generating content from scratch rather than animating entire real-world scenes from static reconstructions. Dynamic scene reconstruction methods (e.g., 4DGS, Shape-of-Motion, Geo4D) reconstruct time-varying 3D scenes from dynamic observations but rely on captured time-varying input, whereas our setting starts from canonical static reconstruction without scene-level ground-truth dynamic supervision.

Animating an entire 3D scene faces two core challenges. First, there is no reliable motion prior or supervision providing temporally consistent motion across multiple viewpoints for the whole scene. Video diffusion models (e.g., LTX-Video) excel at turning images into high-quality videos but are not designed to generate multi-view consistent videos. Meanwhile, obtaining scene-level ground-truth dynamic supervision requires multiple calibrated cameras recording concurrently, infeasible for casual capture. Second, the animation must preserve static regions (e.g., walls, floors) that should remain rigid. Video diffusion models often introduce slight motion in static areas, amplified over iterative training into obvious artifacts—a window should remain static while leaves in front sway naturally; if static content is not clearly separated from animated regions, viewing experience degrades noticeably.

AniGS addresses these challenges. It uses a canonical 3DGS for scene geometry and appearance, a time-conditioned deformation field (DCT-based frequency-domain parameterization) for motion, and a pretrained video diffusion model for motion cues. The core innovations are an iterative dataset-model update strategy with incremental viewpoint expansion and a composed video-to-video refinement (ComposedV2V) restricting motion to desired regions. Unlike object-centric animation methods, AniGS handles large navigable spaces with distributed motion. Unlike 4D generation, it animates from existing static reconstructions rather than generating from scratch. Unlike dynamic scene reconstruction, it requires no time-varying input observations. This positioning fills the gap between static reconstruction and dynamic scenes, providing a practical path for virtual tours, AR, and immersive media.

Method Details

AniGS alternates between dataset update and model update phases, bootstrapping scene-level animation from canonical static reconstruction $\mathcal{G}^0$. The deformation field $D_\phi$ produces $\mathcal{G}^t=D_\phi(\mathcal{G}^0,t)$, rendered from arbitrary viewpoints as $\hat{I}_{p,t}=\mathrm{Render}(\mathcal{G}^t,p)$.

Figure 1: AniGS method overview. Left: canonical 3DGS + time-conditioned deformation field (DCT coefficients for offset and rotation). Right: iterative dataset-model update with incremental viewpoint expansion and ComposedV2V refinement.

Dynamic Scene Representation. Time-varying deformation is parameterized on top of canonical static 3DGS $\mathcal{G}^0$. Each Gaussian is parameterized by mean $\mu$, covariance $\Sigma=RSS^{\mathsf{T}}R^{\mathsf{T}}$, color $c$, and opacity $\alpha$. Rendering sorts Gaussians along rays and accumulates via over-compositing: $C=\sum_{i\in\mathcal{N}}c_i\alpha_i\prod_{j=1}^{i-1}(1-\alpha_j)$. Inspired by AmbientGS, per-Gaussian deformations use a compact DCT-based frequency-domain parameterization. A time-varying scalar deformation $v(t)$ is modeled as:

$$v(t)=\sqrt{\frac{2}{K+1}}\sum_{k=1}^{K}\phi_{v,k}\cos\!\Big(\frac{\pi}{2T}(2t+1)k\Big)$$

where $\phi_{v,k}$ are learnable coefficients and $K$ controls basis size. This representation is storage-efficient and smoothly generalizes over time—at any timestamp $t$, $\Delta\mu_i(t)$ and $\Delta r_i(t)$ are recovered via inverse DCT and applied to canonical Gaussians before rendering. DCT frequency-domain parameterization represents periodic motion (e.g., wind-driven foliage swaying) with few coefficients and naturally ensures temporal smoothness, avoiding per-frame parameterization's storage overhead and discontinuities. The video diffusion model uses a rectified-flow formulation: linear interpolation between clean latent $z_0$ and Gaussian noise $\epsilon$: $z_\tau=(1-\tau)z_0+\tau\epsilon$, predicting velocity $v=\epsilon-z_0$.

Dataset Update. Learning scene-wide animation in a single shot is impractical—optimizing all Gaussians and viewpoints simultaneously is computationally prohibitive and often causes OOM errors, while dense ground-truth dynamic supervision with full scene coverage is unavailable. An incremental viewpoint expansion strategy selects $N=4$ camera locations based on region-of-interest segmentation (e.g., trees, flowers), using farthest-point sampling for full coverage, ordered counter-clockwise. Each location considers three viewing directions $\psi\in\{0°,90°,-90°\}$ with fixed FOV=95° for overlap. A view queue is maintained; each dataset update pops one view, renders a clip with the current model, and refines it with ComposedV2V. This incremental strategy provides a curriculum effect—learning simpler views first before gradually expanding to complex ones.

Composed Video-to-Video Refinement (ComposedV2V). Directly running video diffusion on rendered views often fails—diffusion models may introduce slight motion in static areas, amplified over iterations into obvious artifacts. ComposedV2V estimates a dynamic mask $M_0$ from the first frame (using SAM2) and creates a composed video by copying static pixels from the first frame to all subsequent frames:

$$\bar{R}_t = M_0 \odot R_t + (1-M_0) \odot R_0$$

This enforces static region stability—intuitively telling the diffusion model "static areas shouldn't move." The composed video conditions a rectified-flow video diffusion model in VAE latent space: from clean latent $z_0$ and noise $\epsilon$, construct $z_\tau=(1-\tau)z_0+\tau\epsilon$, predict velocity $v_\theta(z_\tau,\tau,l)$ conditioned on text prompt $l$, integrate the reverse process from pure noise, and VAE-decode to refined video $A_{0:T}$.

graph LR
  A[Canonical Static 3DGS] --> B[Deformation Field D_phi
DCT Frequency Domain]
  B --> C[Render Camera-fixed Video]
  C --> D[ComposedV2V Refinement
Static Stabilization + Diffusion Motion]
  D --> E[Refined Video as Training Data]
  E --> F[Model Update
Deformation + Canonical Position]
  F --> B
  G[Incremental View Expansion] --> C
  style D fill:#fff3cd,stroke:#856404,stroke-width:2px
  style B fill:#e3f2fd,stroke:#1565c0,stroke-width:2px

Figure 2: AniGS iterative pipeline. Canonical 3DGS renders video via DCT deformation (blue), ComposedV2V refines (yellow) as training data to update the model, with incremental view expansion.

Model Update. The renderable representation is optimized using the refined dataset. For training view $p_i$ and timestep $t$, rasterized deformed Gaussians render frame $R_t^{p_i}$ supervised by photometric loss:

$$\mathcal{L}_{\mathrm{ani}}=\|R_t^{p_i}-A_t^{p_i}\|_1+(1-\mathrm{SSIM}(R_t^{p_i},A_t^{p_i}))$$

To stabilize optimization and reduce drift artifacts, canonical space regularization enforces canonical rendering close to $t=0$ static appearance:

$$\mathcal{L}_{\mathrm{cano}}=\|R_0^{p_i}-I_0^{p_i}\|_1+(1-\mathrm{SSIM}(R_0^{p_i},I_0^{p_i}))$$

Additionally, score distillation sampling (SDS) loss renders $4m+1$ frame clips, encodes to diffusion latent space, adds noise, and uses pretrained DiT to predict velocity, encouraging rendered animation to follow the video diffusion motion prior:

$$\mathcal{L}_{\mathrm{SDS}}=\mathbb{E}\left[w(\tau)\|v_\theta(z_\tau^{p_i},\tau,l)-(\epsilon-z_0^{p_i})\|_2^2\right]$$

Total loss: $\mathcal{L}=\mathcal{L}_{\mathrm{ani}}+\lambda_{\mathrm{cano}}\mathcal{L}_{\mathrm{cano}}+\lambda_{\mathrm{SDS}}\mathcal{L}_{\mathrm{SDS}}$. Due to memory constraints, SDS gradients backpropagate through only one sampled frame. Learnable parameters: optimizing only $D_\phi$ with frozen canonical attributes is suboptimal—static geometric bias limits deformation expression—so canonical positions $\{\mu_i^0\}$ are additionally fine-tuned for better alignment with diffusion-refined targets, while other attributes (rotation, scale, opacity, color) remain fixed for stability.

Implementation. Canonical static reconstruction: AdamW, 30k iterations, ~1.6 hours. Animation training: LTX-Video as diffusion prior, 8 sampling steps, $T=121$ frames at 1024x1408. Dataset updated every 2500 steps, 70k total iterations, ~7.3 hours (single GPU). Canonical mask via SAM2.

Experimental Results

Evaluated on 5 large-scale outdoor scenes (Fireplace, Garden, Flowers, Forest, Trail), each with ~1.5k-2.0k monocular images, poses estimated via COLMAP. Also evaluated on 5 DL3DV outdoor scenes (Gazebo, Courtyard, Bush, Playground, Park). Baselines: Gaussians2Life and PhysDreamer, both provided the same pretrained static reconstruction.

Quantitative Evaluation. FVD measures animation quality across three diffusion reference distributions. AniGS achieves FVD-LTX average of 415.89 across all 5 scenes, far below Gaussians2Life (902.03) and PhysDreamer (839.46), a reduction exceeding 50%. FVD-Cosmos similarly leads (226.99 vs 630.05/632.52). Per-scene, Forest shows AniGS at 407.86 vs 605.64/550.37; Flowers at 537.14 vs 1154.95/1078.05—this scene has dense complex vegetation where baselines fail at distributed motion. PhysDreamer often produces less realistic dynamics—inducing near-global shifting rather than foliage-specific motion.

Method	FVD-LTX ↓	FVD-Cosmos ↓	FVD-DynamiCrafter ↓
Gaussians2Life	902.03	630.05	824.13
PhysDreamer	839.46	632.52	860.28
AniGS (Ours)	415.89	226.99	495.48

Table 1: Average FVD across 5 outdoor scenes. AniGS leads substantially (>50% reduction) across all reference distributions.

Multi-view Consistency. A 3D reconstruction experiment validates multi-view consistency—at novel timestep $t=21$, multi-view images rendered from 50% training cameras are used to train 3DGS, with PSNR evaluated on validation cameras. AniGS's novel-time PSNR approaches original capture PSNR, confirming multi-view consistent animation.

Scene	Original Capture PSNR ↑	AniGS Novel Time PSNR ↑
Fireplace	27.40	25.94
Garden	28.77	27.13
Gazebo	26.29	24.41
Courtyard	28.35	26.87

Table 2: Multi-view consistency verification. Novel-timestep PSNR approaches original capture, confirming multi-view consistent animation.

User Study. Under 2AFC, participants significantly prefer AniGS for both motion realism and visual quality across all scenes. Motion realism: AniGS preferred over Gaussians2Life 78.3%-91.7%, over PhysDreamer 85.7%-91.7%.

Figure 3: Qualitative comparison. AniGS produces natural dynamics in vegetation while preserving static structures. Baselines confine motion to small regions or produce global shifting.

Ablation. Disabling dataset-model update (generating pseudo-training videos only once) significantly degrades quality—supervision misaligns with the evolving renderer. Removing incremental view expansion further destabilizes training—losing the curriculum effect. Disabling ComposedV2V also hurts—independently generated clips are less temporally consistent across views and introduce conflicting supervision. Optimizing all canonical attributes causes blur and drift; optimizing only deformation is limited by static geometric bias; allowing canonical position fine-tuning provides better alignment so deformation focuses on residual time-varying motion.

Figure 4: AniGS generalization to indoor scenes (clothes, umbrella, plastic bags), demonstrating applicability beyond outdoor vegetation.

Limitations

First, AniGS's animation quality depends on the pretrained video diffusion model's prior quality. If the diffusion model lacks motion priors for specific scene types (e.g., rare vegetation or extreme weather), generated animation may be unnatural. Video diffusion inference has computational overhead—each viewpoint's ComposedV2V refinement requires running the diffusion model, with total training ~7.3 hours. SDS loss memory constraints limit gradient backpropagation to one sampled frame, potentially underutilizing the motion prior. The evaluated animation duration is 121 frames; longer animations' quality retention is unverified. The DCT basis size $K$ requires manual tuning per scene.

Second, the current method primarily targets ambient dynamics (e.g., vegetation motion) with limited capability for complex physical interactions (e.g., collisions, fluid motion). The DCT frequency-domain parameterization suits periodic motion but has limited expressiveness for non-periodic or sudden motions. ComposedV2V protects static regions but slight artifacts may still appear at dynamic/static boundaries. Incremental viewpoint expansion requires manual region-of-interest selection for camera initialization, with room for automation improvement.

Conclusion and Future Work

AniGS proposes scene-level 3DGS animation, distilling video diffusion motion priors into a DCT deformation field through iterative dataset-model update and ComposedV2V refinement, adding natural ambient dynamics to large-scale outdoor scenes. FVD substantially outperforms baselines (>50% reduction) across 5 collected scenes and DL3DV, with user studies confirming significantly better motion realism and visual quality than Gaussians2Life and PhysDreamer. Multi-view consistency experiments confirm 3D-consistent animation. The method also generalizes to indoor scenes (clothes, umbrellas, etc.).

This work demonstrates a practical path from static reconstruction to dynamic scenes, proving video diffusion priors can scale to scene-level animation through iterative bootstrapped supervision. Future work can explore more efficient diffusion inference (e.g., distilling to smaller models or reducing sampling steps), modeling complex physical interactions (e.g., combining physics simulators), and real-time animation generation. Extending to indoor dynamic scenes (e.g., curtains, water ripples) and larger-scale scenes (e.g., city blocks) is also valuable. As video diffusion models continue to improve, swapping in stronger priors may further enhance animation quality.

Golden Quotes

The challenge of scene-level animation is not generating motion, but distributing it across the entire scene while maintaining multi-view consistency—this requires iterative bootstrapping, not one-shot generation.

Static region stability is not a default state but an actively maintained property—ComposedV2V's core is telling the diffusion model "this shouldn't move."