TOM-GS: Editable Video Representation via Temporal Opacity Modulation of Static 3D Gaussians

TOM-GS: Editable Video Representation via Temporal Opacity Modulation of Static 3D Gaussians

Paper: TOM-GS: Editable Video Representation via Temporal Opacity Modulation of Static 3D Gaussians
Link: arXiv:2607.22717　|　Benchmark: DAVIS　|　Pose Estimation: AnyCAM

One-line summary: Forsakes complex spatial deformations in favor of regular 3D Gaussians with continuous temporal opacity, assigning learnable temporal mean and scale to each Gaussian so static components fade in/out smoothly. Static geometry naturally supports editing; outperforms prior editable video representations in fidelity on DAVIS.

Background and Motivation

Decomposing video into structured continuous representations is critical for advanced video processing and editing. Implicit Neural Representations (INRs) encode spatiotemporal signals in network weights but their black-box nature makes controlled editing difficult. 3D Gaussian Splatting combines rasterization efficiency with point-based geometry flexibility. But adapting 3DGS to unconstrained videos requires modeling nonlinear temporal dynamics and complex topological changes.

Splatter-a-Video relies on canonical-space Gaussians with deformation fields but supports only restricted transformations. VeGaS introduces Folded-Gaussian distributions for nonlinear dynamics but this expressiveness stems from a specialized folded representation rather than standard 3DGS primitives, making conventional editing operations require reasoning through the folded parameterization. These methods entangle appearance with time-varying geometry — the primitives no longer behave like standard 3D assets.

TOM-GS's core idea: preserve spatial structure unchanged, model scene dynamics exclusively through temporal opacity — trading representational expressiveness for a simpler, more directly editable 3D structure. Each Gaussian gets a learnable temporal mean $\mu_i^\tau$ and scale $\sigma_i^\tau$ so static components fade in/out. Moving objects are represented by sequences of Gaussians lighting up along trajectories. The underlying geometry remains standard 3DGS, compatible with existing 3D editing tools, physics engines, and manual workflows.

Method Details

1. Temporal Opacity Modulation

Standard 3DGS assigns fixed opacity. TOM-GS redefines opacity as a continuous function of time. For Gaussian $i$ with learnable temporal mean $\mu_i^\tau$ and scale $\sigma_i^\tau$, effective opacity at time $t$:

$$\alpha^{\text{b}}_{i}\cdot\exp\left(-\frac{(t-\mu^{\tau}_{i})^{2}}{2(\sigma^{\tau}_{i})^{2}}\right)$$

where $\alpha_i^{\text{b}}$ is the base learnable opacity. This acts as a temporal window — optimizing $\mu_i^\tau$ and $\sigma_i^\tau$ decides when a component appears/disappears. Moving objects are sequences of Gaussians lighting up along trajectories. Since primitives remain standard 3D Gaussians, any spatial edit (scaling, translation, duplication) automatically preserves temporal duration — making the representation inherently editable. This low-capacity window is a strength: replacing it with a neural opacity predictor reduces quality (PSNR 36.50→29.19) and removes explicit temporal parameters needed for editing.

Figure 1: TOM-GS framework. Static 3D Gaussians fade in/out via temporal opacity modulation, modeling dynamics without spatial deformation.

2. Pose Estimation

Since Gaussian spatial parameters are static, the camera must account for all apparent background motion. AnyCAM feed-forward camera tracking recovers per-frame camera-to-world matrices $\{P_t\}_{t=1}^n$ ($P_t\in SE(3)$), adopted directly and held fixed throughout training. Fixed cameras ground every Gaussian in a single shared coordinate space — enabling editing. Ablation confirms importance: identity poses reduce mean PSNR from 36.50 to 31.88.

3. Optimization Strategy

Three key modifications to standard 3DGS optimization: inverse PSNR frame sampling — tracking running PSNR per frame and sampling proportional to inverse PSNR, driving optimization to allocate Gaussians to dynamic segments; disabling opacity resets — standard periodic resets destroy learned temporal windows coupled to $\sigma_i^\tau$; densification cap — halting at 10M Gaussians for stable memory/throughput.

4. Editing Semantics of Temporal Opacity

Each Gaussian's temporal parameters $\mu_i^\tau$ and $\sigma_i^\tau$ are independently controllable — editors can directly adjust appearance timing or duration. Since spatial parameters are decoupled from temporal, spatial edits automatically preserve temporal behavior. This orthogonality enables independent space/time editing. Because all Gaussians are standard 3D primitives, any 3DGS-compatible tool (Blender, physics engines, game engines) can directly manipulate them — compatibility that deformation-field or folded-Gaussian methods cannot achieve.

5. Moving Object Representation

Moving objects are represented by Gaussians lighting up in sequence along trajectories — not a single Gaussian moving, but static Gaussians activating/deactivating by time. Each Gaussian is active when the object passes its position. The temporal window width $\sigma_i^\tau$ controls active duration — fast objects need narrow windows, slow objects wider ones. This "relay" representation, while more discrete than continuous deformation, maintains spatial geometry's static nature and editability.

graph TD
  A[Input monocular video] --> B[AnyCAM pose estimation]
  B --> C[Fixed camera matrices P_t]
  C --> D[Initialize static 3D Gaussians]
  D --> E[Temporal opacity: α·exp(-(t-μτ)²/2στ²)]
  E --> F[Inverse PSNR frame sampling training]
  F --> G[Disable opacity resets]
  G --> H[Densification cap 10M]
  H --> I[Editable video representation]
  I --> J[Compatible with standard 3D editing tools/physics engines]
  style E fill:#f5a623,stroke:#b97316,color:#fff
  style F fill:#4a90d9,stroke:#2c5f8a,color:#fff
  style J fill:#7ed321,stroke:#4a8a14,color:#fff

Experimental Results

Reconstruction Quality

Trained for 80,000 iterations on DAVIS with inverse PSNR sampling and disabled opacity resets. TOM-GS consistently achieves highest PSNR and SSIM among all baselines. Per-pixel error maps show TOM-GS's errors markedly fainter than VeGaS. Model maintains moderate Gaussian counts (2.80M to 10.07M) with high quality.

Ablation Config	Mean PSNR	Impact
Full TOM-GS	36.50	—
Opacity resets enabled	31.54	Destroys temporal windows
Neural opacity modulation	29.19	Removes explicit temporal params
No temporal opacity (static 3DGS)	24.36	Quality collapse
Identity pose matrices	31.88	Loses global anchoring
Uniform frame sampling	35.52	Lower dynamic-segment quality

Temporal Scale Separation

Foreground Gaussians tracking moving objects should remain opaque briefly, while background Gaussians covering static content persist longer. Inspecting learned $\sigma^\tau$ distributions across three DAVIS scenes shows clear separation: foreground Gaussians concentrate at small $\sigma^\tau$ (brief activation), background at larger values. This emergent separation arises without explicit supervision on $\sigma^\tau$, demonstrating temporal opacity modulation naturally encodes the dynamic-static distinction.

Figure 2: Foreground and background Gaussians occupy distinct temporal scales, emerging without explicit supervision.

Gaussian Budget and Scaling

Gaussian Cap	Bear PSNR	Camel PSNR	Observation
1M	Lower	Lower	Steepest gains in low-budget regime
5M	Medium	Medium	Consistent improvement
10M	Highest	Highest	Diminishing returns

Figure 3: TOM-GS compatible with standard 3D editing tools, supporting manual modifications and physics-based simulation.

The temporal opacity modulation uses a Gaussian function:

$$ \alpha^{\tau}(t,\mu^{\tau},\sigma^{\tau})=\exp\left(-\frac{(t-\mu^{\tau})^{2}}{2(\sigma^{\tau})^{2}}\right) $$

The 3D Gaussian set is defined as:

$$ \mathcal{G}=\{(\mu_{i},\Sigma_{i},\alpha_{i},\mathbf{c}_{i})\}_{i=1}^{N} $$

The covariance matrix is decomposed as:

$$ \Sigma_{i}=R_{i}S_{i}S_{i}^{\top}R_{i}^{\top} $$

The time-dependent opacity for the $i$-th Gaussian:

$$ \alpha^{\text{b}}_{i}\cdot\exp\left(-\frac{(t-\mu^{\tau}_{i})^{2}}{2(\sigma^{\tau}_{i})^{2}}\right) $$

Limitations

Analysis: TOM-GS trades expressiveness for editability — no motion parameters means all non-rigid dynamics must be modeled through temporal opacity alone, which may be insufficient for extreme fast deformation or complex topological changes. The low-capacity Gaussian temporal window, while outperforming neural predictors, may not precisely model high-frequency temporal variations. The 10M densification cap may limit quality in complex scenes. Pose estimation relies entirely on AnyCAM feed-forward prediction — pose errors propagate directly, and scale ambiguity and tracking failure in large-motion scenes may affect results. Only validated on DAVIS; generalization to more diverse video types is unevaluated. While compatible with standard 3D tools, the editability of the temporal opacity parameters themselves (manual temporal window adjustment) is not deeply explored.

Conclusion and Outlook

TOM-GS demonstrates that complex spatial deformations or folded distributions are unnecessary for capturing dynamic video content — static 3D Gaussians with temporal opacity modulation suffice for high-fidelity editable video representation. The core innovation is transferring dynamic modeling entirely from the spatial dimension to the opacity temporal dimension, keeping spatial geometry as standard 3DGS assets compatible with existing tools. The emergent temporal scale separation validates temporal opacity as a sufficient mechanism for dynamic-static disentanglement. The optimization heuristics (inverse PSNR sampling, disabled opacity resets) are tailored solutions to temporal opacity learning's unique challenges. The "simple structure + intelligent modulation" paradigm offers broad inspiration for editable media representation — choosing standardized underlying structures and handling complexity through附加 modulation mechanisms is often more pragmatic than introducing complexity at the base level.

The secret of editability lies not in more complex deformations but in simpler structures — keep Gaussians in place, control only when they appear and disappear. A standard 3D asset becomes an editable video through static geometry plus temporal modulation. This is TOM-GS's elegant trade-off.