Skip to content
RobotWorld
Back to Papers

PAPER DEEP DIVE

3D Gaussian Splatting等变性equivariance

E3DGS: Unified Geometric-Photometric Equivariance for 3D Gaussian Splatting via Color-as-Geometry Embedding

3D Gaussian Splatting (3DGS) captures scenes by coupling explicit geometry (position, covariance) with view-dependent photometry (Spherical Harmonics). However, building $\mathrm{SE}(3)$-equivariant architectures on these primitives presents a fundamental representation bottleneck. Color has been treated as a signal rather than a geometric entity, making it nontrivial to unify symmetry across geometry and appearance as the camera frame changes. While translations are handled by relative coordinates, rotations act heterogeneously across attributes: $μ\mapsto Rμ$, $Σ\mapsto RΣR^\top$, and $f_\ell\mapsto D^\ell(R)f_\ell$. This mismatch complicates strict equivariance, leading existing methods to either discard or flatten SH coefficients, thereby breaking symmetry. We propose a unified solution rooted in representation theory: for SH degrees $\ell\le2$, photometry is algebraically isomorphic to a rank-2 geometric tensor. We prove that the Wigner-$D$ action on these SH coefficients can be exactly reformulated as the conjugation action on $3\times3$ matrices. Leveraging this, we introduce the Unified Matrix Embedding, a lifting that maps all Gaussian attributes into a unified carrier space, $\mathfrak{gl}(3)$. Building on the "Color-as-Geometry" formulation, we present E3DGS, a rigid-body ($\mathrm{SE}(3)$) equivariant architecture that processes 3D Gaussians without Clebsch-Gordan tensor products. Evaluations on object vision and action-conditioned Gaussian world modeling demonstrate that our unified approach yields strong robustness under camera-frame changes and improved data efficiency.

Chankyo Kim, Maani GhaffariJuly 17, 20266 min read
中文
Chankyo Kim, Maani Ghaffari (University of Michigan)
https://arxiv.org/abs/2607.15536 · Code: not released

One-line summary

E3DGS proves spherical harmonics (SH ≤2) are equivalent to 3×3 matrix conjugation, unifying Gaussian position/covariance/photometry into a single $\mathfrak{gl}(3)$ matrix carrier processed by a rigid-body equivariant backbone — without Clebsch–Gordan tensor products, maintaining 95.26% classification under zero-shot rotation while baselines collapse to 18.28%.

Abstract

3D Gaussian Splatting (3DGS) captures scenes by coupling explicit geometry (position, covariance) with view-dependent photometry (Spherical Harmonics). Building SE(3)-equivariant architectures on these primitives hits a fundamental representation bottleneck: color is treated as a signal rather than a geometric entity. Rotations act heterogeneously across attributes ($\mu\mapsto R\mu$, $\Sigma\mapsto R\Sigma R^\top$, $f_\ell\mapsto D^\ell(R)f_\ell$). This mismatch complicates strict equivariance, leading existing methods to discard or flatten SH coefficients. E3DGS proposes a representation-theoretic solution: for SH $\ell\leq 2$, photometry is isomorphic to a rank-2 geometric tensor, and the Wigner-D action reformulates exactly as 3×3 matrix conjugation. The Unified Matrix Embedding maps all Gaussian attributes into a single carrier space $\mathfrak{gl}(3)$, enabling E3DGS — a rigid-body SE(3)-equivariant architecture without CG tensor products. Evaluations on object vision and action-conditioned Gaussian world modeling demonstrate strong robustness and improved data efficiency.

1. Background and Motivation

3D Gaussian Splatting (3DGS) models the duality of geometry and light by assigning Spherical Harmonic (SH) coefficients to each primitive for view-dependent appearance. Yet in deep representation learning, this rich photometric information is often discarded — state-of-the-art methods strip away view-dependent SH coefficients to avoid rotational symmetry complexity, reducing glossy textured scenes to matte colored point clouds. This architectural compromise limits the model's ability to distinguish objects with identical geometry but different materials (e.g., matte plastic cup vs. shiny metal mug).

The fundamental bottleneck is strict SE(3) equivariance. Translation equivariance is elegantly handled by relative coordinates, but rotation is severe: geometric attributes (position $\mathbf{x}$, covariance $\Sigma$) transform by standard tensor rules, while SH coefficients transform via Wigner-D matrices. Integrating these disparate types into a single equivariant pipeline typically requires Tensor Field Networks (TFNs), and Clebsch–Gordan (CG) tensor products are computationally prohibitive for million-Gaussian scenes.

E3DGS's paradigm shift: treat photometry as geometry. For SH $\ell\leq 2$, the coefficient spaces $\mathbb{R}^{2\ell+1}$ realize the same irreducible representations ($V_0\oplus V_1\oplus V_2$) that appear in the 3×3 matrix space $\mathfrak{gl}(3)$ under conjugation, replacing the Wigner-D action with the simple conjugation rule governing geometric tensors.

E3DGS overview

Figure 1: E3DGS overview. Gaussian primitives couple geometry (mean/covariance) and photometry (SH), transforming by heterogeneous SO(3) rules. E3DGS lifts both into a unified matrix carrier processed by a rigid-body equivariant backbone.

2. Core Method: Unified Geometric-Photometric Equivariance

2.1 Gaussian Primitives and Group Actions

A Gaussian primitive $\mathcal{G}_i=(\mu_i,\Sigma_i,\alpha_i,\{f_{i,\ell}^{(c)}\}_{\ell=0}^2)$. Under rotation $R\in\mathrm{SO}(3)$: $\mu_i\mapsto R\mu_i$, $\Sigma_i\mapsto R\Sigma_iR^\top$, $\alpha_i\mapsto\alpha_i$, SH $f_{i,\ell}^{(c)}\mapsto D^\ell(R)f_{i,\ell}^{(c)}$. Centered coordinates $\bar{\mu}_i$ isolate the rotational component.

2.2 Unified $\mathfrak{gl}(3)$ Carrier (Core Theorems)

Matrix space $\mathfrak{gl}(3)=\mathbb{R}^{3\times3}$ with conjugation $\mathrm{Ad}_R(M):=RMR^\top$ decomposes into SO(3)-irreducibles:

$$\mathfrak{gl}(3)=\langle I\rangle\oplus\mathfrak{so}(3)\oplus\mathrm{Sym}_0(3)$$

matching SH types up to $\ell=2$. Since $\mathfrak{gl}(3)\cong V_1\otimes V_1\cong V_0\oplus V_1\oplus V_2$, the 3×3 carrier realizes exactly SH $\ell\leq 2$.

Theorem 1: For each $\ell\in\{0,1,2\}$ there exists a linear SO(3)-equivariant map $\Phi_\ell$ with:

$$\Phi_\ell(D^\ell(R)f_\ell)=\mathrm{Ad}_R(\Phi_\ell(f_\ell))=R\,\Phi_\ell(f_\ell)\,R^\top$$

Theorem 2 extends to any degree: each SH band $L$ is realized exactly as conjugation on a suitable carrier $\mathrm{End}(V_k)$, without runtime CG tensor products.

flowchart TB
    G["3D Gaussian primitive
mu, Sigma, alpha, SH f_l"] --> SPLIT["Heterogeneous SO(3) actions"] SPLIT --> MU["mu -> R*mu"] SPLIT --> COV["Sigma -> R*Sigma*R^T"] SPLIT --> SH["f_l -> D^l(R)*f_l
(Wigner-D)"] MU --> LIFT["Unified Matrix Lifting
Phi_<=2 into gl(3)"] COV --> LIFT SH --> LIFT LIFT --> H["H_i = [P_i, C_i, S_i^r,g,b]
all transform as M -> R*M*R^T"] H --> ENC["SE(3)-equivariant backbone
(no CG tensor products)"] ENC --> TASK["Object recognition
(Gaussian-MAE) +
Action world model
(ManiGaussian)"] style LIFT fill:#e0e7ff,stroke:#2563eb style ENC fill:#dcfce7,stroke:#16a34a

2.3 Unified Matrix Embedding

Covariance lift: $C_i:=\log(\Sigma_i)\in\mathrm{Sym}(3)$; $\log(R\Sigma_iR^\top)=R\log(\Sigma_i)R^\top=\mathrm{Ad}_R(C_i)$. Position lift: $P_i:=\widehat{\bar{\mu}_i}\in\mathfrak{so}(3)$. Photometry lift: $S_i^{(c)}:=\Phi_{\leq 2}(f_{i,\leq 2}^{(c)})\in\mathfrak{gl}(3)$. Equivariant features $H_i:=[P_i,C_i,S_i^{(r)},S_i^{(g)},S_i^{(b)}]$ plus Type-0 invariants $s_i$ (opacity, diffuse color, task semantics).

2.4 Action-Conditioned Gaussian World Model

Action $a_t=(v_t,q_t,c_t)$ converted to axis-angle $\omega_t$, lifted via hat map to $\mathfrak{so}(3)$: $H_t^\text{act}=[\widehat{v}_t,\widehat{\omega}_t]$. The world model predicts state deformation:

$$\Delta H_{t+1}=f_\text{wm}^\text{eq}([H_t^\text{state},H_t^\text{act}],c_t)$$

guaranteeing SO(3)-equivariant dynamics $\Delta H_{t+1}\mapsto\mathrm{Ad}_R(\Delta H_{t+1})$.

3. Key Experiments

3.1 Object Recognition (ShapeSplat / Gaussian-MAE)

MethodID/IDID/SE(3) zero-shotSE(3)/SE(3)
Point-BERT94.82××
Gaussian-MAE; SH(0)94.7118.2888.32
Gaussian-MAE; SH(0,1,2)94.0512.8988.87
E3DGS-MAE; SH(0)95.1595.1594.82
E3DGS-MAE; SH(0,1)95.2695.2695.26
E3DGS-MAE; SH(0,1,2)94.7994.7994.49

Table 1: ModelNet10 classification accuracy(%). Baselines collapse under zero-shot transform; E3DGS stays consistent.

Invariant descriptor similarity

Figure 2: Pooled invariant descriptor similarity across input rotations. E3DGS (left) stays stable; G-MAE (right) shows severe drift.

Reconstruction qualitative

Figure 3: MAE reconstruction. G-MAE collapses under arbitrary rotation; E3DGS-MAE consistently reconstructs geometry and photometry.

3.2 Action-Conditioned World Modeling (RLBench)

Methodclose_jaropen_drawersweep_dustpanmeat_offgrillput_in_drawerdrag_stickAvg
PerAct18.754.70.040.02.75.320.4
GNFactor25.376.028.057.30.037.331.7
ManiGaussian28.076.064.060.016.092.044.8
E3DGS16.088.080.072.036.092.050.4

Table 2: RLBench multi-task success rate(%). E3DGS excels on rigid-body articulation tasks.

Dynamic loss

Figure 4: Dynamic loss. E3DGS accelerates rigid-body state-transition learning, converging faster to lower loss.

4.

位姿参数化

$$ g=(R,t)\in\mathrm{SE}(3) $$

高斯椭球参数

$$ \mathcal{G}_{i}=\bigl(\mu_{i},\Sigma_{i},\alpha_{i},\{f^{(c)}_{i,\ell}\}_{\ell=0}^{2}\bigr) $$

Limitations and Future Work

Author-stated limitations:

  • The $\mathfrak{gl}(3)$ formulation is mathematically bounded to SH $\ell\leq 2$. While original 3DGS often uses degree 3, SH(0–2) captures a substantial photometric subspace and is the maximal bandwidth of the lightweight 3×3 carrier; higher bands need larger carriers (Theorem 2), empirical evaluation left to future work.
  • Parameter-efficient but not runtime-efficient — matrix contractions and invariant bilinear operations increase GMACs; described as parameter-efficient rather than broadly runtime-efficient.

Analysis: The core contribution is mathematizing the "color-as-geometry" paradigm — the exact equivalence of SH(0–2) to $\mathfrak{gl}(3)$ conjugation enables strict equivariance of geometry and photometry under a unified carrier, bypassing the CG tensor-product computational bottleneck. Experimental evidence is strong: 95.26% vs. 18.28% under zero-shot rotation is a dramatic gap proving the value of equivariant constraints over data-augmentation approximations. RLBench gains are largest on rigid-body tasks (open_drawer +12%, sweep +16%, put_in_drawer +20%), but close_jar and stack_blocks (contact-heavy, long-horizon) are weaker than baseline, suggesting the equivariant inductive bias may be too strong where contact dynamics have more complex symmetries. Future work should include empirical evaluation of higher-order carriers and extensions for contact/deformation.

5. Conclusion

The core idea of E3DGS is treating photometry as geometry — proving spherical harmonics equivalent to matrix conjugation so geometry and photometry are strictly equivariant under a unified carrier without computationally expensive Clebsch–Gordan tensor products. It proves Theorem 1 (SH $\ell\leq2$ ↔ $\mathfrak{gl}(3)$ conjugation) and Theorem 2 (matrix realization of any SH degree), builds a unified matrix embedding lifting position/covariance/photometry to $\mathfrak{gl}(3)$, and processes them with a rigid-body equivariant backbone. Empirically, 95.26% classification under zero-shot rotation (vs. 18.28% baseline collapse), 50.4% average RLBench success (vs. 44.8%), and faster dynamic-loss convergence. The work provides a mathematically elegant and practical unified framework for equivariant 3DGS learning, though SH bandwidth limits, runtime efficiency, and applicability to contact-dynamics tasks remain open.

Treat color as geometry — the exact equivalence of spherical harmonics and matrix conjugation lets equivariant learning proceed without the burden of Clebsch–Gordan.

Related Papers

DL-SLAM: Enabling High-Fidelity Gaussian Splatting SLAM in Dynamic Environments based on Dual-Level Probability

DL-SLAM: Enabling High-Fidelity Gaussian Splatting SLAM in Dynamic Environments based on Dual-Level Probability

Recent advances in 3D Gaussian Splatting (3DGS) have enabled significant progress in dense dynamic Simultaneous Localization And Mapping (SLAM). Prevailing methods typically discard predefined dynamic objects, ignoring that transiently static objects offer valuable geometric constraints for pose estimation. A recent work attempts to leverage this potential by employing per-pixel uncertainty maps to quantify the magnitude of motion. While this approach enables transiently static objects to enhance pose estimation, it erroneously integrates these objects into the static map, resulting in persistent artifacts. Moreover, its reliance on purely geometric information leads to ambiguous object boundaries in the uncertainty maps. To overcome these limitations, we present DL-SLAM, a monocular Gaussian Splatting SLAM system built upon a novel dual-level probabilistic framework. Our method computes dynamic probability maps by combining semantic and geometric information. These pixel-level probabilities are lifted to 3D and aggregated to derive an object-level dynamic probability for each instance. Object-level probability enables the categorical pruning of dynamic Gaussians, resulting in an artifact-free static map. The static map, in turn, provides a geometrically consistent guidance to refine the pixel-wise probabilities, enhancing their reliability. Experimental results demonstrate that DL-SLAM outperforms existing approaches, improving tracking accuracy by up to 13\% while generating high-fidelity semantic maps.

动态环境Dynamic EnvironmentsSLAMJul 2, 2026
VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes

VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes

VLK synthesizes paired vision-language-kinematics supervision inside 3DGS-reconstructed real scenes: it generates navigation and object-interaction trajectories with privileged scene info, renders egocentric views after the fact, and produces 48,000 paired trajectories to train a policy predicting Unitree G1 whole-body motion, enabling sim-to-real perception-based humanoid loco-manipulation.

humanoid人形机器人loco-manipulationJun 29, 2026
Exploration Matters for Escaping the Blur Trap in 3D Gaussian Splatting

Exploration Matters for Escaping the Blur Trap in 3D Gaussian Splatting

3D Gaussian Splatting (3DGS) employs Gaussian primitives for explicit scene representation, facilitating real-time, high-fidelity reconstruction and novel view synthesis of complex scenes. However, the explicit modeling inherent in 3DGS introduces a gradient bias during optimization, rendering its non-convex optimization process highly susceptible to convergence toward local suboptimal solutions. This constitutes a fundamental limitation in 3DGS optimization, which we term the Blur Trap. To address this limitation, we integrate simple explicit exploration into the 3DGS optimization framework. First, through rigorous mathematical analysis of the 3DGS optimization formulation, we identify the underlying optimization bias responsible for the Blur Trap and categorize it into two distinct subtypes: the Far-Side Blur Trap and the Near-Side Blur Trap. Subsequently, we propose two highly straightforward exploration strategies (Random Seeding and Random Splitting) to mitigate the far-side and near-side blur traps, respectively. Experimental validation demonstrates that the incorporation of these exploration operators effectively and complementarily overcome the Blur Trap, achieving high-quality rendering performance across multiple datasets. Project page: https://chengbo-wang.github.io/ExploreGS/

Paper3D Gaussian Splatting3DGSJul 20, 2026
QIRF Quantum-Inspired Non-Orthogonal Function-Space Compression for 3D Gaussian Splatting

QIRF Quantum-Inspired Non-Orthogonal Function-Space Compression for 3D Gaussian Splatting

3D Gaussian Splatting (3DGS) achieves high-quality real-time rendering by representing a scene with a large collection of anisotropic Gaussian primitives. However, complex scenes often require millions of Gaussians, resulting in substantial storage and rendering costs. Existing compression methods mainly reduce redundancy through primitive-wise pruning, attribute quantization, clustering, or neural coding, while redundancy caused by strongly overlapping and non-orthogonal Gaussian basis functions remains largely unexplored. We present QIRF, a quantum-inspired non-orthogonal function-space compression method for 3D Gaussian Splatting. QIRF models neighboring Gaussian primitives as a local non-orthogonal basis and formulates primitive reduction as a subspace-aware selection problem. Specifically, an analytic Gaussian overlap matrix and a radiance-response density matrix are constructed to characterize functional redundancy and rendering relevance. Generalized eigendecomposition is then used to identify the dominant local subspace and select representative Gaussian primitives. An RRDM-based response model and detail-aware safeguarding further preserve visually important high-frequency structures under aggressive pruning. Experiments on 13 scenes from Mip-NeRF 360, Tanks and Temples, and Deep Blending show that QIRF reduces the Gaussian count and raw PLY storage by 71.7 percent on average, corresponding to approximately 3.54 times compression, while maintaining reconstruction quality comparable to 3DGS and achieving a marginal average PSNR improvement of 0.10 dB. QIRF also improves the average rendering speed over 3DGS by 34.3 percent. These results suggest that non-orthogonal function-space redundancy is an important yet underexplored source of representational redundancy in explicit Gaussian radiance fields.

Paper3D Gaussian Splatting3DGSJul 20, 2026