Rendering 3D Gaussians on a Graph Processor

Figure 1 — Kernel execution order per tile. A Gaussian is kept if rendered locally, else moved to a communication channel toward its destination.

1. Background & Motivation

3D Gaussian Splatting (3DGS) has become the leading method for real-time novel view synthesis: scenes are modeled as collections of anisotropic 3D Gaussians projected and alpha-composited in screen space, achieving photorealistic quality while remaining fully differentiable. However, most implementations assume GPU hardware with large off-chip DRAM, high memory bandwidth, and a global address space. 3DGS rendering is memory-bound on GPU: DRAM access, not arithmetic, dominates frame time. Gaussian data is repeatedly loaded into thread-block shared memory at every kernel launch.

Efficient computing relies on memory locality: minimizing the distance between data and compute reduces latency and energy. 3DGS rendering is parallel in screen space but involves global data movement—any Gaussian can project to any screen location, and Gaussians with large spatial extent must be shared across multiple screen regions. On a GPU this is handled by global random-access memory at the cost of memory latency and power. On an architecture without random-access memory, this data movement must be made explicit.

"On-sensor" computing—using pixel processor arrays (PPAs) to run meaningful computer-vision tasks entirely on-chip with extremely constrained resources—motivates the question: if the front-end of a spatial computing pipeline (feature detection, tracking, mapping) can run on massively parallel processors with only local memory, what does rendering look like on similar hardware? This paper uses the Graphcore Mk2 IPU to investigate.

2. Core Method

2.1 3D Gaussian Splatting Background

3DGS represents a scene as 3D Gaussian primitives, each parameterized by mean $\boldsymbol{\mu}\in\mathbb{R}^{3}$, 3D covariance matrix $\boldsymbol{\Sigma}$, opacity $\alpha$, and view-dependent color encoded via spherical harmonics. The covariance is reparameterized as $\boldsymbol{\Sigma}=RSS^{T}R^{T}$ where $R$ is a rotation matrix (stored as quaternion) and $S$ is a diagonal scaling matrix.

Rendering proceeds in four stages: (1) frustum culling; (2) projection to 2D screen space, producing 2D mean and covariance (the "conic" matrix $Q$); (3) depth sorting; (4) per-pixel alpha compositing in front-to-back order using the EWA splatting formulation:

$$C(\mathbf{x})=\sum_{i=1}^{N}c_{i}\,\sigma(\alpha_{i})\exp\!\bigl(-\tfrac{1}{2}\bar{\mathbf{x}}_{i}^{T}Q_{i}\,\bar{\mathbf{x}}_{i}\bigr)\prod_{j=1}^{i-1}\bigl(1-\sigma(\alpha_{j})\exp\!\bigl(-\tfrac{1}{2}\bar{\mathbf{x}}_{j}^{T}Q_{j}\,\bar{\mathbf{x}}_{j}\bigr)\bigr)$$

where $\bar{\mathbf{x}}_{i}=\mathbf{x}-\boldsymbol{\mu}_{i}^{2D}$ is the offset from the projected mean.

2.2 Graphcore IPU Architecture

The IPU Mk2 (GC200) comprises 1,472 independent tiles, each with a processor core (6 hardware threads) and 624 KB local SRAM—no external DRAM, no L2 cache, no shared address space. Execution follows the BSP model: each superstep consists of local computation, an exchange phase (tiles communicate via a structured fabric), and a global barrier synchronization. Communication patterns are defined at compile time via the Poplar graph compiler. The key contrast with GPUs: all data movement is explicit and predetermined—a GPU thread can load any global memory address at runtime; an IPU tile can only send/receive tensors defined when the program is compiled.

Table 1 — Architectural properties of GPU, IPU, and PPA hardware classes.

2.3 Rendering Pipeline Design

The pipeline maps the four 3DGS stages onto the IPU's BSP execution model:

• Projection (local compute): Each tile projects its stored Gaussians using the view matrix streamed from the host, computing 2D means, conics, and bounding boxes.
• Routing (compute + exchange, repeated): Gaussians whose projected means fall outside the local tile are evicted toward their destination via Manhattan-distance hops on the NEWS grid. Repeats over BSP supersteps until convergence, guaranteed within $\max(W,H)$ supersteps ($W$, $H$ are grid dimensions).
• Bloom (compute + exchange, repeated): Gaussians at their anchor tile whose 2D screen extent spans multiple framebuffer tiles are propagated to overlapping neighbors in an expanding tree pattern—horizontal (left/right) then vertical (up/down), eliminating cyclic copying.
• Compositing (local compute): Each tile sorts local Gaussians by depth and performs front-to-back alpha compositing onto its framebuffer slice.

flowchart TB
    HOST["Host: View/Projection Matrix"] --> PROJ["Projection (Local Compute)"]
    PROJ --> ROUTE{"Anchor Local?"}
    ROUTE -->|Yes| BLOOM{"Spans Multi-Tile?"}
    ROUTE -->|No| NEWS["NEWS Grid Routing
Manhattan-Distance Hops"]
    NEWS --> ROUTE
    BLOOM -->|Yes| TREE["Tree-Pattern Bloom
Horizontal→Vertical"]
    BLOOM -->|No| COMP["Local Compositing"]
    TREE --> COMP["Depth Sort + Alpha Compositing"]
    COMP --> FB["Framebuffer Slice"]
    FB --> OUT["Host: Assemble Framebuffer"]
    style PROJ fill:#e0e7ff,stroke:#2563eb
    style NEWS fill:#fef3c7,stroke:#d97706
    style TREE fill:#fce7f3,stroke:#db2777
    style COMP fill:#dcfce7,stroke:#16a34a

2.4 Framebuffer Partitioning & Gaussian Representation

The output framebuffer (1280×720) is divided into 1,440 equal 32×20 pixel slices, each pinned to a separate IPU tile's SRAM. Tiles are arranged in a 2D grid matching the framebuffer's spatial layout, so tile adjacency corresponds to screen-space adjacency. Each Gaussian is stored as a 60-byte struct (mean 12B + color+opacity 16B + quaternion rotation 16B + log-space scale 12B + ID 4B), with view-dependent color simplified to zeroth-order spherical harmonics.

3. Experiments

3.1 Render Quality

For moderate-density scenes, IPU rendering is near-identical to the GPU baseline, preserving fine details including legible text. However, the dense Bonsai scene (273K Gaussians) reveals current limits: regions of very high Gaussian density exhibit rectangular tiling artifacts caused by channel saturation during the bloom phase preventing full Gaussian propagation.

3.2 Performance Analysis

Total sums the slowest-tile (max) contribution per stage, as the BSP barrier is set by the slowest tile. Inter-tile exchange averages 0.07ms.

3.3 Data Locality & Churn Rate

The IPU's core advantage is exploiting data locality for incremental view changes. Static views have 0% churn (Gaussians don't move between frames), while GPU churn is effectively 100% per frame (all Gaussians potentially reloaded from DRAM).

Churn is much lower for incremental view changes than teleports, confirming data movement scales with view-change magnitude. Suited for online SLAM, AR glasses, mobile robots with incremental viewpoints.

Figure 2 — Gaussian "blooming" protocol. Arrows show the BSP timestep order.

4. Contributions

• First implementation of 3DGS rendering on an SRAM-only MIMD processor architecture.
• A routing scheme for distributing Gaussian primitives across tiles within compile-time communication constraints.
• Experimental analysis of rendering quality and performance.
• Bottleneck evaluation: inter-tile bandwidth saturation, per-tile SRAM pressure, load imbalance.
• Insights for 3D representation and algorithm design for GPUs and future DRAM-free/on-sensor architectures.

5.

The 3D Gaussian covariance is decomposed as:

$$ \boldsymbol{\Sigma}=RSS^{T}R^{T} $$

The color at pixel $\mathbf{x}$ is computed by alpha-blending:

$$ C(\mathbf{x})=\sum_{i=1}^{N}c_{i}\,\sigma(\alpha_{i})\exp\!\bigl(-\tfrac{1}{2}\bar{\mathbf{x}}_{i}^{T}Q_{i}\,\bar{\mathbf{x}}_{i}\bigr)\prod_{j=1}^{i-1}\bigl(1-\sigma(\alpha_{j})\exp\!\bigl(-\tfrac{1}{2}\bar{\mathbf{x}}_{j}^{T}Q_{j}\,\bar{\mathbf{x}}_{j}\bigr)\bigr) $$

The offset from the Gaussian center:

$$ \bar{\mathbf{x}}_{i}=\mathbf{x}-\boldsymbol{\mu}_{i}^{2D} $$

The final pixel color in standard 3D Gaussian Splatting is computed by front-to-back alpha blending:

$$ C=\sum_{i=1}^{N}c_i\,\alpha_i\prod_{j=1}^{i-1}(1-\alpha_j) $$

where $\alpha_i=\sigma(\alpha_i^{raw})$ is the opacity, $c_i$ is the color, and the product $\prod(1-\alpha_j)$ represents the transmittance of all preceding Gaussians.

Limitations & Future Work

Bottlenecks: (1) Inter-tile bandwidth: NEWS channel capacity is fixed at compile time; dense regions cause rectangular artifacts when channels saturate. (2) Propagation latency: A Gaussian advances one tile per BSP superstep; an anchor moving $d$ tiles takes $d$ frames to arrive. (3) SRAM capacity: A single tile's 192KB buffer holds only 12.7% of Sloth, 1.2% of Bonsai. (4) Load imbalance: Non-uniform Gaussian density overloads some tiles.

Future: Compact representations (quantize the 60-byte struct for PPA-scale memory); hierarchical multi-resolution Gaussians; hybrid architectures (PPA front-end + capable tiles for rendering); direct inter-SM communication on GPUs to reduce DRAM access; backward pass (gradients flow primarily between neighboring tiles in incremental settings); large-scale rendering with scene data routed across CPU clusters.

6. Conclusion

This work demonstrates for the first time that 3D Gaussian Splatting's forward pass does not fundamentally require DRAM—once each Gaussian reaches the right tile, rendering is embarrassingly parallel, just as on a GPU. What changes is that data movement, normally hardware-managed with a global address space and cache hierarchy, becomes a core part of the algorithm: DRAM access is replaced by a network of nearest-neighbor exchanges. The central theme is exploiting spatial and temporal locality: standard GPU 3DGS re-sorts the entire Gaussian list in DRAM on every view change, while the IPU fabric allows moving data between tiles only when necessary. When viewpoints change gradually (e.g., robotic SLAM), most primitives remain in place, making local nearest-neighbor exchange a natural fit. This is an early step toward rendering pipelines for on-sensor/edge architectures and GPU kernels that spend less time waiting on DRAM.