Compressed 3D Gaussian Splatting for Accelerated Novel View Synthesis

CVPR 2024

Technical University of Munich

Compressing 3D Gaussian Splats up to 31x (26x avg.) for reduced storage requirements and accelerated novel view synthesis.

Abstract

Recently, high-fidelity scene reconstruction with an optimized 3D Gaussian splat representation has been introduced for novel view synthesis from sparse image sets. Making such representations suitable for applications like network streaming and rendering on low-power devices requires significantly reduced memory consumption as well as improved rendering efficiency.
We propose a compressed 3D Gaussian splat representation that utilizes sensitivity-aware vector clustering with quantization-aware training to compress directional colors and Gaussian parameters. The learned codebooks have low bitrates and achieve a compression rate of up to 31x on real-world scenes with only minimal degradation of visual quality. We demonstrate that the compressed splat representation can be efficiently rendered with hardware rasterization on lightweight GPUs at up to 4x higher framerates than reported via an optimized GPU compute pipeline. Extensive experiments across multiple datasets demonstrate the robustness and rendering speed of the proposed approach.

Compression Method

We address the storage and rendering issue of 3D Gaussian Splatting (3DGS) by compressing the reconstructed scene parameters and rendering the compressed representation via GPU rasterization. To compress the scenes, we first analyze its components and observe that the Spherical Harmonics (SH) coefficients and the multivariate Gaussian parameters take up the majority of storage space and are highly redundant. Our compression pipeline consists of three steps:


Rendering

Gaussians are rendered in sorted order via GPU rasterization. For each Gaussian, one planar quad (a so-called splat) consisting of two triangles is rendered. A vertex shader computes the screen space vertex positions of each splat from the 2D covariance information. The size of a splat is set such that it covers the 99% confidence interval of the projected Gaussian. The vertex shader simply outputs the color computed in the pre-pass and the 2D splat center as input to the pixel shader. The pixel shader then discards fragments outside the 99% confidence interval. All remaining fragments use their distance to the splat center to compute the exponential color and opacity falloff and blend their final colors into the framebuffer.

The renderer is implemented with the WebGPU graphics API in the Rust programming language. Thus, it can run in a modern web browser on a large variety of devices. A demo can be found below.

The source code for our renderer can be found here.

Results

We evaluate our method on 13 real scenes and the NeRF Synthesis dataset. Our method achieves a 26x compression rate on average on real scenes at minimal loss in visual quality. Our rasterization based renderer achieves a 3.5x speedup for the compressed scenes compared to the compute shader based renderer of Kerbl et al. The FPS measures shown below were obtained on the bicycle scene from Mip-NeRF360 on an NVIDIA GeForce A5000.

Interactive Web Demo

Examples

Image comparisons for the baseline and compressed reconstruction. All images are taken from the test set.

Ours
47 MB
3DGS
1.5 GB
Ours
47 MB
3DGS
1.4 GB
Ours
21 MB
3DGS
600 MB
Ours
22 MB
3DGS
602 MB

BibTeX

@InProceedings{Niedermayr_2024_CVPR,
    author    = {Niedermayr, Simon and Stumpfegger, Josef and Westermann, R\"udiger},
    title     = {Compressed 3D Gaussian Splatting for Accelerated Novel View Synthesis},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2024},
    pages     = {10349-10358}
}