Compressed 3D Gaussian Splatting for Accelerated Novel View Synthesis

Abstract

Recently, high-fidelity scene reconstruction with an optimized 3D Gaussian splat representation has been introduced for novel view synthesis from sparse image sets. Making such representations suitable for applications like network streaming and rendering on low-power devices requires significantly reduced memory consumption as well as improved rendering efficiency.
We propose a compressed 3D Gaussian splat representation that utilizes sensitivity-aware vector clustering with quantization-aware training to compress directional colors and Gaussian parameters. The learned codebooks have low bitrates and achieve a compression rate of up to 31x on real-world scenes with only minimal degradation of visual quality. We demonstrate that the compressed splat representation can be efficiently rendered with hardware rasterization on lightweight GPUs at up to 4x higher framerates than reported via an optimized GPU compute pipeline. Extensive experiments across multiple datasets demonstrate the robustness and rendering speed of the proposed approach.

Compression Method

We address the storage and rendering issue of 3D Gaussian Splatting (3DGS) by compressing the reconstructed scene parameters and rendering the compressed representation via GPU rasterization. To compress the scenes, we first analyze its components and observe that the Spherical Harmonics (SH) coefficients and the multivariate Gaussian parameters take up the majority of storage space and are highly redundant. Our compression pipeline consists of three steps:

Sensitivity-aware clustering: We derive a sensitivity measure for each scene parameter by calculating its contribution to the training images. Color information and Gaussian parameters are encoded into compact codebooks via sensitivity-aware vector quantization.
Quantization-aware fine-tuning: To regain information that is lost during clustering we fine-tune the scene parameters at reduced bit-rates using quantization-aware training.
Entropy encoding: 3D Gaussians are linearized along a space-filling curve to exploit the spatial coherence of scene parameters with entropy and run-length encoding.

Rendering

Gaussians are rendered in sorted order via GPU rasterization. For each Gaussian, one planar quad (a so-called splat) consisting of two triangles is rendered. A vertex shader computes the screen space vertex positions of each splat from the 2D covariance information. The size of a splat is set such that it covers the 99% confidence interval of the projected Gaussian. The vertex shader simply outputs the color computed in the pre-pass and the 2D splat center as input to the pixel shader. The pixel shader then discards fragments outside the 99% confidence interval. All remaining fragments use their distance to the splat center to compute the exponential color and opacity falloff and blend their final colors into the framebuffer.

The renderer is implemented with the WebGPU graphics API in the Rust programming language. Thus, it can run in a modern web browser on a large variety of devices. A demo can be found below.

The source code for our renderer can be found here.

Results

We evaluate our method on 13 real scenes and the NeRF Synthesis dataset. Our method achieves a 26x compression rate on average on real scenes at minimal loss in visual quality. Our rasterization based renderer achieves a 3.5x speedup for the compressed scenes compared to the compute shader based renderer of Kerbl et al. The FPS measures shown below were obtained on the bicycle scene from Mip-NeRF360 on an NVIDIA GeForce A5000.

Interactive Web Demo

All scenes used in the paper.

Examples

Image comparisons for the baseline and compressed reconstruction. All images are taken from the test set.

Ours

47 MB

3DGS

1.5 GB

Ours

47 MB

3DGS

1.4 GB

Ours

21 MB

3DGS

600 MB

Ours

22 MB

3DGS

602 MB

Concurrent Work

There's a lot of concorrent work that was introduced around the same time as ours:

LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS

Compact3D: Compressing Gaussian Splat Radiance Field Models with Vector Quantization

Compact 3D Gaussian Representation for Radiance Field

Compact 3D Scene Representation via Self-Organizing Gaussian Grids

BibTeX

@misc{niedermayr2023compressed,
    title={Compressed 3D Gaussian Splatting for Accelerated Novel View Synthesis}, 
    author={Simon Niedermayr and Josef Stumpfegger and Rüdiger Westermann},
    year={2023},
    eprint={2401.02436},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}

Compressed 3D Gaussian Splatting for Accelerated Novel View Synthesis

CVPR 2024

Compressing 3D Gaussian Splats up to 31x (26x avg.) for reduced storage requirements and accelerated novel view synthesis.

Abstract

Compression Method

Rendering

Results

Interactive Web Demo

Examples

Concurrent Work

BibTeX