Summary
Highlights
Julia Federrico introduces the goal of NeRF and Gaussian Splatting: to create a 3D representation from existing photos to synthesize novel views and potentially extract 3D models from angles not originally captured.
The explanation begins with a simplified model using voxels, where each voxel has an associated RGB color and opacity. Rays are cast through pixels from camera views, and points are sampled along these rays. The formula for computing the final pixel color is introduced, emphasizing how color and opacity values are learned during training.
The speaker elaborates on how a voxel's color can depend on the viewing direction. The concept of transmittance is explained using the Lambert-Beer law, demonstrating how a voxel's density affects light absorption and how much of its color contribution reaches the eye, acting as a weight in the final color calculation.
After explaining the voxel-based rendering, NeRF is introduced as an improvement. Instead of directly optimizing voxel colors and densities, a neural network (MLP) predicts these values given a voxel's position and viewing direction, allowing for higher resolution and richer detail.
A common problem with MLPs is their inability to model high-frequency components, leading to overly smooth outputs. This is solved by using Fourier features vectors, which convert Cartesian coordinates into sinusoidal functions, enabling NeRF to capture finer details.
While NeRF can theoretically extract meshes, the quality is often poor due to noise. The video introduces NeRF-based alternatives like Neus (Neural Implicit Surface Representations), which leverages Signed Distance Functions (SDF) to predict distances from a surface, resulting in smoother and more detailed meshes.
To address the slow training times of traditional NeRF, Instant NGP (EMGP) is presented. This variant uses a multi-resolution hash grid encoding to store learnable vectors, which are then interpolated and fed into a smaller MLP, dramatically speeding up training without sacrificing quality.
The final topic is Gaussian Splatting, a real-time rendering algorithm. It starts with a point cloud, where each point is associated with a 3D Gaussian defined by its center, scale, rotation, color, and opacity. These Gaussians are optimized to represent the scene.
The process of projecting 3D Gaussians onto the 2D camera plane is explained, involving transformation into camera coordinates and using the Jacobian matrix for accurate 2D Gaussian representation. The final pixel color is a weighted sum of contributions from multiple Gaussians, with transmittance used for front-to-back compositing.
The video highlights how Gaussian Splatting achieves real-time performance by efficiently determining which pixels are affected by each Gaussian using covariance matrix analysis. The parameterization of the covariance matrix is discussed, ensuring mathematical properties like symmetry and positive semi-definiteness through eigenvalue decomposition and quaternions for rotation.
Finally, the color of each Gaussian is encoded using spherical harmonics. These functions allow the Gaussian's color to vary depending on the viewing direction, similar to NeRF. The degree of the spherical harmonic function (L) dictates the level of detail and directional variation in color.