A neural radiance field is a small neural network whose weights encode one specific scene — not a model of scenes in general, a model of your kitchen. Gaussian splatting keeps the idea of fitting a scene to photographs and throws away the network, which is why it renders in milliseconds instead of seconds.
A scene becomes weights, concretely
The function being learned takes a point in space and a viewing direction, and returns a colour and a density:
F(x, y, z, theta, phi) -> (r, g, b, sigma)
5 numbers in, 4 numbers out.
Original NeRF (Mildenhall and colleagues, 2020):
8 hidden layers of 256 units
roughly 600,000 parameters
about 2.4 MB in fp32
Training input: 50-100 photographs of ONE scene, with camera poses
recovered by structure-from-motion.
Training time: hours to about a day on one GPU, in the original work.
That is what “a scene becomes weights” means. There is no mesh, no point cloud, no texture atlas. The geometry and the appearance of the room are stored as 600,000 floating-point numbers in a multi-layer perceptron, and the only way to find out what is at a point is to evaluate the network there.
It also means the model generalises to nothing. Train on your kitchen and you have a kitchen; a different room requires training again from scratch. These are per-scene fits, which is a genuinely unusual thing for a neural network to be and is the source of most confusion about them.
One detail is load-bearing: the input coordinates are not fed in directly. They are expanded with sines and cosines at many frequencies first — the same Fourier positional encoding idea used in transformers. Without it the MLP produces a smooth blur, because a small network fed raw coordinates has a strong bias toward low-frequency functions. With it, sharp edges become representable. That one change is the difference between the technique working and not.
How a pixel is produced
There is no rasteriser. To find the colour of one pixel you cast a ray from the camera through it and integrate what the ray passes through:
sample points along the ray: t_1 ... t_n
query the MLP at each: colour c_i, density sigma_i
transmittance (how much light survives to sample i):
T_i = exp( - sum over j<i of sigma_j * delta_j )
pixel colour:
C = sum over i of T_i * (1 - exp(-sigma_i * delta_i)) * c_i
delta_i = distance between consecutive samples.
The whole rendering equation is differentiable, which is the reason this works at all: you compare the rendered pixel to the real photograph and backpropagate the error into the weights. The scene is fitted by rendering it repeatedly and correcting.
Sampling is done in two passes — a coarse network to find where the surfaces are, then a fine network concentrating samples there — because most of a ray is empty air and evaluating the MLP in empty air is wasted work.
Why it was slow: count the evaluations
One 800 x 600 frame:
rays = 800 * 600 = 480,000
samples per ray = 64 coarse + 128 fine = 192
MLP evaluations = 480,000 * 192 = 92,160,000
each evaluation: an 8-layer, 256-wide MLP
roughly 600,000 MACs -> about 5.5 x 10^13 MACs per frame
Ninety-two million forward passes through a small network, for one image. At 30 frames per second that would be 2.8 billion network evaluations per second, sustained. This is not an implementation problem; it is what the representation demands, and it is the entire reason everything that came afterwards came afterwards.
The first fix: put features in a grid
The network is doing two jobs at once: storing the scene and computing the colour. Split them. Store learned feature vectors in a spatial data structure, look up the ones near the query point, and let a much smaller MLP turn the looked-up features into a colour.
Instant-NGP (NVIDIA, 2022) is the well-known version, using a multiresolution hash table of feature vectors and an MLP of two or three tiny layers. Reported training times drop from hours to seconds or minutes and rendering becomes interactive.
The trade is stated in one line, and it is the same trade as everywhere else in this cluster: memory for compute. The 2.4 MB MLP becomes tens or hundreds of megabytes of hash tables, and in exchange the arithmetic per sample falls by an order of magnitude.
What splatting changed
3D Gaussian splatting (Kerbl and colleagues, SIGGRAPH 2023) takes the next step and removes the network from the scene representation entirely. A scene is a large set of explicit 3D Gaussian blobs, each with its own parameters:
per Gaussian:
position 3 floats
scale 3
rotation (quaternion) 4
opacity 1
colour, spherical 3 (degree 0) ... 48 (degree 3)
harmonics
----
~59 floats at degree 3 = about 236 bytes
typical scene: 1 to 5 million Gaussians
= 240 MB to 1.2 GB uncompressed
Spherical harmonics are how the colour is allowed to change with viewing angle, which is what preserves specular highlights and gloss — the thing that makes a reconstruction look like a photograph rather than a model.
The change that matters is in how a frame is produced. Rendering is no longer “evaluate a network many times along each ray”. It is:
- Project each 3D Gaussian to the image plane, which gives a 2D Gaussian — a soft ellipse — by a linear transformation of its covariance.
- Assign the ellipses to screen tiles and sort them by depth.
- Alpha-blend front to back within each tile until the accumulated opacity saturates.
That is rasterisation: project, sort, blend. It is the operation graphics hardware has had dedicated paths for since the 1990s, and it is why the original paper reports real-time rendering at 1080p at frame rates above 100 — against seconds per frame for a comparable-quality NeRF. Nothing about the fitting got fundamentally smarter; the representation was changed into one the hardware already knew how to draw.
Training is still gradient descent against photographs, and it still uses a differentiable renderer. The Gaussians are additionally cloned, split and pruned during optimisation, so the model adds detail where the reconstruction error is high rather than starting from a fixed budget.
How a splat scene is actually fitted
The optimisation is worth describing, because it explains something that otherwise looks arbitrary: nobody chooses how many Gaussians a scene has. The number is an outcome.
- Initialise from the point cloud you already have. Structure-from-motion produced camera poses, and it produced a sparse point cloud as a by-product — typically 50,000 to 200,000 points. Each becomes one Gaussian, small and roughly isotropic, with the point’s colour.
- Render and compare. Rasterise the current scene from a training camera and compute the loss against the real photograph. The loss is an L1 term plus a structural-similarity term rather than plain squared error, because pixel-mean error is a poor stand-in for how a photograph looks.
- Backpropagate into everything. Positions, scales, rotations, opacities and spherical-harmonic coefficients are all ordinary parameters with gradients. There is no network in the loop.
- Adaptive density control, every hundred iterations or so. Gaussians in regions the reconstruction is missing show large positional gradients. Small ones there are cloned; large ones covering high-detail regions are split into two smaller ones; and any Gaussian whose opacity has fallen below a threshold is deleted. Opacity is also periodically reset so that the pruning has something to bite on.
- Stop after a fixed budget. Around thirty thousand iterations in the original work, on the order of tens of minutes on one GPU for a typical scene, by which point the population has grown from a hundred thousand to a few million.
Step four is the interesting one and it has no equivalent in ordinary neural network training: the model changes its own size during optimisation, adding capacity exactly where the error is. That is why detailed scenes end up with more Gaussians than empty ones, and why the file size of a splat scene is a measurement of how complicated the room was rather than a hyperparameter anybody set.
What neither of them gives you
| Limit | Description |
|---|---|
| Camera poses are an input | Both need to know where each photograph was taken, typically from structure-from-motion. That preprocessing step fails on textureless surfaces, repeated patterns and small baselines, and when it fails nothing downstream works. |
| One scene, one fit | Neither produces a model that generalises. Feed-forward variants that predict a scene from a few images in one pass exist and are moving quickly, but the classic pipeline is per-scene optimisation. |
| Lighting is baked in | The appearance captured is the appearance under the lighting at capture time. Relighting, moving a light, or inserting the scene into another environment all need inverse rendering, which is a harder and less solved problem. |
| No usable geometry by default | A NeRF is a density field and a splat scene is a cloud of blobs. Neither is a watertight mesh, so physics, collision and most content pipelines need a conversion step that is lossy. |
| Storage versus compute | The NeRF is megabytes and slow to render; the splat scene is hundreds of megabytes and fast. On a phone or over a network that difference is the deciding factor, and compressing splat scenes is an active area precisely because of it. |
This field moves faster than almost anything else in this cluster. Compression, dynamic and animated scenes, and feed-forward reconstruction from a handful of images are all active, and the numbers above will improve. The arithmetic that will not change is the reason for the shift: querying a network per sample per ray is a different order of work from projecting and blending primitives.
Top comments (0)