Most neural networks give you a point estimate: one set of weights, one prediction, no honest statement about how sure the model is. Full Bayesian inference would fix that, but it is usually intractable for a network with millions of parameters. The Laplace approximation is a pragmatic middle path. It takes a model you have already trained, looks at the curvature of the loss around its optimum, and turns that local geometry into a Gaussian posterior over the weights. No retraining, no sampling loop, no new architecture. Just a careful second-order look at a solution you already have.
This article explains where the approximation comes from, what the Hessian is really telling you, how to turn it into predictive uncertainty, and where the method breaks down. It is most useful when you already have a trained model and you want calibrated uncertainty cheaply, not when you need an exact posterior.
The Bayesian problem, stated plainly
Suppose we have data D and parameters theta. Bayes rule gives the posterior:
The denominator, the evidence , is an integral over every possible parameter value. For a neural network that integral has no closed form. Markov chain Monte Carlo can approximate it, and variational inference can approximate it with optimisation, but both add substantial cost and engineering.
The Laplace approximation asks a simpler question. If the posterior is sharply peaked around its mode, can we approximate it by a Gaussian centred at that mode, with a covariance set by the local curvature? In many trained models the answer is: well enough to be useful, provided we check the assumptions rather than trusting them blindly.
Step 1: Find the mode
Define the unnormalised log posterior, or equivalently the negative log posterior that we minimise during training:
The maximum a posteriori (MAP) estimate minimises . For a network trained with weight decay, this is familiar territory: the data term is the usual loss, and the log prior is the regulariser. A Gaussian prior with variance corresponds exactly to L2 regularisation, which is why MAP training feels so ordinary. The Bayesian step starts after training, not instead of it.
One subtlety matters here. The mode of a high-dimensional posterior is not necessarily a typical sample from it. In high dimensions, most posterior mass sits in a shell away from the mode, not at the mode itself. Laplace ignores that distinction. It is a local approximation, and its quality depends on how well a Gaussian describes the neighbourhood of the mode, not on whether the mode is representative in a global sense.
Step 2: Taylor-expand to second order
Expand around . The first-order term vanishes because we are at a minimum, so we get:
Here is the Hessian of at the MAP, the matrix of second derivatives. Exponentiating the negative of this quadratic gives a Gaussian. That is the whole trick: a quadratic log density is a Gaussian density.
The resulting Laplace approximation is:
The covariance is the inverse Hessian. That single fact explains both the power and the limits of the method.
What the Hessian is telling you
Curvature is information. If the loss rises steeply when you move a parameter, that parameter is well determined by the data: the Hessian entry is large, its inverse is small, and the posterior variance is small. If the loss is almost flat in some direction, that direction is poorly determined: the Hessian is small, its inverse is large, and the posterior is wide.
In a neural network, flat directions are everywhere. Symmetries such as permuting hidden units, scaling a layer up and the next layer down, and genuinely redundant parameters all create near-flat directions. The Hessian captures this locally, which is why Laplace uncertainty can be large exactly where the model has freedom it has not earned from data.
There is a catch. The full Hessian for a modern network is far too large to form, let alone invert. A network with one million parameters has a Hessian with one trillion entries. Practical Laplace therefore never uses the full matrix. The standard choices are:
- Diagonal approximation: keep only the diagonal of the Hessian. Cheap, but it throws away all correlation between parameters.
- Kronecker-factored approximation (KFAC): approximate each layer's Hessian block as a Kronecker product of two much smaller matrices, one for inputs and one for output gradients. This keeps within-layer correlation at a manageable cost and is the workhorse for larger networks.
- Last-layer Laplace: apply the approximation only to the final linear layer, treating the rest of the network as a fixed feature extractor. This is surprisingly effective, because much of the predictive uncertainty in a classification model lives in that last layer, and the problem becomes small and well behaved.
- Low-rank plus diagonal: keep a diagonal term plus a low-rank correction capturing the dominant curvature directions.
Which one you choose is a trade-off between fidelity and feasibility, and it should be stated explicitly whenever Laplace results are reported. "Laplace approximation" without saying which Hessian approximation was used is an incomplete description.
A related practical point: the Hessian of the loss is not always positive definite at a point that optimisation merely approached rather than reached. In practice, the Gauss-Newton approximation or a damped Hessian, with a small , is used to guarantee a valid covariance. Damping is not a cosmetic fix; it directly widens the posterior, so the damping value is part of the method, not an implementation detail to hide.
From weight uncertainty to predictive uncertainty
A posterior over weights is only useful if it becomes uncertainty over predictions. For a new input , the predictive distribution is:
With the Laplace Gaussian in place of the true posterior, this integral is still not closed form for a nonlinear network. The standard workaround is linearisation: approximate the network output around using its Jacobian with respect to the parameters. Then the predictive distribution over the network output becomes Gaussian with covariance . For classification, that Gaussian over logits is passed through a softmax, either by sampling logits or by a probit-style approximation, to give class probabilities that reflect weight uncertainty.
This linearised view reveals something important. Laplace predictive uncertainty grows when the Jacobian is large in directions where the Hessian inverse is large: that is, when the prediction is sensitive to parameters the data did not pin down. On in-distribution inputs, a well-trained model is usually insensitive in those directions or the data has pinned them down, so uncertainty stays low. On out-of-distribution inputs, sensitivity often grows, and so does uncertainty. That is the behaviour you want from an uncertainty method, even if the numbers are approximate.
A minimal worked example
Consider logistic regression, where Laplace is exact enough to be instructive and small enough to compute directly. The model has a Gaussian prior and a Bernoulli likelihood, so the log posterior is concave and the Laplace approximation is well behaved. The code below fits the MAP by gradient descent in a simple loop, computes the Hessian analytically, and samples weights from the Laplace Gaussian to show predictive spread.
import numpy as np
rng = np.random.default_rng(0)
X = rng.normal(size=(200, 2))
true_w = np.array([2.0, -1.5])
logits = X @ true_w
y = (rng.random(200) < 1 / (1 + np.exp(-logits))).astype(float)
# MAP with a Gaussian prior: loss = negative log likelihood + prior term
prior_var = 4.0
w = np.zeros(2)
for step in range(500):
p = 1 / (1 + np.exp(-(X @ w)))
grad = X.T @ (p - y) + w / prior_var
w -= 0.05 * grad
# Hessian of the negative log posterior at the MAP
p = 1 / (1 + np.exp(-(X @ w)))
S = p * (1 - p)
H = (X * S[:, None]).T @ X + np.eye(2) / prior_var
cov = np.linalg.inv(H)
# Predictive spread on a new point, from Laplace samples
x_new = np.array([1.0, 1.0])
samples = rng.multivariate_normal(w, cov, size=2000)
pred_probs = 1 / (1 + np.exp(-(samples @ x_new)))
print("MAP weights:", w)
print("Predictive mean:", pred_probs.mean())
print("Predictive std:", pred_probs.std())
Two things are worth noticing in this toy case. First, the Hessian is a weighted Gram matrix: points near the decision boundary, where is large, contribute most to curvature. Points the model is already sure about contribute almost nothing, which matches the intuition that confident points teach you little about the boundary. Second, the predictive standard deviation is not a property of the point alone; it depends on how the point aligns with the directions of weight uncertainty, through the Jacobian, which here is just itself.
For a neural network, the same recipe applies with two changes: the Hessian is replaced by a tractable approximation such as KFAC or a last-layer Hessian, and the Jacobian is computed by backpropagation for each test input. Libraries that implement Laplace for PyTorch follow exactly this pattern, so the toy example is a faithful miniature of the real workflow.
Why Laplace is attractive in practice
The appeal is mostly operational. You do not retrain the model. You do not change the architecture. You do not run a sampling chain at prediction time, beyond cheap Gaussian samples or a closed-form linearised variance. If you have a production model and a stakeholder asks for uncertainty next week, Laplace is often the fastest honest answer.
It also composes well with other tools. A last-layer Laplace on top of a pretrained feature extractor gives calibrated uncertainty for a classifier with minimal code. Laplace can be combined with temperature scaling or other post-hoc calibration, although calibration should be checked on held-out data rather than assumed. And because the method is post-hoc, it can be applied retrospectively to models that were never trained with uncertainty in mind, which covers most models in existence.
Another underappreciated strength is interpretability of failure. When Laplace gives absurdly wide or narrow uncertainty, the Hessian usually explains why: a flat direction, a poorly chosen prior variance, or an undertrained optimum. That diagnosability is a real advantage over methods whose uncertainty is a black box.
Limitations and common pitfalls
Laplace is a local, unimodal, Gaussian approximation to a posterior that is typically multimodal and non-Gaussian. That sentence contains most of its failure modes.
First, multimodality. A neural network posterior has many modes connected by symmetries and by genuinely different solutions. Laplace picks one mode, the one optimisation found, and ignores the rest. If different modes disagree on a prediction, Laplace will understate uncertainty, sometimes badly. Ensembles capture some of that between-mode disagreement; Laplace alone does not.
Second, Gaussianity. The true posterior around a mode may be skewed or heavy-tailed, especially when the data are scarce relative to the parameters. A Gaussian fitted by curvature at the mode can badly misjudge tail probabilities, which are exactly what matter for risk-sensitive decisions. Laplace uncertainty is best treated as a relative signal, which inputs are more uncertain than others, rather than as exact probabilities.
Third, the prior matters more than it looks. The prior variance sets the regularisation strength during MAP training and the Hessian damping afterwards. A prior that is too tight makes Laplace overconfident; a prior that is too loose lets flat directions dominate and makes uncertainty explode. Tuning the prior by marginal likelihood, which Laplace itself can approximate, is principled but adds another optimisation loop that is easy to get wrong.
Fourth, the MAP must actually be a minimum. If training stopped early, or if the optimiser settled in a sharp valley it only partially explored, the Hessian at that point may not describe the local posterior at all. Checking that gradients are small and that the Hessian approximation is positive definite is not pedantry; it is a precondition for the method to make sense.
Fifth, scale. For large networks, even KFAC requires care: computing Jacobians per test input, storing curvature factors, and inverting them all cost memory and time. Last-layer Laplace avoids most of this, but it also assumes the feature extractor is certain, which is false for out-of-distribution inputs where features themselves shift. The approximation you can afford may not be the approximation you need, and that gap should be acknowledged.
A final pitfall is evaluation. Laplace predictive probabilities should be checked with proper scoring rules, calibration curves, and out-of-distribution detection benchmarks, not with accuracy alone. A method can keep accuracy identical while making uncertainty much better or much worse, and only uncertainty-aware evaluation will show the difference.
When to use Laplace, and when not to
Use Laplace when you have a trained model, you need uncertainty quickly, you can afford a Hessian approximation that matches your model size, and you will validate calibration on held-out and shifted data. It shines for last-layer uncertainty in classifiers, for regression with a clear likelihood, and for retrospective uncertainty on models you cannot retrain.
Do not use Laplace as your only uncertainty method when the decision is high stakes and modes are known to disagree, when the posterior is strongly non-Gaussian, or when you need exact tail probabilities. In those cases, combine it with ensembles or sampling, or use a method that represents multimodality explicitly. Laplace is a strong baseline and a useful component, not a complete Bayesian solution.
A practical checklist before reporting Laplace results:
- State the Hessian approximation: diagonal, KFAC, last-layer, or low-rank.
- State the prior and any damping, because both shape the covariance.
- Verify the MAP is a genuine minimum with small gradients.
- Evaluate calibration and out-of-distribution behaviour, not just accuracy.
- Report uncertainty as approximate and relative where the Gaussian assumption is doubtful.
Takeaway
The Laplace approximation earns its place by turning a trained model into an approximate Bayesian model with only local information: the optimum and the curvature around it. That is both its elegance and its limit. Curvature tells you which directions the data constrained and which it left free, and the inverse Hessian turns that into a posterior you can actually compute. Used with a clear statement of its Hessian approximation, its prior, and its calibration, it gives useful uncertainty for free. Used as if it were the exact posterior, it will quietly understate what the model does not know. The difference is not in the formula; it is in how honestly you check it.
Connect with me on LinkedIn: linkedin.com/in/umeshtharukaofficial
Top comments (0)