Train a network on task A, then on task B. It does not degrade on A — it becomes task B: after 300 steps it scores 1.1139 on the old task and task B's own teacher scores 1.1180 on it.
See it: https://dev48.infy.uk/dl/day77-catastrophic-forgetting-and-ewc.html
Elastic weight consolidation prices the move away from A's solution using the Fisher information — one number per parameter. The task here is a teacher network over an enumerated 64-point input space, so the score is an exact mean KL (the teacher scores exactly 0, above a 1.0878-nat floor), and the Fisher is a closed-form sum rather than an estimate: three separately written code paths agree to 1.7e-16.
The headline: what EWC charges you is not a property of the network
Write the same model in different coordinates — an orthogonal change of basis that leaves every function it can compute untouched, and leaves plain fine-tuning landing in the same place to 4.9e-15, no further than a 1e-13 nudge moves it.
The diagonal of the Fisher becomes a different penalty. 84.8% of everything the Fisher's shape buys over a shapeless penalty of the same mass is gone, and the run lands 0.5670 from the plain-L2 run and 1.9185 from EWC's own.
The full Fisher is worse than the approximation
At all four budgets — 0.6754 against 0.5647. Not a rounding artefact: the second-order model the full Fisher minimises predicted the old task would lose 0.0226, and it lost 0.8796. A factor of 38.94.
The diagonal's over-charging accidentally keeps the run inside the radius where that model is still true. It is better because it is wrong in the right direction.
And the empirical Fisher — the one that actually gets implemented — is worth 0.1%: 0.5653 against 0.5647. At a model that has learned the task the two are bit-identical, exactly 0.
Per parameter it is nearly perfect, which is the confusing part
| what you rank parameters by | agreement with true curvature |
|---|---|
| Fisher diagonal | 0.997 (exactly 1.000 in the last layer) |
| weight magnitude | 0.474 |
| gradient size | 0.398 |
Both heuristics people reach for first score below a coin. So the Fisher is very nearly right about which parameters matter — and still loses 84.8% of its value to a change of basis, because "which parameters" is not a basis-free question.
Gradients checked at initialisation against forward-mode dual numbers to better than 1e-14 of the largest entry, where a central difference on the same code manages 1.3e-10.
Verifier 314 asserts, 240 in-page assertions, 0 failures.
Top comments (0)