Every sequence model is trained on the true previous token and then asked to generate from its own. The objection has a name - exposure bias - and almost everything said about it is quoted rather than measured. The reason it gets argued is that scoring a model on its own generations needs the right answer for prefixes that never occurred.
Make the data process a finite-state machine and that problem goes away:
process = { V, S, start, emit[S][V], step(s, y) }
p_true(. | history) = emit[ fold(step, start, history) ] // for ANY token string
Both loops are then the same functional - expected KL from the true conditional - differing in exactly one line: whose tokens the prefix is drawn from. Every figure is a dynamic program over the joint (true state, model context), with no sampling noise anywhere. The control that pins it: a model that can represent the process has a gap of 3.1e-33. Dependency-free JavaScript: https://dev48.infy.uk/dl/day68-teacher-forcing.html
The number reported to justify the cure does not predict the cure
Ninety processes, a 0.05 grid over epsilon on each, one question per process: does scheduled sampling's best epsilon beat plain teacher forcing on free-running excess? It wins in 28 of them, by up to 31.1%, and loses by up to 376%. Then: which quantity, measurable before you try anything, would have told you which?
| predictor, measured before trying the cure | winners | losers | AUC |
|---|---|---|---|
| exposure gap RATIO (free / teacher-forced) | 3.33 | 3.52 | 0.529 |
| absolute gap (free - teacher-forced) | 0.19 | 0.07 | 0.855 |
| 1 / switch rate (how long the process remembers) | 11.73 | 6.27 | 0.770 |
| plain teacher-forced excess (how bad the model is) | 0.08 | 0.03 | 0.956 |
Chance is 0.500. The exposure gap ratio is the number people report to argue that exposure bias needs addressing, and it carries almost no information about whether addressing it will help. How bad the model already is under plain teacher forcing carries nearly all of it.
The fixed point is not a minimum of anything it computes
Scheduled sampling draws a hard token, so nothing differentiates through the draw and the applied gradient treats the rollout's own weights as constants. Run the procedure to its fixed point at epsilon = 0.5: the applied gradient is 4.7e-17 - it has stopped - while central differences on the exact expected loss give 3.0e-3, and one step along that direction lowers the loss it is evaluating from 1.50634701 to 1.50628351. Asserting both at the same point turns "the estimator is biased" into a test that would fail if it were not.
Two things I had to take back
The first pass swept three values of epsilon on one process, found scheduled sampling losing at every one, and was going to say it never helps. The fine grid is what caught it, and the assertion set now includes one that fails if the page goes back to claiming that.
The second is the RNN, which exists so a backward pass is real - every tensor within 1e-9 of central differences - and which was supposed to demonstrate the gap. On a 2-mode latch it came back at 0.999x, per-seed 0.995 to 1.005. Two hidden units are well specified for a 2-mode latch, so there was no model error to expose. The gap had to be earned honestly, by starving it: six modes, the same two units, 1.267x.
Part of a from-scratch series - one deep-learning idea a day, computed in-browser: https://dev48.infy.uk/deeplearningfromzero.php
Top comments (0)