“Formatting got worse after the switch” is not a finding. It becomes one when you can say which requirement broke, on what proportion of a defined sample, with an interval wide enough to be honest and narrow enough to act on. All of that is arithmetic you can do on a laptop.
Define conformance as a validator, not a feeling
An output conforms if a deterministic function returns true for it. If you cannot write that function, you do not have a formatting requirement; you have a preference, and a preference cannot regress measurably. Write the validator first.
Score per requirement, never per output. A single conforming boolean tells you the number moved and nothing about what to fix, and it hides the common case where one rule collapses while the rest improve slightly, leaving the aggregate flat.
const REQUIREMENTS = {
parses: (s) => tryParse(s) !== null,
schema: (s) => schemaValid(tryParse(s)),
no_preamble: (s) => s.trimStart().startsWith("{"),
no_fence: (s) => !s.includes("\u0060\u0060\u0060"),
heading_count: (s) => (s.match(/^## /gm) || []).length === 3,
date_format: (s) => /^\d{4}-\d{2}-\d{2}$/.test(tryParse(s)?.date ?? ""),
};
// Returns one row per output: { parses: true, no_fence: false, ... }
function score(output) {
return Object.fromEntries(
Object.entries(REQUIREMENTS).map(([k, f]) => [k, Boolean(f(output))]),
);
}
The per-requirement table is the deliverable. The headline number is a summary of it, and you should be suspicious of anyone who reports the summary without the table.
How to draw the sample
Sample from production inputs, not from your test fixtures. Fixtures are selected for being interesting and therefore over-represent the hard cases; the score you get from them is not the score your users see.
Stratify by input type if your traffic has obvious classes — document length bands, language, tenant tier, whichever dimension you believe affects formatting. Draw a fixed number from each stratum rather than sampling proportionally, so a small stratum is still measurable, then weight back to the traffic mix when reporting the overall figure. Record the sampling window with the result: a score measured on a Tuesday afternoon and one measured over a full week are not comparable, and someone will try to compare them.
The interval around a single score
Suppose you draw n = 200 outputs (an assumption — substitute your own) and 196 conform on the no_fence requirement. The point estimate is 196/200 = 0.98. The naive normal interval is a bad choice this close to 1; use the Wilson score interval, which behaves at the boundary.
Wilson 95% interval, z = 1.96, p = 0.98, n = 200
centre = (p + z^2/(2n)) / (1 + z^2/n)
= (0.98 + 3.8416/400) / (1 + 3.8416/200)
= 0.98960 / 1.019208
= 0.97095
half = (z / (1 + z^2/n)) * sqrt( p(1-p)/n + z^2/(4n^2) )
= (1.96 / 1.019208) * sqrt( 0.0196/200 + 3.8416/160000 )
= 1.92306 * sqrt( 0.000098 + 0.000024 )
= 1.92306 * 0.011045
= 0.021242
interval = [0.9497, 0.9922]
So a measured 98% on 200 samples is consistent with anything from about 95% to 99%. If someone then reports 96% on the new model from a similar sample, the intervals overlap heavily and you have learned nothing. That is the whole problem with informal before-and-after comparisons: the honest interval is usually wider than the difference people are arguing about.
How many samples to detect a real drop
Decide the effect size you care about first. Say a drop from 98% to 94% would matter operationally and anything smaller would not — both numbers are assumptions you are choosing, not measurements. Use the standard two-proportion sample size at 5% significance and 80% power:
Assumptions (chosen, not measured):
p1 = 0.98 baseline conformance
p2 = 0.94 the smallest drop worth detecting
delta = 0.04
alpha = 0.05 two-sided -> z_a = 1.96
power = 0.80 -> z_b = 0.84
p_bar = 0.96, q_bar = 0.04
term1 = z_a * sqrt(2 * p_bar * q_bar)
= 1.96 * sqrt(0.0768) = 1.96 * 0.27713 = 0.54318
term2 = z_b * sqrt(p1*q1 + p2*q2)
= 0.84 * sqrt(0.0196 + 0.0564)
= 0.84 * sqrt(0.0760) = 0.84 * 0.27568 = 0.23157
n per arm = (term1 + term2)^2 / delta^2
= (0.77475)^2 / 0.0016
= 0.60024 / 0.0016
= 375
Roughly 375 outputs per model, so 750 calls, to detect a four-point drop with independent samples. That is the honest price of the comparison, and it is why most teams quietly report a comparison they cannot support.
Pairing cuts the cost by an order of magnitude
You do not have to use independent samples. Run both models on the same inputs and the comparison becomes paired, which removes input difficulty as a source of variance. Now only the disagreements carry information, and you test them with McNemar’s test on the two discordant cells.
Same 200 inputs through both models:
new: pass new: fail
old: pass 174 18 <- b = 18
old: fail 4 4 <- c = 4
Only b and c matter. n_d = b + c = 22.
Under "no difference", each discordant pair is a coin flip.
Two-sided exact binomial, P(X <= 4 | n = 22, p = 0.5):
sum of C(22,k) for k = 0..4 = 1 + 22 + 231 + 1540 + 7315 = 9109
2^22 = 4194304
one tail = 9109 / 4194304 = 0.00217
two-sided p = 0.0043
Twenty-two disagreements out of two hundred inputs is enough to reject the null at any conventional threshold, where the independent design needed 375 per arm. Pair whenever you can — and you almost always can, because a migration comparison is exactly the situation where the same inputs are available to both sides. This is the same shape of argument used in detecting regression baseline drift.
Three ways the number lies
- Changing the validator mid-comparison. If you tightened
no_preamblewhile investigating, the before and after scores measure different things. Version the validator alongside the score and refuse to compare across versions. - Averaging across strata. A model that formats English perfectly and Thai badly can post the same aggregate as one that is mediocre at both. Report per stratum; see the language-specific version of exactly this failure.
- Sampling after a retry. If your client retries on a parse failure and you measure the final output, you are measuring the retry loop, not the model. Instrument the first attempt, and record the retry rate as its own number — a stable conformance score sitting on top of a doubled retry rate is a cost regression wearing a quality score’s clothes.
A fourth, subtler one: reporting the point estimate without the interval. Every number above has a width, and the width is what determines whether a stakeholder should act. “98%” invites a comparison with last quarter’s “99%” that the data does not support; “98% (95% CI 95.0–99.2, n=200)” makes the same comparison visibly unavailable. Put the interval and the sample size in the same string as the score, in the dashboard and in the migration document, so that they cannot be separated by copy-and-paste.
Finally, decide the decision rule before you run anything. Write down, in advance, the conformance level below which you will not migrate and the level below which you will roll back — as numbers, with the sample size that will produce them. A threshold chosen after seeing the result is not a threshold; it is a narrative. The arithmetic above exists so that the threshold can be argued about while it is still cheap, which is before anyone has an answer they are attached to. Write it into the migration document next to the sample size, name the person who signs off on it, and treat a request to revise it after the run as the significant event it is rather than as a detail.
Top comments (0)