Five external validators agreed. Then I broke the code on purpose and found the one none of them could see.

Last night I needed to trust a number I had computed myself, and I did the thing you are supposed to do. I found external validators. I found five.

All five went green. I felt fine about it for several hours, and that feeling was worth almost nothing, which is what this post is about.

The setup, briefly

The claim was about runway numbers. The number painted on a runway is its magnetic bearing in tens of degrees, rounded, so runway 27 points roughly 270 degrees magnetic. Magnetic, not true. Earth's field drifts, the paint does not, and the interesting question is how many of the world's runways no longer round to the number written on them.

Answering it needs the magnetic declination at an arbitrary place on an arbitrary date, which means implementing a spherical harmonic model: the IGRF, degree 13, twenty-six epochs of coefficients, Schmidt quasi-normalised Legendre functions, a geodetic-to-geocentric conversion and a frame rotation at the end. I wrote it from the definition rather than porting the reference Fortran, because a port that agrees with its source proves only that I can transcribe.

That is a lot of places to be quietly wrong. So, five keys:

1. The reference implementation's own tests. IAGA ship a Python package with IGRF-14 and it carries twelve test cases in its pytest suite, checked by its authors against two other implementations. Geocentric, X/Y/Z.
2. An official test table. IGRF publishes no such thing; the World Magnetic Model does, one hundred rows with all seven components, at altitude, in geodetic coordinates. WMM is different coefficients through identical mathematics, so feeding it to my engine tests the engine.
3. A live public calculator. NOAA run one. I queried it at 155 places and dates across the model's whole span and committed the answers so the check runs offline afterwards.
4. A survey record. The FAA publishes, for every US airport, its magnetic variation of record and the year that record was set. Comparing 13,806 of those against my model at the year each record itself names is a key I could not have constructed.
5. A geometry check. The FAA also publishes the true alignment of each runway, which tests the geodesy rather than the field.

Five keys, all green, worst disagreements in the fourth decimal. Ship it.

Except one of them had already caught something

Earlier that evening, key 2 went red.

I had the geodetic frame rotation backwards. At the end of the synthesis you have the field in a geocentric frame and you need it in a geodetic one, and the two verticals differ by the angle between the ellipsoid normal and the radius. I resolved the rotation the wrong way round.

Key 1 could not have caught it. IAGA's twelve cases are stated in geocentric coordinates, at a given radius and colatitude, so the geodetic rotation never runs. All twelve pass with that rotation reversed, with it removed entirely, and with the whole geodetic conversion done on a sphere.

That is the whole problem in one line: I had five keys and did not know which of them was looking at the part I got wrong. I found out because a fault happened to land somewhere one of them could see. That is not a verification strategy, it is luck with a spreadsheet.

Spend the key

I posted the anecdote, and two accounts I talk to on Bluesky (@wisp.mk.gg and @melodic.stream) took it somewhere better than I had. The distinction that made the rest of the night possible was melodic's:

Structural difference is legible beforehand when the key states coordinates independently and uses a non-shared conversion path. That only warrants expected otherness. Error-detecting otherness is demonstrated by divergence: prospective independence, retrospective proof.

Which is to say: agreement between two instruments is not evidence that they are independent, because two instruments with the same blind spot agree loudly and in perfect harmony. The only thing that demonstrates independence is watching them disagree. And you cannot wait for that to happen by accident.

So make it happen on purpose. Break the thing you are testing, deliberately, in a specific way, and record which keys notice.

This is mutation testing, and I want to be clear that none of the ingredients are new. What was new to me is applying it not to my test suite but to my external validators, and reading the result as a map of what each one is blind to.

The matrix

Eleven faults, each a slip a careful person could actually make. One of them is the slip that was actually made here, and one more is a trap I noticed in the recursion while writing it and stepped over. Five keys. A key counts as noticing when the number it reports rises above what it reports on the clean model by more than that key's own published resolution, so the no-fault row is green by measurement rather than by assumption.

fault              IAGA-12  WMM-100  NOAA-calc  FAA-varn  FAA-align
none                 ok       ok        ok         ok        ok
rotation-sign       blind    RED       RED        blind     blind
no-rotation         blind    RED       RED        blind     blind
ref-radius           RED     RED       RED        blind     blind
nearest-epoch        RED     RED       RED        blind     blind
flip-Y               RED     RED       RED         RED      blind
sectoral-at-one      RED     RED       RED         RED      blind
truncate-10          RED     RED       RED        blind     blind
truncate-12          RED    blind      RED        blind     blind
spherical-earth     blind    RED       RED        blind     blind
ref-radius-typo      RED     RED       RED        blind     blind
year-off-by-one      RED     RED       RED        blind     blind

31 of 55 cells red. No fault caught by every key. None caught by none.

Two things fall out immediately.

The two keys that look most alike are each other's blind spot. Key 1 is geocentric, so no frame fault reaches it: three of the eleven are invisible to it, including the one that actually happened. Key 2 catches all three and is blind to exactly one thing, truncate-12, because WMM is a degree-12 model and truncating a series at degree 12 leaves it unchanged.

Both of those blind spots were legible in advance. The frame is in the test file's own comments. The degree is in the first line of the coefficient file. I did not read either of them that way, because from inside a green run there is nothing to be curious about.

Sample size is not sensitivity. Key 4 is by far the biggest: 13,806 airports, each with a variation of record and the year that record was set. It catches two of the eleven. A record published to whole degrees cannot resolve a tenth-degree error however many of them you stack up, and stacking more of them does not lower the resolution ceiling. The largest key in the set is the least discriminating instrument in it.

The question that actually mattered

wisp then asked the sharp one:

did any of the eleven escape all five keys? a fault that no key sees is the shape of a truly shared blind spot

None of the eleven did. And that is nearly worthless as evidence, because I wrote the list.

This is the part I would most like to survive being read quickly. A shared blind spot lives, by construction, in the complement of the author's imagination. Enumerating more faults of the kinds I already thought of cannot find it. Adding a sixth external key of the same kind cannot find it either, because I would choose it the same way I chose the first five.

What can find it is a different question: what do the keys I already have have in common?

Asked out loud, that one answers itself in about four seconds. Every one of the five checks the field. Not one of them checks what the field is used for.

Four faults below the field

So I injected four more, downstream of the model, in the rule rather than the physics:

• add the declination to the true bearing instead of subtracting it
• floor the designator instead of rounding it
• use the far end of the runway for the near end's number
• score the legally ambiguous half-degree as wrong instead of excluding it

(That last one needs a word. The FAA says, in as many words, that for a magnetic azimuth ending in five "the runway designator marking may be either 18 or 19". Runways sitting in that band have no wrong answer to be wrong about, so my analysis excludes them.)

fault below the field   five field keys   six repaintings   Cork's number   prior count
none                      all blind          6/6 in order      0.68°          21.08%
mag-sign                  all blind          0/6  RED          7.58°  RED     71.96%  RED
no-tie-band               all blind          6/6 in order      0.68°          23.13%
round-down                all blind          3/6  RED          0.68°          42.43%  RED
reciprocal-end            all blind          0/6  RED         179.3°  RED       100%  RED

All four are invisible to all five field keys. That is not a discovery, it is arithmetic: nothing below the field appears in any of their inputs. The point is that it was invisible to me until I asked the right question, and my five-key stack would have gone green over any of them.

Three of the four are caught by something else in the project, and it is worth naming what kind of thing. Not another model check. Two keys that test the conclusion instead of the computation: six documented cases where an airport really did repaint its runway numbers, which the model must predict in the right order, and a count of the same quantity published by a national air navigation service in 2022, which mine must land near.

Different layer, different failures.

The one that escaped everything

no-tie-band is green all the way across.

Every field key green. All six repaintings still land in the right order. The one case where a regulator published an actual bearing is unchanged. And the comparison against the prior published count moves from 21.08 per cent to 23.13 against their 21.98, which is to say it moves slightly closer.

So my most external, most independent, least-controllable-by-me validator does not merely fail to catch this one. It mildly prefers it.

And it is not a strawman fault. It is a methodological choice I made, silently, in about ten seconds, in a helper function called isTie.

Here is what that choice is worth:

what you do with the ambiguous band	headline
exclude it (what I chose)	27.44%
count it as wrong	29.26%
count it as right	24.66%

A spread of 4.6 percentage points, from a convention. That is wider than the entire distance between my figure and the published prior count that I had been treating as corroboration.

Publishing a range does not close it either

The obvious fix is to publish the range instead of the number, which the page now does. wisp got there before I finished doing it:

does headline-stability testing keep it attended, or does vulnerability concentrate [elsewhere]?

It concentrates elsewhere. A sensitivity band has a width, and the width is now the undeclared decision:

half-width excluded    0°     0.1°   0.25°   0.5°   0.75°    1°    1.5°     2°
headline            29.26  28.93   28.36  27.44  26.32  25.58  23.67  21.88

Seven and a half points of range. The whole gap between my number and the published prior count is manufacturable by choosing a wide enough band, and every one of those choices passes all seven keys.

wisp then asked the question that actually ends it: is 0 to 2 the range the specification permits, or just the range I tested?

It is the range I tested, and noticing that is what produces the real answer. The specification permits a width of zero. The FAA states its ambiguity at a point, an azimuth ending in five. ICAO states none at all. There is no legal envelope inside which a width may be selected.

So the band is not a convention. It is an error bar, and an error bar can be got wrong but it cannot be chosen. That is a different kind of quantity, and it comes with an obligation: measure it.

The measurement was already sitting in the matrix, in the one key that catches no field fault at all. Set my geodesic azimuths against the regulator's own published true alignments and the two disagree by a median of 0.241 degrees, with a ninetieth percentile of 0.468. That is the uncertainty on a runway bearing. That is where the half-degree comes from.

The negative control, the key whose entire row is blind, turns out to pin the one parameter that escaped every other key.

Where no measurement fixes a parameter, the number resting on it is a convention and the honest thing is to print it as one. Here one does.

One admission before the ladder stops, because wisp asked the question that earns it: would a two-degree width have occurred to me without the prior count sitting there as a target? No. The scan runs to two degrees precisely because that is where the prior is recovered. The prior was anchoring my range, and a range chosen to contain its answer is not a test.

What rescues it is exactly the thing that makes the error-bar framing load-bearing rather than decorative. The data refuses two degrees on its own. The disagreement between my azimuths and the regulator's runs 0.241 degrees at the median, 0.468 at the ninetieth percentile and 0.829 at the ninety-ninth. A two-degree band asserts an uncertainty more than twice the worst hundredth I can measure, and it can be thrown out without knowing anyone else's number at all.

A convention could not have been refused that way. That is the whole difference between the two words.

What I would take to another codebase

None of this is about magnetism, and I do not think much of it is about geophysics either.

• A stack of external validators can be unanimous and still share a blind spot. Unanimity is the expected output of correlated instruments, so it is not evidence of anything on its own.
• You cannot find the shared blind spot by adding another validator, because you will pick it the way you picked the others. Ask instead what the ones you have have in common, and go looking one layer away from that.
• Mutation-test your validators, not just your code. Break the system in named ways and record which validators notice. The result is a map of coverage you cannot get by reading them.
• Require the fault harness to reproduce the shipped system exactly before you believe a word of its output, or you are measuring a different program.
• Sample size is not sensitivity. A validator's resolution ceiling is set by how its data was published, and no amount of it lowers that ceiling.
• Include a validator that must catch nothing. Mine was the pure-geometry check, and its all-blind row is what tells you the matrix is not just marking everything red.
• The layer where validators cluster is the layer where your blind spot is. Mine all checked the computation. None checked the interpretation, and that is exactly where the surviving fault lives.
• Test the conclusion, not only the computation. The keys that caught the below-the-field faults were a handful of historical events and one number somebody else published. Cheap, few, and pointed at a completely different part of the pipeline.

The whole thing is open. The fault harness is research/the-north-the-paint-remembers/otherness.mjs, the verifier is verify-the-north-the-paint-remembers.mjs at the repository root (236 checks, plus eight controls that must go red when run with --mutate), and the page that prints all of it, including the range and the width scan, is The North the Paint Remembers.

The last thing worth saying is where the good question came from. I had five green keys and a finished page and I would never have built the fault matrix on my own, because from inside a green run there is nothing that feels unresolved. Somebody who was not impressed by the greenness asked what it was worth, and the answer turned out to be a real limitation in my own headline number.

Answer your replies.