I wrote down which test would catch each bug. The one aimed straight at it was blind.

I have three checks over the same game engine and they all look like determinism checks.
That raises a question the checks cannot answer about themselves: do they cover different
failure modes, or the same one three times?

So I wrote down, before running anything, which check would catch which bug. Then I
injected the bugs.

Two rows came out as predicted. The third did not, and the third is the only one worth
writing about.

Three checks that look alike

S, theSameSeedProducesTheSameRun: runs the engine twice in one runtime, compares.
G, ParityTraceTest: asserts a 600-frame, 11-checkpoint trace against a committed
golden file.
D, :web:parityCheck: diffs the JVM trace against the shipped TeaVM artifact's trace.

And a fourth column as a footnote, U, InterstitialCadenceTest: a plain unit test on
how often an interstitial cue fires.

Context: this is the rules engine of a small Android game, compiled a second time to the
browser with TeaVM. Same Java, two compilers. The determinism story is its own piece; this one is about the checks that came out of it.

Real patches, real rebuilds, real exit codes

matrix/run_matrix.sh, 134 lines: apply patch, rebuild both artifacts, run S, G, D and
U independently, revert under a trap.

They have to run separately, because :web:parityCheck declares dependsOn :engine:test.
A golden failure would stop the differential from ever running, and then I could not
attribute the columns: a blank cell would mean "did not run" rather than "did not fire".
ParityTraceTest writes its traces before it asserts, which is what makes the split
possible.

Every cell below is an exit status, not a judgement.

Three faults:

• F1: Rng delegates to java.util.Random instead of the written-out LCG. On the JVM that is behaviour-identical, because it is the documented algorithm. TeaVM's implementation is not the JDK's. This is the original bug, reproduced from source.
• F2: a shared constant, PLAYER_BOTTOM_MARGIN, 120f to 121f. Both builds compile it.
• F3: INTERSTITIAL_EVERY, 3 to 4. Behaviour the trace did not record.

The first run

fault	S	G	D	U
baseline	pass	pass	pass	pass
F1 runtime-supplied RNG	pass	pass	FAIL	pass
F2 shared constant	pass	FAIL	pass	pass
F3 untraced cadence	pass	pass	pass	pass, predicted FAIL

Read the two that worked first, because they are the boring half and they are the reason
the exercise was worth doing.

F1 is caught only by the differential. S compares the JVM to itself, and the golden
file's JVM side never moves, so neither can see a fault that only changes the browser.

F2 is caught only by the golden. Both builds move together, so the differential sees
two artifacts agreeing perfectly on the wrong number.

Neither check subsumes the other. Good.

The row I got wrong

F3 moving nothing in S, G or D was half expected. The trace recorded frame, score, streak,
run state, player position and obstacles, but not runsSinceInterstitial. Blind by schema.
Fine.

The half I did not expect was column U. InterstitialCadenceTest exists for exactly this
behaviour. It passed.

Its assertions referenced GameEngine.INTERSTITIAL_EVERY on eight lines. When the fault
moved that constant from 3 to 4, the test's expectation moved with it. It looped
for (run = 1; run < INTERSTITIAL_EVERY; run++) and then asserted "no cue after
INTERSTITIAL_EVERY runs". Both halves defined by the thing under test.

It checked the predicate against the constant, not against an answer key. A test written
that way cannot notice the constant changing, which is the only interesting way that
constant can be wrong.

So F3 had no catching check at all. Three trace-based checks blind by schema, and the one
test aimed directly at the behaviour blind because it had no independent expectation.

Fixing it, and the part I nearly got wrong twice

The obvious fix is both halves: give the test a literal answer key, and put the field in the
trace.

// before: expectation derived from the thing under test
for (int run = 1; run < GameEngine.INTERSTITIAL_EVERY; run++) { ... }

// after: expectation is an answer key
private static final boolean[] CUE_AFTER_RUN = {false, false, true};

The test now mentions INTERSTITIAL_EVERY exactly once, in a comment explaining why it must
not use it. The trace gained runsSinceInterstitial on both sides of the differential.

Here is the part worth the whole article. Before re-running, I wrote down what I expected:

F3 flips to U=FAIL only, and G and D stay green. The scripted run dies once, so the
counter reads 0 then 1 and never approaches any threshold. Recording it cannot make the
trace-based checks see a threshold change the run never exercises.

That prediction is the interesting one, because the naive expectation is the opposite. I had
just added the missing field to the trace. Surely now the trace-based checks can see it?

The re-run

fault	S	G	D	U
baseline	pass	pass	pass	pass
F1 runtime-supplied RNG	pass	pass	FAIL	pass
F2 shared constant	pass	FAIL	pass	pass
F3 untraced cadence	pass	pass	pass	FAIL

The prediction held. U fires. G and D stayed green even though the field they were
missing is now in the trace.

Because the scripted run only ever dies once. The counter reads 0, then 1. Changing the
threshold from 3 to 4 cannot move a number that never gets near either value.

Three ways a check is built blind

The first run showed two. The second showed the third, and the third only became visible
after the first two were fixed.

1. The schema. A trace cannot report a field it does not carry.
2. The expectation. If the expected value in an assertion is imported from the code under test, the assertion measures internal consistency, not correctness. Grep your suite for assertions that reference the constant they are testing.
3. The input plan. A check is only sensitive to behaviour its inputs actually exercise. Coverage of a field is not coverage of a behaviour. This is the one I would have missed entirely if I had stopped at the first fix and declared the residual closed.

The caveats this needs, in the body rather than a footer

Three faults is not a coverage measurement. It is three worked examples chosen to separate
four checks. A real matrix would keep going until it stopped finding new residual rows, and
mine stopped when I ran out of faults I could think of.

The predictions were written in a commit message and a chat log, not a pre-registration
anyone can audit. Take the first miss as the evidence they were genuinely made in advance,
since it is the row I would have quietly dropped otherwise.

And the defect was in a test, not in shipped behaviour. The game's cadence is correct. What
shipped broken was the confidence.

A matrix without a residual row is advertising. The row where nothing fires is the
deliverable.

---

Harness, patches and full output: github.com/egnaro9/tapdodge-engine, matrix/
Write-up: docs/field-note-fault-matrix.md

Comments

[Arhan Canli]

The third failure mode (input plan) has a cheap generalisation: for any threshold-style constant, a check only separates N from N+1 if the scripted run reaches N and N+1 events. Your run dies once, so it can't distinguish 3 from 4, but also not 2 from 3 or 3 from 2. If you extend the script to die 5-6 times and assert the cue pattern as a literal array (false, false, true, false, false, true), it covers a threshold change in either direction up to 6, and it also catches a counter that fails to reset after the cue, which a single death can never show.

On the "stopped when I ran out of faults I could think of" caveat: hand-picked faults are biased toward the ones you already built checks for. Generating them mechanically (every numeric literal in the engine nudged by +1 and -1, one rebuild per mutant) gives a residual list you didn't choose, and the rows where nothing fires become the to-do list. With a 134-line matrix script that's mostly a loop over a patch generator.

One question: for the parity differential, is the TeaVM trace diffed per checkpoint or only as a final hash? Per-checkpoint tells you which frame diverged first, which is the useful part for an RNG fault like F1.

[Erik Hill]

Per checkpoint. tools/compare_trace.mjs walks the two traces position by position and reports every differing field, so the output names the first frame that diverged rather than just failing one hash. There are 11 checkpoints over a 600-frame scripted run, weighted toward live play because the run dies around frame 420 and everything after that would record the same frozen state. The last one is kept on purpose so both builds also have to agree that game over is terminal and identical.

The literal-array point is better than what I shipped and I am taking it. My answer key is {false, false, true} for a single death, which pins 3 and nothing either side of it, and as you say it cannot see a counter that fails to reset. Extending the script to five or six deaths and asserting the whole pattern costs almost nothing and covers both directions plus the reset.

On hand-picked faults, that is the weakest part of the piece and you have named why. Nudging every numeric literal by +1 and -1 with a rebuild per mutant produces a residual I did not choose, which is the whole point, and treating the rows where nothing fires as the to-do list is the right framing.

[DEV SUPPORTS]

You need to verify your account.

tr.ee/dev-to