The test asserted exactly what could not fail

Over the last month I found correctness bugs in three different quantum
software projects. All three had tests covering the broken code. All three
tests passed.

That is not a story about quantum computing. It is a story about a way of
writing assertions that feels like testing and is not, and I keep running into
it, so I want to write it down.

The shape of it

Here is the one that made it click. It is from a compiler framework that exports
quantum circuits to OpenQASM:

assert!(qasm.contains("OPENQASM 3.0;"));
assert!(qasm.contains("qubit[2] q;"));
assert!(qasm.contains("h q["));
assert!(qasm.contains("cx q["));

Look at the third line. It checks that an h gate was emitted. It stops the
string one character before the qubit index.

The exporter was putting every gate on the wrong qubit. It derived the index
from a counter that went up once per gate instead of reading the operands, so
the Nth gate landed on qubit N. Three gates on one qubit came out on three
different qubits.

That test passes on the broken output. It would pass on almost any output. It
is not testing the export, it is testing that the function returned a string
with some letters in it.

The same shape, twice more

In another project, a Rust simulator, the Rz gate had a sign error: Rz(theta)
was implemented with the sign of theta flipped, so Rz(pi/2) gave you S dagger
where it should give you S.

The test:

assert_eq!(magnitudes(result), expected_magnitudes);

Magnitudes. The bug was in the phase. A sign flip does not move a magnitude, so
the test could not see it, and it had been passing since the code was written.

Back in the first project, a different test, this time for a pass that rewrites
gates into a hardware native set:

assert!(names.contains(&"quantum.rz".to_string()));
assert!(names.contains(&"quantum.sx".to_string()));
assert!(names.contains(&"quantum.cx".to_string()));

The pass is supposed to replace a gate with an equivalent sequence. This checks
that the new sequence showed up. It never checks that the old gate went away.

It did not go away. The pass inserted the replacement and left the original in
place, so the circuit ended up doing both. A T gate became an S gate. That one
had been shipping in a release.

Three ways to write a test that cannot fail

Once you have seen them together the pattern is easy to name.

Asserting a magnitude when the defect is a sign. The assertion projects away the
exact dimension the bug lives in.

Cutting the string before the field that breaks. contains("h q[") is one
character short of useful, and the missing character is the whole subject of the
function under test.

Asserting presence when the defect is absence. A correct rewrite is two claims:
the new thing appears, and the old thing is gone. Checking only the first half
lets every "forgot to delete" bug through.

There is a fourth one I found in the same file, and it is the pure form:

let _ = (result, rz_before);

That is the last line of a test called test_rz_stays_untouched. It computes the
before state, runs the pass, and then discards both values. It asserts nothing.
It passes unconditionally. Line coverage counts it as covered.

What I do about it now

I have one question I ask, and it is boring, and it works:

What change to the code under test would make this assertion fail?

If I cannot answer quickly and concretely, the assertion is decoration. For the
export test the honest answer was "almost nothing", since any output with an h
somewhere passes. For the magnitudes test the answer was "any change that
affects magnitude", which excludes the entire class of bug that was actually
there.

The other thing I do now is run the test against the broken code on purpose.
When I sent a patch for the sign error I described above, I flipped the sign
back before opening the pull request, watched my new test fail, and then flipped
it again. Two minutes. It is the only way to know the test has any power, and
you can only really do it while the bug is still in front of you.

That is also the honest reason these tests exist in the first place. Nobody sets
out to write a test that cannot fail. You write it after the code already works,
so it passes on the first run, and a test that passes on the first run never
shows you what it would have caught.

Why this bites harder in numerical code

In ordinary application code a wrong result usually turns into something visibly
wrong: a crash, an exception, a null, a screen full of garbage.

In numerical and quantum code, a wrong result is a perfectly well formed number.
An amplitude with a flipped sign is still a valid amplitude. A circuit on the
wrong qubits is still a valid circuit, and the file it exports still parses.
There is no shape you can check for that says "this is broken", which is exactly
why weak assertions survive here: nothing downstream complains about them either.

So the burden falls entirely on the assertion. If the assertion looks away, the
bug ships, and it ships quietly.

The reports

If you want the details, they are public:

Qiskit, sign of the dividend in a Rust transpiler pass, fixed in 2.5.1:
https://github.com/Qiskit/qiskit/issues/16594

Lift, the export and decomposition bugs above:
https://github.com/rustnew/Lift/issues/2
https://github.com/rustnew/Lift/issues/3

The tool I use to find them is a differential fuzzer, which is a longer post:
https://github.com/cleitonaugusto/CleitonForge