When a Compile-Fail Test Fails for the Wrong Reason

I found a test recently that was doing exactly what it was supposed to do: it failed to compile.

The problem was that it failed before it reached the thing I thought I was testing.

That was a useful reminder. With compile-fail tests, “the compiler rejected it” is not enough. The reason for the rejection matters.

I had a few cases like this while hardening some small Rust types. The production code was mostly fine. The stale part was the verification around it.

The examples below are simplified, and the type names are intentionally generic.

The compiler never reached the privacy boundary

Imagine a public type with a private field and const-generic bounds:

pub struct BoundedValue<const MIN: u64, const MAX: u64> {
    value: u64,
}

Code outside the defining module should not be able to construct it by writing the field directly. A compile-fail test for that boundary might start like this:

let _ = BoundedValue {
    value: 5,
};

It certainly does not compile.

But in the case I was reviewing, the omitted const parameters caused a type-inference error first. The compiler had rejected the program without actually proving that direct field construction was blocked.

The test looked healthy because it failed. The diagnostic told a different story.

Making the const parameters explicit removed the unrelated ambiguity:

let _ = BoundedValue::<0, 10> {
    value: 5,
};

Now the test can actually reach the privacy boundary.

That is a much better negative test. Not because it fails “more strongly,” but because it fails for the property I meant to check.

Stale syntax can hide the same problem

I ran into another version of this with a named-field byte wrapper.

Simplified:

pub struct ByteBlock<const N: usize> {
    bytes: [u8; N],
}

An old compile-fail case was trying to construct it using tuple-struct syntax:

let _ = ByteBlock([0u8; 32]);

Again, the compiler rejected it.

But that test was not checking field privacy. It was only demonstrating that ByteBlock was not a tuple struct.

The type had changed shape at some point, while the negative test had kept an older idea of the API.

If privacy is the thing I want to verify, the misuse needs to match the real shape of the type:

let _ = ByteBlock::<32> {
    bytes: [0u8; 32],
};

From outside the defining module, that gives the compiler a chance to reject the actual private-field access.

The old test contained a real compiler error. It was just evidence for the wrong claim.

This is easy to miss with trybuild

I like trybuild for this kind of testing. A compile-fail case is compiled separately, and the compiler output is compared with the expected .stderr file.

That makes regressions in diagnostics visible, which is useful.

It also means I need to be careful when accepting or updating those snapshots.

If a stale test starts failing for an incidental reason and I simply bless the new .stderr, I can preserve a broken test while making the suite green again.

That is not a problem with trybuild. It is a problem with what I am asking the test to establish.

I used to look at a successful compile-fail suite and move on fairly quickly. Now I read the diagnostic and ask whether it actually reaches the boundary named by the test.

A compile-fail test should make one clear claim

The rule I use now is simple: I want each negative test to have one intended reason to fail.

Before I accept one, I check:

• Claim: What exact misuse is this test supposed to reject?
• Reachability: Can the compiler reach that boundary without an unrelated error firing first?
• Diagnostic: Is the observed error actually about the property I care about?
• Currentness: Does the test still match the current API, type shape, imports, and trait surface?

This sounds obvious in hindsight. It becomes less obvious after a test suite has lived through a few rounds of refactoring.

A stale import can fail.

Old constructor syntax can fail.

Missing generic information can fail.

An operation that used to be forbidden may even become valid later.

The word “fail” does not tell me which of those happened.

Runtime failure is a different boundary

One other case was a useful sanity check.

Something like this:

SomeType::new(0).unwrap();

might panic at runtime if new(0) returns an error or None.

But the program itself can still be perfectly valid Rust and compile successfully.

So that is not compile-fail evidence. It belongs to runtime behavior.

Again, obvious once you say it out loud. But mixing compile-time and runtime expectations is surprisingly easy when old tests and old comments have been copied forward for long enough.

The bigger lesson was about verification code

What stuck with me was not the individual Rust errors. It was the fact that verification code can drift while production code keeps moving.

Tests encode assumptions too.

They encode assumptions about type shape, visibility, crate paths, trait surfaces, invariants, and what the API is supposed to forbid.

Those assumptions can go stale.

A missing test is easy to notice. A test that still “works” can be more misleading because it gives you confidence without checking the contract you think it checks.

That applies beyond compile-fail tests. Property tests can model an old semantic rule. Fuzz targets can exercise an obsolete API. Snapshot-based tests can preserve an error that no longer represents the intended boundary.

So I have become a lot more suspicious of verification artifacts during hardening work.

I do not just ask whether the test suite passes anymore.

For negative tests, I also ask:

Why did the compiler say no?

If the answer is not the boundary I meant to protect, the test is not done yet.