DEV Community

Michael Yousrie
Michael Yousrie

Posted on

A Pest test that asserts nothing: toContain takes needles, not a message

#laravel #php #webdev #testing

I added a test last week that was doing absolutely nothing, and it passed every time I ran it.

The rule it was guarding is boring: no em dashes in anything my project publishes. I write marketing copy alongside the code, em dashes are the single most obvious tell that a machine wrote something, and I'd already had to go back and rewrite 126 of them out of published posts once. So I wanted a test, not a habit.

Here's what I wrote:

it('uses no em or en dashes in copy', function () {
    foreach (config('comparisons') as $key => $comparison) {
        foreach (allStrings($comparison) as $string) {
            expect($string)
                ->not->toContain("\u{2014}", "An em dash appears in {$key}: {$string}")
                ->not->toContain("\u{2013}", "An en dash appears in {$key}: {$string}");
        }
    }
});
Enter fullscreen mode Exit fullscreen mode

Green. Every run. And completely inert.

toContain takes needles, not a message

Most assertion libraries have a second parameter for a custom failure message. PHPUnit does. expect($x)->toBeTrue('why this matters') does. So when I typed a helpful string as the second argument, I was pattern matching off everything around it.

Pest's toContain isn't that shape. It's variadic:

public function toContain(mixed ...$needles)
Enter fullscreen mode Exit fullscreen mode

Every argument is a needle. My "message" was a second thing it went looking for.

The positive form does what you'd guess: it requires all of them.

expect('alpha beta')->toContain('alpha', 'beta');  // passes
expect('alpha beta')->toContain('alpha', 'zzz');   // fails
Enter fullscreen mode Exit fullscreen mode

The negated form is where it gets you. not negates the whole conjunction, so ->not->toContain(a, b) asserts "it is not the case that both are present". One missing needle satisfies that on its own, and the other needle is never examined.

I ran the four cases against Pest 4.4.1 rather than reasoning about it:

toContain(present, absent)        => FAILED
not->toContain(present, absent)   => PASSED   <-- the one that bit me
not->toContain(absent, absent)    => PASSED
not->toContain(present)           => FAILED
Enter fullscreen mode Exit fullscreen mode

Line two is the whole bug. My failure message was, by definition, never in the string being checked. That made the conjunction false, which made the negation true, forever, no matter how many dashes were sitting in the copy.

The single-needle version on line four works exactly as expected. That's what makes this so easy to ship: the code you wrote first was correct, and then you improved it by adding a message.

The fix

Route it through a plain predicate, which does take a message:

expect(str_contains($string, "\u{2014}"))
    ->toBeFalse("An em dash appears in {$key}: {$string}");
Enter fullscreen mode Exit fullscreen mode

Slightly uglier. Actually asserts something. If you'd rather keep the fluent style, drop the message and let the diff speak:

expect($string)->not->toContain("\u{2014}");
Enter fullscreen mode Exit fullscreen mode

Both are fine. The one to avoid is the one that reads best.

Why I believed it

The fix took a minute. The harder question is why I trusted a test I'd never seen fail.

A test that only ever passes is indistinguishable from a test that can't fail. Green tells you nothing on its own. It only means something if you know the test is capable of turning red, and for most tests you learn that by accident: you write it, it fails, you fix the code, it goes green. The red came free.

Guard tests don't work like that. You write them for a bug you've already fixed, or for a rule you're trying not to break in future. They're green from birth. Nobody ever sees them fail, so nobody finds out they can't.

So now, whenever I write one, I break it on purpose first:

# put the banned thing back
git stash
php artisan test tests/Feature/BlogClaimsTest.php   # expect FAILURES
git stash pop
php artisan test tests/Feature/BlogClaimsTest.php   # expect green
Enter fullscreen mode Exit fullscreen mode

For the dash test I injected a real em dash into one config value and watched it fail with the message I'd written. Ten seconds. It's the same red-green loop as TDD, just applied after the fact, and it's the only thing that separates a guard from a comment.

While I was doing this I found a second one in the same file that had the identical mistake, and I'd have shipped both.

One more trap in the same family

The other guard I wrote that day banned a phrase from my published copy. It flagged something correct on the first real run, and the reason is worth mentioning because it's the opposite failure.

I'd been claiming a feature in a blog post that my product doesn't actually have. Reasonable response: ban the phrase in a test. Except the phrase also appeared in a perfectly true sentence about a competitor's plan, which does have that feature. The test was doing something, and what it was doing was wrong.

So the two failure modes sit next to each other. A guard that asserts nothing tells you your copy is clean when it isn't. A guard that's too blunt tells you your copy is broken when it's fine, and the fastest way to make that alarm stop is to delete the test. Neither one announces itself.

The version I kept scopes the check per line, so it only fires when the banned phrase shows up on a line that's talking about my own product. More code, narrower blast radius, and it survived contact with the real corpus. That last part is the test of a guard: run it against everything you've already published, not just against the case you invented it for. If it flags something true, it's not ready yet.

Summary

  • Pest's toContain is variadic. There is no message parameter.
  • ->not->toContain($needle, 'message') passes whenever the message is absent, which is always. The assertion is dead.
  • Use expect(str_contains($h, $n))->toBeFalse('message') if you want a message, or a bare single-needle not->toContain($n) if you don't.
  • Break every guard test on purpose once, before you trust it. Green from birth means nothing.
  • Then run it against your whole existing corpus. A guard that flags a true statement gets deleted by whoever hits it next.

I found both of these while auditing my own marketing copy for claims that had drifted from what the product does, which turned out to be a much bigger problem than the tests were. That's a different post.

Top comments (0)