DEV Community

Hammad Shams Uddin
Hammad Shams Uddin

Posted on

My check said "all 17 articles". There were 18.

#testing #devops #php #webdev

Every morning a script of mine fetches each of my published articles and checks that the outbound links are still followed. Yesterday's run ended with this line:

dev.to: links on all 18 published articles are still followed.
Enter fullscreen mode Exit fullscreen mode

Today's run, same script, nothing removed:

dev.to: links on all 17 published articles are still followed.
Enter fullscreen mode Exit fullscreen mode

I had published one article since. The honest number was 18 and rising. The word doing the damage is all.

The loop 

foreach ($articles as $a) {
    if (empty($a['published']) || empty($a['url'])) { continue; }

    $html = DevTo::fetch((string) $a['url']);
    if ($html === '') { continue; }        // <-- here
    $checked++;

    // …find every link to my site, look at its rel attribute…
}

// later
$did[] = 'links on all ' . $checked . ' published articles are still followed.';
Enter fullscreen mode Exit fullscreen mode

One article did not come back — a throttle, a hiccup, it does not matter. The loop skipped it, $checked stayed where it was, and the summary printed $checked as if it were the total.

So the sentence was not "17 of 18 are fine, 1 is unknown". It was "all 17", which asserts something about a set I never measured. The one article that could have gone bad was precisely the one that was not looked at, and the report was worded to sound most confident about exactly that gap.

Why the denominator is the whole thing

$checked answers "how many replied". The summary claims to answer "how many exist and are fine". Those are the same integer only when nothing fails, which is the one case where you do not need the check.

A count of successes cannot describe a set unless you also carry the size of the set:

$checked   = 0;
$unchecked = [];

foreach ($articles as $a) {
    if (empty($a['published']) || empty($a['url'])) { continue; }

    $html = DevTo::fetch((string) $a['url']);
    if ($html === '') { sleep(2); $html = DevTo::fetch((string) $a['url']); }
    if ($html === '') { $unchecked[] = (string) $a['url']; continue; }
    $checked++;
    // …
}

$total = $checked + count($unchecked);
Enter fullscreen mode Exit fullscreen mode

Now the report can say what actually happened: links still followed on 17 of 18; 1 could not be fetched twice and was NOT checked. And when most of them fail, that is no longer a quiet line at the bottom of an email — it is escalated, because a check that could not run is not a check that passed.

This is the third time

The same shape has bitten this project three times in a month, in three different files:

where what it printed what was true
sitemap check "1 of 54 child sitemaps is empty" the child was throttled, not empty
robots.txt check passed, six days running it was reading an attacker's planted file
link check "all 17 articles" there were 18; one was never fetched

Every one of them turned I could not measure this into I measured this and it is fine. That direction of error is the dangerous one: a false alarm gets investigated, a false all-clear gets filed.

So I wrote the test as a source check rather than a behaviour test, because the bug lives in a shape, not in a value:

foreach ($lines as $n => $l) {
    if (!preg_match('~^\s*if \(\$html === .{2}\) \{ *continue; *\}~', $l)) { continue; }
    $silentSkips[] = sprintf('line %d drops a page without recording it: %s', $n + 1, trim($l));
}
$check('no fetch failure is skipped without being counted', $silentSkips, []);
Enter fullscreen mode Exit fullscreen mode

I put the old line back to make sure the test fails on it, then took it out again. It fails on the bug and passes on the fix — which is the only way to know a regression test is a test and not decoration.

Three questions worth asking your own reports

  1. What is the denominator? If a line says "all N", find where N comes from. If N is a counter that increments on success, the word "all" is a lie waiting for a bad day.
  2. Can it tell "nothing failed" from "nothing was checked"? Zero failures out of zero attempts prints the same as a clean run. Report failures against attempts.
  3. Does the count come from the source of truth or from the transport? The number of URLs you have comes from your list — for a site, the sitemap itself, which you can pull into a plain list of every URL and count — not from how many of them your script happened to load. Then check each one's status code against that list, so the two numbers stay separate and the gap between them stays visible.

The fix is four lines. The habit is harder: when a script tells you everything is fine, ask it how many things it actually looked at.

Top comments (0)