Your code is tested. Your measurements are not.

I keep a ledger of every change I ship that changes behaviour. Each entry carries a written prediction and a pre-ship baseline, and a review date is set the day it ships, usually five days out. On that date I have to come back and read the real numbers against what I said would happen.

Twelve changes have now been through that review. None has been reverted.

That is not the interesting number. The interesting number is that eight of the twelve reviews found something genuinely broken - and in all eight it was the measurement, not the change.

What an entry looks like

- id: adaptive-source-cap
  metric: digest_fill_rate
  target: >
    Sunday 2026-08-09 and Monday 2026-08-10 fill rates return toward
    their pre-2026-07-29 level (91.7-97.6%) and clearly beat the
    76.7%/59.0% measured after the cap change [...]
  baseline: >
    [...] A/B on 2026-08-03's real pool for the reporting subscriber:
    cap1 -> 11, cap2 -> 13, cap3 -> 14, against 20 requested -- so the
    relaxation is expected to recover roughly 2-3 stories, not all 8.
    The rest is genuine weekend thinness and should stay unfixed.

That is the real entry, trimmed only where marked. Read the two fields together.

The baseline contains an A/B on real data: relaxing the cap takes the subscriber from 11 stories to 13 or 14, against the 20 they asked for. That is a ceiling of 65-70% fill, and the baseline says so in words - recover roughly 2-3 stories, not all 8.

The target, written in the same sitting, a few lines below, demands 91.7-97.6%.

The fix did exactly what its own experiment predicted it would do, and failed its target for it. Verdict recorded as "Worked, against a target it could never have reached."

The three failure modes, in order of how hard they were to see

1. The target contradicts its own baseline

The one above. Easy to catch once you look, and the fix is mechanical: if an entry contains an experiment bounding the effect, the target may not exceed that bound. That is now a rule.

2. The baseline and the metric are in different units

An entry recorded a baseline of "286 collected, 71 delivered" over 30 days, described as computed with the metric's own logic. It was not. Those were distinct stories; the live metric counted slots across digests. Same nouns, different denominators.

The review came out wrong in a flattering direction, which is the dangerous kind. A result that disappoints you gets re-checked. One that confirms you does not.

3. The metric moves when your population does

This is the one worth the article.

The metric pooled a per-slot share across every subscriber who received a digest that day:

# what it did, in effect
share = matching_slots_across_everyone / total_slots_across_everyone

That number moves when the subscriber mix moves, even when nothing in the pipeline has changed. And it did move: two subscribers stopped receiving partway through the review window.

Same five days, same rows, three defensible readings:

reading	result
pooled across everyone	+1.45
late-leaving subscribers dropped	+4.26
per-subscriber, each against themselves	+3.84

Only the third is composition-free. It is also the only one that showed what actually happened: 6 of 6 subscribers improved, at +0.59 slots per digest.

I had already drafted the verdict "failed target, not failed fix" off the pooled number.

Two defensible readings disagreeing by 3x on identical data is not an inconvenience to resolve by picking one. It is the finding.

The related trap: the open day

One more, because it is the cheapest to fix and the easiest to ship.

My subscribers choose their own delivery hour, so the archive table gains rows all day - the send window runs 05:04 to 12:00 UTC. Every "since date X" metric treated today as a finished day.

A partial day does not read as missing data. It reads as a data point.

And the error is directional, not random: late digests skew one way, so an early read is always wrong in the same direction, and always on the newest point - exactly where you look to decide whether something is trending. It manufactured a four-day decline that was not there.

def _last_complete_day() -> str:
    """
    The most recent date whose archive rows are final: yesterday, UTC.

    Yesterday is provably safe rather than merely conservative: every row
    was created AND last updated within its own digest_date, checked
    across all 90 days of retention, zero exceptions.
    """
    return (date.fromisoformat(today_str()) - timedelta(days=1)).isoformat()

Two details in that docstring matter more than the one-liner under them.

It is not a time-of-day check. The first version of this guard hardcoded a cutoff derived from the current subscriber list, and it was wrong immediately - it assumed a single late subscriber at 11:00, because the other two happened to have that send day switched off on the day I measured. Dropping the whole open day is correct no matter who sends when.

It is deliberately not applied everywhere. The pool-backed metrics read a file the 05:00 pipeline writes once, so today's file is either absent and skipped, or already complete. Clamping those too would throw away a good data point to fix a problem they do not have. A guard applied uniformly because uniformity feels tidy is just a different way of being wrong.

Ten call sites route through it. A random error adds visible noise. A directional error on your most recent point manufactures a slope.

The takeaway

We have a whole discipline for testing code and essentially none for testing the things we measure code with. A broken function throws. A broken measurement returns a number, and the number looks exactly like an answer.

If you keep any kind of before-and-after metric, three questions worth asking today:

1. Does my target contradict a bound recorded in the same entry?
2. Are my baseline and my live metric counting the same unit?
3. Does my denominator move when my population moves rather than when my system does?

I build devdigest, a daily tech and AI digest that shows you why it picked each story - the ledger above is how I keep myself honest about whether any of it works. It launched on Product Hunt this weekend: https://www.producthunt.com/products/devdigest-io