DEV Community

Cover image for Where Your Metrics Fold
Harry Floyd
Harry Floyd

Posted on • Originally published at harryfloyd.substack.com

Where Your Metrics Fold

The dashboard read 87 percent complete, and it was right: 87 percent of the scheduled tasks for the launch were genuinely done, ticked off, verified. The board did its job. Then the launch slipped by six weeks, and in the post-mortem someone put that same 87 percent back on the screen and the room went quiet, because nothing about it had been wrong.

Look at what the number saw and what it could not. It observed completed work, accurately. From that, everyone in the room inferred the launch was on track. What it left out was where the unfinished 13 percent sat: almost all of it behind a single unresolved dependency on the critical path. A project with its hard problem solved and a project with its hard problem untouched both report 87 percent complete. The reading was true. It simply could not tell those two projects apart, and they needed opposite decisions.

The failure did not live in the measurement. It happened in the instant the situation was flattened into a single number.

A scalar metric is a lossy projection. It presses a many-dimensional reality down to one value, and what the pressing throws away cannot be recovered from that value alone. When many dimensions map to one, distinct states end up sharing a reading. Call any decision-relevant collision a fold: two states that receive the same value but would demand different actions if you could see them apart. The 87 percent was a fold. A perfect sensor reading exactly what it was designed to measure can still fold, because the loss happens when that measurement is used to stand in for a larger decision.

Take a customer rating sitting at 4.7. In one store almost everyone rates it between 4 and 5. In another, most customers give it 5 while one strategically important segment consistently gives it 1, and the two average out to the same 4.7. Both readings are honest, drawn from complete data. One store has broad satisfaction. The other has a concentrated failure hidden inside an excellent mean, and the number gives you no way to tell which store you are running.

Folds are easier to hunt once you know what a number tends to lose. Four useful kinds cover many of the folds you meet in practice.

A composition fold hides different groups behind the same average. The 4.7 that is broadly fine and the 4.7 that hides a segment stuck at 1 star are the same point. Statisticians know an extreme special case as Simpson's paradox, where the aggregate can run opposite to every subgroup inside it.

A trajectory fold hides direction behind a level. Monthly churn of 5 percent can be steady, recovering fast, or coming apart, and this month's number reads identically in all three.

A structure fold hides where the value sits behind a total. The project that is 90 percent complete with the critical path finished and the one that is 90 percent complete with every hard dependency still open share one headline.

A mechanism fold hides how the result was produced. The same quarterly profit can come from stronger customer economics or from deferred maintenance and postponed investment, and the profit line looks identical either way.

Four questions to run at any number: what it is made of, which way it is moving, where the value is concentrated, and what produced it. Each names a dimension the number quietly averaged away.

This would stay a curiosity if folds were rare corners you might wander into. They are where effort flows. You often cannot move the outcome you want directly. You move the number you can see, and the cheap way to move it runs straight through a fold. Nudging an already-satisfied majority from 4 to 5 may be cheaper than repairing the experience of the segment stuck at 1 star. Both lift the average. Only one closes the concentrated failure the mean was hiding. Once a score is the thing people are paid to move, the shortest path to the score and the shortest path to the goal stop being the same path.

Capability does not save you here. The more capable the optimiser, the more thoroughly it searches the states that score well, and when the score has folds that thoroughness cuts both ways: capability expands the search for shortcuts as well as solutions. Watch it in machine evaluation. As of mid-2026, frontier models that resolve over 70 percent of SWE-bench Verified's single-issue fixes drop to roughly 23 percent on SWE-bench Pro, whose tasks demand larger changes across multiple files in professional repositories. The easier score had folded the two capabilities together: the system that can carry the wider process and the one that cannot share a reading, and that fold is exactly where a capable optimiser lands.

By now you may be filing this under Goodhart's law. When a measure becomes a target, it stops being a good measure. But look again at what you did a few paragraphs ago. You found a fold in your own number before any optimiser entered this argument, before anyone was pushing on anything. Goodhart tells you that optimisation can separate a measure from the goal it represents. The fold was there before any optimisation, sitting in the number's structure, waiting. The fold-map asks the prior, operational question: which different realities does your measure already treat as the same? Find those collisions in advance and you know where to watch once targeting pressure arrives.

The number does not have to lie to mislead the decision.

So map it. Take your one number and write the question you believe it answers. Then look for one fold of each kind. For every fold you find, fill six columns: the metric, state A, state B, the reading they share, the different decision each would call for, and the one signal that would separate them. Then rank the folds by the cost of confusing the two states, how likely that confusion is, and how cheaply an optimiser could produce the misleading one. For the fold that combines severe consequences, plausible confusion, and a cheap path to the misleading state, start watching its separating signal alongside the number. A mean plus a variance still cannot rebuild the whole distribution. It splits the one collision you care about most, which is enough to act on.

Every important number should get this treatment before it reaches a dashboard. It takes ten minutes, and it turns part of the post-mortem into a pre-mortem: the likely failure sites, named before the failure instead of after.

Run it tonight on one number you are judged by. Write two situations that score the same, name the decision each would change, and start watching the signal that tells them apart. You may not find a fold of all four kinds the first time. The empty rows are the reward: they mark the parts of your number you have never inspected.

Which number do you steer by that you have never checked for a fold, and what do you now suspect it has been hiding?

Top comments (0)