DEV Community

ilya mozerov
ilya mozerov

Posted on

Seventy-four percent, and no information

A dashboard field can be correct, stable, and completely uninformative. That
combination is easy to miss, because a number that rarely changes looks like a
confident one.

The field that fed every guess

One of the inputs to a best-guess display was a "busiest area" reading: which
of a couple of zones a source reported as most active. It was used on every
estimate, so it looked like evidence about where someone was at a given time.

Measured over the retained history, the reading named the same area in 74% of
788 quarter-hour buckets. Not 74% on average across a few good hours — 74% in
every hour of the day, from midnight through the evening.

The test that answers "does this tell us anything?"

A feature is useful only if it does better than ignoring it. So we ran the
obvious control: predict the busiest area for a day using the other days, and
compare against a predictor that always answers with the most common area.

The learned predictor scored 0.7408. The always-same answer scored 0.7408.
Identical. On held-out days the field carried no more information than its own
base rate.

That is the whole finding, and it is worth stating plainly: a value can be
extremely predictable and still tell you nothing about the moment you care
about. "It's usually this area" is a fact about the week, not about now. A
best-guess display that treats it as evidence about now has mistaken a
constant for a signal.

Why the base rate hides in plain sight

Constants are comfortable. They make a pane look populated: there is a value,
it has a plausible shape, it is usually the same. A reader scanning for
something that locates a person will happily accept "the busy area is usually
this one" as a clue, when the honest reading is "we have no area signal here."

The reverse error is more familiar — a value that claims to know something it
does not — and gets more attention. The quieter failure is a value that is
true, stable, and irrelevant, quietly carrying weight it has not earned.

What changed

The two views of the display — the live pane and the saved copy that other
tools read — now print the measurement beside the reading: the share of
buckets, the number of buckets, and how many hours it covers. When the reading
is near-constant, they say so and call it explicitly not a person signal. When
the source is missing, stale, or too thin to measure (fewer than a day's worth
of shared buckets), both views print UNKNOWN instead of a confident constant.
No threshold was tuned to make the finding appear; the control was added and
the answer was reported.

The tests cover the cases that would let the claim drift: a near-constant
series, a series with one hour that flips to the other zone (named, with its
hour), a missing source, and a too-thin series. Each of the last two must say
UNKNOWN, never a clean zero.

The habit

When a number feeds a guess, ask what it would look like if it carried no
information at all. Then check whether your data can tell that case apart from
the one you hope for. If a leave-one-out control ties the trivial baseline,
the number is a base rate, not evidence — say so on the pane, in both copies,
before a reader does the guessing for you.

Top comments (0)