DEV Community

ilya mozerov
ilya mozerov

Posted on

Zero of three, and nothing failed

A score of zero can describe two very different situations. The model can be
wrong, or the examples can be unable to tell anything apart. The number looks
identical in both cases, so the only place the difference can live is beside
the number, in words.

A score with nothing to separate

One of our panes scores a signal-based location guess with leave-one-out
testing: take each labelled example out, guess it from the rest, and count how
many guesses were right. The evidence line printed the count and a majority
baseline, in the usual way.

Now suppose every labelled example carries the same signal value. Take a
constructed fixture of three labelled moments spread across two rooms, all
measuring -60 dBm. With one feature and no variation in it, the nearest
neighbour cannot separate the rooms: leave-one-out scores 0 of 3 correct, and
the majority baseline is no better. Nothing broke. The scoring code did what
it was told; the data simply had no room information in it.

That is the trap. "0/3 correct, majority baseline 1/3" reads like a failed
test. A reader reasonably concludes the method is useless, when the truth is
narrower: this particular set of labels cannot support the claim the score
appears to make.

Naming the degenerate case

The producer now reports the shape of the label set alongside the score. When
there are at least two labelled examples and they all share one finite signal
value, it records that value and the label count. The evidence line then says
so plainly: all three labels are at one level, so there is no room spread to
measure.

The display keeps the score. It does not hide the zero or round it into
something friendlier. It adds the sentence that turns a misleading number into
an honest one: 0 of 3 is correct, and it is not a room claim.

The guards matter as much as the sentence

A "no room spread" note is itself a claim, so it is only emitted when the data
supports it.

With a single labelled example, the pane keeps its nearest-match reading but
does not claim there is no spread — one example cannot establish that. When
the labelled values differ, it says nothing about spread, because the scoring
may be meaningful. Non-numeric values are excluded too; a boolean is technically
a number in the implementation language, and must not be mistaken for a signal
level.

Focused tests pin each boundary: the all-equal case names the degeneracy and
keeps the zero; differing values and a single value both stay quiet; and the
line without labels is reported as UNKNOWN rather than as a room. The check
distinguishes "the score could not be computed" from "the score was computed
and means little".

The habit

Before trusting an accuracy figure, ask what varied in the examples. If the
input feature was constant, the score measures the dataset, not the method.
A number without its degeneracy condition is only half a result; the useful
thing to publish is the sentence that says which half you are looking at.

Top comments (0)