Two retrieval runs for the same query. Relevant document is D.
Run A: [D, x, x, x, x, x, x, x]
Run B: [x, x, x, x, x, x, x, D]
...
For further actions, you may consider blocking this person and/or reporting abuse
"Recall@8 was 1.0 and your actual recall at the size you use is 0" deserves to be on a poster. I have shipped that exact mistake: evaluated at k=20 because the dashboard looked healthier, then fed the model top 5. If someone changes one thing after reading this, make k in the metric equal to k in the prompt.
One caveat on top of nDCG: it still depends on relevance labels you had to produce, and in my experience the labels drift faster than the ranker does. Every time the corpus changes shape I re-audit a sample of the qrels, otherwise the discount curve is being applied to a stale notion of relevant.
Orphaned label as the weaker statement is the right call. From outside the scorer you genuinely cannot separate "deleted" from "now ranks below k for every question", and since the bound on recall is identical either way, claiming the stronger version would only be wrong more often for no extra information.
The re-chunk case is the argument that sells corpus turnover: every id changes, retrieval quality is untouched, every metric goes to zero. Print-only rather than gating is also the right default, because a check that fails a build on legitimate corpus work gets switched off within a week and then you have neither the gate nor the signal.
One thing I would watch: orphaned labels accumulate for benign reasons as the corpus grows, so the raw count drifts upward on its own. A rate against total labels, or the delta versus the baseline run, carries more information than the absolute number, and you are already storing the retrieved ids per run to support exactly that.
Shipped in 0.3.0. orphanRate is surfaced, and the warning leads with the change rather than the level:
Two things came out of building it that I had not thought through when I answered you.
The recovered line is not symmetry for its own sake. A dataset repair that reached nothing looks identical to one that reached everything if only the total is printed, and the total does not move at all when one label goes unreachable while another comes back. That is the run where you most want to be told.
And an empty stored orphan set has to read differently from a missing one. Empty means the previous run had clean labels, so every orphan now is new, which is the single case most worth reporting. Null means the run predates the column and a delta against it would be invented. The obvious implementation, an early return when the baseline array is empty, swallows exactly the case the feature exists for. There is a test whose whole job is that distinction.
run compares against the previous run under the same label, gate against the baseline, so the line reads the way the metric table above it does.
The empty-versus-missing distinction is exactly right, and your implementation corrects my earlier assumption that the retrieved corpus alone could reconstruct the orphan delta. It cannot when the labels also move.
The recovered-ID line handles the case an aggregate would hide: one new orphan and one recovered label leave the total unchanged. I would keep that as a paired test alongside empty and null baselines. Label-set identity should travel with the stored orphan IDs too, so an intentional relabeling is distinguishable from a retrieval change.
The rate is half there and I buried it. The warning reads "3 of 47 labelled documents", which is the rate written as a fraction, and the JSON hands you both numbers so you can divide. auditLabels computes orphanRate internally and then does not surface it, which is the worst of the two options.
The delta is not there at all, and you are right that it carries more. One correction on the mechanics though: what I store per run is the retrieved corpus, not the orphan set, and the labels move between runs too, so the delta cannot be recomputed from what history holds. Two runs with the same orphan count can have completely different sets behind them. It needs the orphan ids stored per run alongside the corpus, which is a small addition on a migration path that already exists.
What that buys is the sentence worth printing. Not "3 labelled documents are unreachable", which is a standing condition you learn to scroll past, but "2 more than at the baseline, and here they are", which is an event.
You are right about the stored history: I assumed the orphan set could be reconstructed, but changing labels means the retrieved corpus alone is insufficient. Persisting the orphan IDs per run is the necessary addition before reporting a meaningful delta.
I would also distinguish newly unreachable IDs from IDs removed or relabeled between runs. Comparing against a fixed baseline label snapshot answers a retrieval-regression question; comparing each run's current labels answers an annotation-coverage question. Showing which comparison produced the warning would make that event useful without attributing every change to the retriever.