A 75% score over four rows
Put one number on the table and break it open. Suppose an eight-row inventory and its predictions show 75% correct. That fraction covers four records: id0, id1, id4, and id6. id2 abstains, id3 has no prediction line, id5 is invalid because of conflicting duplicates, and id7 has null gold. None enters the A/C=3/4 numerator or denominator, but the inventory still has eight rows. The number itself does not say which rows left, so the per-row filters belong next to it.
prediction-metric-desk is a local audit desk that compares an inventory with saved predictions and counts them. It trains nothing, infers nothing, and does not rewrite invalid rows. The eight-row fixture is a fixed fault set included in the package. The inventory has N=8 records, with K=7 known gold and null gold for id7. The predictions file has 11 physical lines, including one broken JSON line, one extra ID, and duplicate id5 lines. These rows stay in the report.
The screenshot comes from the standalone App page and is uncropped. The App renders NLL as ∞ and notes zero true-class probability. The exported JSON writes the same condition as value:null and infinite:true, two representations with no change to the metric definition.
Record-level map
One clarification first: labels are ["a","b","missing"], and "missing" is a legitimate taxonomy string, not a missing value. id4 has gold label "missing" and predicted label "missing", so this line counts as correct. That is separate from id3's missing state, which means there is no prediction line at all.
| id | gold | prediction line | C | D | E |
|---|---|---|---|---|---|
| 0 | a | predicted a, probabilities [0.8,0.1,0.1] | yes | yes | yes |
| 1 | a | predicted b, probabilities [0,1,0] | yes | yes | yes |
| 2 | b | predicted null, probabilities [0.1,0.8,0.1] | no | yes | no |
| 3 | b | no prediction line | no | no | no |
| 4 | "missing" | predicted "missing", no probabilities | yes | no | yes |
| 5 | "missing" | two lines: a and typo, invalid | no | no | no |
| 6 | a | predicted a, probabilities [0.1,0.8,0.1] | yes | yes | yes |
| 7 | null | predicted b, probabilities [0.1,0.8,0.1] | unscored | no | yes |
From the table: C contains id0, id1, id4, and id6; three of them are correct, so A=3, C=4, and A/C=75%. id2 has probabilities but a null predicted_label, so it is abstained and enters D but not C or E. id4 has a decision and no probabilities, so it enters C/E and not D. id5 has two conflicting physical lines, so the record is invalid and stays out of C/D/E; both original lines remain visible in prediction_lines, while line_accounting carries counts such as invalid_lines and not the raw rows themselves. id7 has null gold, is not scored, and enters E only. The extra ID is separate from inventory cases.
C and D both equal 4 in this fixture by coincidence, but their members differ: id2 is in D and not C, id4 is in C and not D. Reading 4 as the same filter misses that difference.
Coverage needs three rates
E counts a single valid prediction line with a non-null predicted_label, regardless of gold, so id7 qualifies and E=5. C counts decided rows among known gold, so C=4. C/K=4/7 asks what share of known-gold rows decided; E/N=5/8 asks what share of inventory emitted a label; A/C=3/4 asks what share of those decisions were correct. Treating A/C as overall accuracy drops the first two denominators.
U=K-C=3. If I force all three undecided rows to answer, completed accuracy runs from A/K=3/7 to (A+K-C)/K=6/7. The range assumes all three are wrong or all three are right. It is not a confidence interval and not a promise. It shows how sensitive the current 0.75 is to those rows.
Probability support and decision labels
D counts known-gold rows with one valid prediction line and a non-null probabilities vector, four in this fixture. Label order is a, b, "missing"; id1 has true class a and vector [0,1,0], whose first term for a is zero. Mean NLL is therefore infinite. The exported report writes value:null, infinite:true, zero_probability_rows:1, and support:4; the page renders ∞. I do not clip the value, renormalize the vector, or fabricate a one-hot output. predicted_label may also differ from probabilities argmax: id6 has predicted_label a while its argmax is b. The report records that mismatch and does not override the policy label.
Reproduce the fixture with one command
From the package root, with Node 22 or newer and no existing output file:
node audit_cli.mjs data/fault-inventory.json data/fault-predictions.jsonl my-audit.json
stdout is JSON. This block is an excerpt of key fields, not the full stdout; the full report is written to my-audit.json with cases as record-level status, prediction_lines as original physical lines with line errors, and line_accounting as a physical-line summary.
{
"denominators": {"N": 8, "K": 7, "E": 5, "C": 4, "A": 3, "U": 3, "D": 4},
"metrics": {
"coverage": 0.625,
"known_gold_coverage": 0.5714285714285714,
"conditional_accuracy": 0.75,
"completion_bounds": [0.42857142857142855, 0.8571428571428571],
"nll": {"value": null, "infinite": true, "zero_probability_rows": 1, "support": 4}
}
}
That command verifies the table without recomputing the 3080-row result. Entry points and fixed version:
I still recommend the standalone App page as the stable entry point. The inline Hub view appeared blank in a minimized Edge environment on 2026-11-13, and that cause remains unproven. On 2026-11-14, a signed-out inline browser environment previewed and applied the fault fixture and showed N=8, C=4, A=3, D=4, conditional accuracy 3/4, known-gold coverage 4/7, and infinite NLL. That success is limited to the new signed-out context; it does not show the original Edge issue is fixed, the root cause is known, or every browser and entry point works.
What the report carries
Active audit work and newly imported inputs live in browser memory and clear on refresh. There is no autosave and no crash checkpoint. A downloaded JSON report itself remains on disk, but it cannot be reimported as a checkpoint, and refreshing does not reload it automatically. The exported report contains full original inventory and predictions, so I check what is inside before sharing. SHA256 proves byte consistency, not provenance, authorization, or gold-label reliability.
The 3080-row BANKING77 predictions were generated from saved int8 logits and frozen temperature 0.75 by the earlier work; prediction-metric-desk audits that saved result. It gives 2779 correct and 301 incorrect while retaining the original seven normalized train/test overlaps. The tool does not regenerate logits, fit a temperature, or perform new calibration. That is not new training, inference, blind evaluation, or generalization evidence. The implementation and synthetic fixture are Apache 2.0. The BANKING77 source data and derived outputs stay CC BY 4.0 with attribution to PolyAI and Casanueva et al. 2020.
By Renchao Wu.

Top comments (0)