I released Rejected Check Audit, an offline tool for auditing denominators in multi-stage filters. It retains events rejected by earlier gates, keeps unknown labels separate from negative labels, and traces known misses to the stage where each event stopped.
Reviewing only the final releases can reveal errors among those releases. It does not reveal how many relevant events were rejected. Treating unreviewed events as negatives, or inventing downstream decisions for stages that never ran, changes the meaning of the reported metrics.
The repository contains twelve author-constructed events in three labeling views: complete labels, released-only labels, and a purpose-selected review subset. These are the same twelve identities, not thirty-six independent observations. When only releases are labeled, the auditor retains the original denominator and returns null for overall recall with an explanation. It does not extrapolate a selected review subset to the whole population.
The tool runs locally with the Python standard library and exports JSON plus stage, path and metric CSV files. A separate verifier, a frozen hand-counted contract and a CPU notebook are included. The final publication package passed fresh-directory execution and additional independent state-combination checks. Published HF file contents and the three searchable dataset views were also checked.
These results validate accounting behavior. They are not model-quality measurements or evidence that a filter improved. There are no production traces, model responses, inference calls or training runs. Real use still requires a defensible event identity, labeling rule, sampling design and stage semantics. Recall uses a conservative complete-label rule; sampling weights, population confidence intervals and partial-identification bounds are outside scope.
Dataset and code: https://huggingface.co/datasets/WideChaos/rejected-check-audit
Code and synthetic fixtures are licensed under Apache-2.0. By Renchao Wu.
Originally published on widechaos.cn.
Top comments (0)