FINAL-Bench: A Diagnostic for Whether a Model Knows What It Does Not Know
What FINAL-Bench measures
Most benchmarks ask "did the model get the right answer?" FINAL-Bench asks a harder, safety-relevant question: does the model know what it does not know? It is a metacognition and safety diagnostic - it probes calibration, hallucination-awareness, and whether a model refuses appropriately instead of confidently making things up.
Why calibration, not accuracy
A model that is wrong 20% of the time but knows which 20% is far safer to deploy than a model that is wrong 10% of the time and confident about all of it. In regulated settings - health, finance, law - the failure that hurts is confident wrongness. FINAL-Bench targets exactly that axis: the gap between what a model claims to know and what it actually knows.
Two signals we can share:
- Single-pass self-doubt is surprisingly informative. A model's own hesitation, read from a single forward pass, separates correct from incorrect answers with AUROC around 0.90 (research-stage signal).
- But a model's own confidence is a weak judge of itself. An external judge scores around AUROC 0.72, while the model grading its own confidence lands near 0.59. The lesson: self-reported confidence is not enough; independent evaluation beats introspection.
Mapping to regulation
Findings are mapped, at a high level, to the frameworks teams actually answer to - the EU AI Act, the NIST AI Risk Management Framework, and similar. The goal is to turn an abstract "is this model safe?" question into evidence a risk or compliance team can read: where the model over-claims, where it refuses well, where it needs a human gate.
The diagnostic is built from 117 items.
Who this is for
Teams shipping models into regulated or high-stakes settings, and the risk and compliance people who have to sign off. If you already track accuracy, FINAL-Bench adds the orthogonal axis you are probably missing: whether the model's confidence can be trusted at all.
Honest scope
The 117-item methodology - how items are constructed and how responses are scored - is patent-pending and under NDA. This article describes only the purpose (metacognition and safety), the calibration idea, and the regulatory mapping; it deliberately does not disclose item design or scoring internals. The AUROC figures (single-pass self-doubt ~0.90, external judge ~0.72, model self-confidence ~0.59) are research-stage signals from our own runs, not certified scores for any specific commercial model. FINAL-Bench is a diagnostic, not a certification: it produces evidence and risk mapping, it does not "pass" or "approve" a model for regulated use, and the law-mapping is an interpretive aid, not legal advice. Independent replication and third-party audit are the right next step, and we treat it that way.
More: https://vidraft.net
Top comments (0)