I founded Krasyn, an outpatient EMR with an AI scribe inside it. Krasyn has run a working outpatient clinic's real patient records since March 2026, so what our scribe drafts ends up in charts that real clinicians sign. This post covers two things we shipped in August: a published benchmark of how faithful those drafts are to the transcript, and Note Check, a tool that reads any scribe's note against its transcript and lists what the transcript does not support.
Why a fluent note is the problem
A faithful note and a note with one invented blood pressure look the same on the screen, and the clinician who signs it owns every sentence. Published evaluations put ambient-scribe hallucination at about 1 to 3 percent of notes. A March 2026 analysis of 71,173 AI-drafted and finalized note sections found a confirmed edit in 5.8 percent of them. The drafting got automated. The checking did not.
I wanted a number for our own scribe that I could defend, with the definitions printed next to it. A benchmark without definitions is marketing.
The unit: a clinical assertion
We measure at the level of a clinical assertion, one atomic statement about the patient that could be true or false on its own. "Denies fever, chills, and nausea" is three assertions. A measurement and its value are one. Hedging is kept verbatim.
Every assertion gets exactly one label against the transcript:
- Supported: the transcript says it, or it is a faithful paraphrase or clinical translation.
- Inferred: not stated, but a reasonable clinical inference with a basis in the transcript. Tracked separately because it is the contested category.
- Unsupported: no basis in the transcript at all.
- Contradicted: the transcript says the opposite, including a symptom the patient denied, a treatment the clinician declined, or another person's symptom attributed to the patient.
Hallucination rate is unsupported plus contradicted over all assertions. Coverage is measured separately against key facts per case, because a note that says only "Patient was seen" scores perfect faithfulness. A faithfulness gain bought by dropping content is a regression.
Traps
Each case also defines traps, specific fabrications the note must not contain, each a regex plus a written rationale. Traps need no model, so they cannot drift when a model changes. A trap fires only if a mention survives negation and irrealis suppression scoped to the containing sentence, and every suppressed mention is written to the artifact with the rule that suppressed it.
The corpus
Twelve synthetic transcripts, 144 key facts, 61 traps. All original invented dialogue, no real or de-identified patient data. Seven of the twelve are adversarial: a spouse describing her own symptoms, eleven explicit denials in one visit, pervasive hedging, audio dropouts, treatments discussed and then declined. In those cases the faithful answer is to say less. The corpus was committed to git before the harness existed, because this repo has a documented habit of expectations written to match current output.
The two arms
One generation feeds both arms. The ungrounded arm is the draft as generated. The grounded arm is the same draft after our production grounding verifier, which strips sentences it cannot trace to the transcript. There is no competitor arm. We cannot test other scribes fairly with no API access and no matched corpus, so we publish no number for them.
The 2026-08-09 run
| Metric | Ungrounded | Grounded |
|---|---|---|
| Assertions extracted | 296 | 279 |
| Hallucination rate | 1.7% | 0.0% |
| Contradiction rate | 1.0% | 0.0% |
| Faithfulness precision | 94.6% | 97.8% |
| Key-fact coverage, strict | 88.2% | 86.8% |
| Traps fired | 6 of 61 | 5 of 61 |
Negative control: every run injects three unambiguous fabrications into every grounded note and re-scores. 36 of 36 were caught. If that figure were ever below 100 percent the report would say at the top that every other number is suspect.
The coverage cost is real. Grounding removed 12 sentences and strict coverage fell 1.4 points. All of that loss sits in one case, FB-008, where the judge stripped a whole sentence that bundled a fabricated denial with a true hedged finding. Whole-sentence removal is the residual defect.
Five traps still fire in the grounded arm. Two are in the crosstalk case, where the note correctly recorded that the spouse described her own headaches and the regex fired anyway. We leave that unsuppressed on purpose. A rule good enough to silence it would also hide a real wrong-patient attribution.
The limitations, stated plainly
The labels come from gpt-4o, and gpt-4o also wrote the notes. Self-preference bias in model judges is documented, and it is uncontrolled here. No clinician has adjudicated a single label. The harness author, an AI agent, also wrote the corpus, the traps, and the judge prompts. We measure agreement each run on 163 assertions with the list held fixed: the same judge re-run agreed 100 percent, and gpt-4o-mini agreed 96.3 percent on the exact label. Agreement between two models is not agreement with a clinician.
Twelve cases is small, and per-case differences are anecdotes. The transcripts are written English, not ASR output. The trap suppression rules were tuned after a first run produced 13 false positives, and a self-test now pins every rule against an assertive counterpart.
Neither layer reliably scores a fabricated denial. An invented "denies fever" reads as grounded to the judge when the transcript merely mentions fever. That defect was found by reading notes, not by a metric moving. A number that cannot go up when a defect occurs is not evidence the defect is absent.
Note Check
The benchmark measures our scribe. It does not help someone who uses a different one. So we built Note Check. You paste a transcript and the note any scribe drafted from it, and you get a report on that note. It works on pasted text from Krasyn, Freed, Heidi, Nabla, Abridge, DAX, Suki, Upheal, Mentalyc, or anything that produces a note as text. It does not connect to another vendor's system.
What it does. Each sentence of the note gets one of five labels against the transcript: Supported, Unsupported, Contradicted, Scaffolding, or Unverified. Three flags come from plain code with no model involved: a number in the note that never occurs in the transcript, a pertinent negative about a topic the transcript raised but never denied, and specific content placed where the transcript only has an inaudible marker. An omissions list names clinically material facts the transcript stated and the note left out, each with its excerpt. Contradicted and unsupported lines are lifted to the top, and the report exports as plain text.
What it does not do. It does not edit the note, file it, or sign it. An exam finding the clinician observed and never spoke reports as Unsupported, which means "not in the transcript" and nothing more. It compares against the transcript you paste, not the chart. And we publish no accuracy figure for Note Check, because it has not been measured against a clinician-adjudicated reference set. It is a review aid. Read the whole note.
The honest determinism note
The pure-code layer always runs and its flags are identical on every run. The model judge runs at temperature 0 with a pinned prompt version, and it is not seeded, because the pinned SDK exposes no seed. On 2026-08-21 we ran the same flawed synthetic note twice on the bench and once in production. All five planted contradictions were caught every time and the two code flags were identical. The reason prose differed on all five contradicted statements, and one borderline statement, "Hypertension, controlled", moved between Supported and Unsupported between environments.
So we do not call the judge deterministic. Instead the finished report is stored under a fingerprint of the transcript, the note, the engine version, and the judge model. Within your account the same inputs return the stored report, with no second model call. Every report states whether the judge ran, ran on only part of the transcript, failed, or did not run, and any statement it did not assess is labelled Unverified rather than counted as Supported.
Where to find it
The full method with every definition is at https://krasyn.com/resources/ai-scribe-faithfulness-benchmark. The source document is docs/ai/FAITHFULNESS_BENCHMARK.md in our repository, which is private today. Note Check lives inside the free Krasyn Scribe account, which includes 10 AI-drafted notes a month with no card, and each check uses one of those notes. A synthetic worked example is at https://krasyn.com/note-check?utm_source=devto&utm_medium=article&utm_campaign=note-check-landing.
If you run a scribe evaluation of your own and your definitions differ from ours, I would like to read them.
Top comments (1)
Faithfulness checks need to be portable if the category is going to mature. A checker that can run against any scribe note is much more useful than a private benchmark that only proves one vendor's demo.