Scribe red-team, installment 02. Published 2026-09-05 by Krasyn.
Dialogues and notes in this piece come from the Omi Health medical-note-eval corpus, MIT license, copyright (c) 2025 Omi Health B.V. All of them are synthetic. None of them are records of real people.
What this series is
Every two weeks we run Krasyn Note Check, the part of our product that reads an AI-written clinical note against its transcript, over data we did not write, and we publish every disagreement. That includes the disagreements where the check is the one that is wrong. The two series rules that shape this installment are these: sources must carry an open licence we can name, and when the engine version changes, the previous installment's exact inputs get rerun on the new engine before any new data is touched.
Both rules bit this time.
The therapy source we did not run
Installment 01 announced that installment 02 would take a therapy or behavioural-health source, because there is a real gap there. Studies of ambient documentation report accuracy falling in psychiatry and behavioural encounters compared with general medicine, and no published error data exists for AI-generated therapy notes anywhere. Behavioural health is also the segment where patients are most anxious about what a machine writes down about them.
The candidate was AnnoMI, a collection of expert-annotated motivational interviewing conversations transcribed from public demonstration videos. It is a good dataset for the question. It fails our licence rule.
Read on 2026-09-05: the GitHub API for uccollab/AnnoMI returns "license": null. The repository root holds AnnoMI-full.csv, AnnoMI-simple.csv and README.md, and no LICENSE, LICENSE.md, LICENSE.txt or COPYING. The README asks to be cited. A citation request is not a grant of terms, and "publicly available" is not a licence name.
Our rule says the licence has to be nameable and reproduced in every piece. It is not. So we did not run it, and this section is the record of that reading, which the rule also requires.
What we ran instead keeps the subject and changes the corpus. The MIT-licensed Omi Health set we used in installment 01 ships a file describing which body systems each of its 300 dialogues touches. 52 carry a psych tag. We took the eight with the densest behavioural-health content, excluding the six installment 01 already used, and paired each with notes from three named model writers: GPT-5.5, Claude Opus 4.8 and DeepSeek-V4-Pro. 24 pairs.
These are primary-care and specialty visits with substantial behavioural-health content, not psychotherapy sessions. The gap that motivated the therapy installment is still open. We are still looking for a therapy transcript source with a licence we can read.
First, the fixed set
Between installment 01 and now, the Note Check engine went from notecheck-1 to notecheck-6. The series rule says the previous installment's exact set gets rerun first, so that engine changes are visible against a fixed baseline rather than against new data that might be easier or harder.
We regenerated installment 01's 36 pairs from the same pinned corpus commit with installment 01's own preparation script, and confirmed the statement text came out identical in 36 of 36 pairs before comparing anything.
| Metric | notecheck-1 | notecheck-6 |
|---|---|---|
| Statements judged | 1,401 | 1,401 |
| Supported | 776 | 737 |
| Unsupported | 3 | 35 |
| Contradicted | 0 | 0 |
| Scaffolding | 622 | 629 |
| Deterministic rule flags | 16 | 2 |
| Omissions listed | 85 | 11 |
Installment 01's headline finding was that the AI judge inside Note Check accepted assessment and plan content the visit never contained. Every statement we named there is now marked Unsupported. The cough note that invented an allergic cause in a visit that ended before any conclusion. Two notes that wrote "Type 2 diabetes mellitus" over a doctor who only ever said "your diabetes". A plan line adding smoking cessation counselling nobody proposed. A note that wrote "Continue metformin" when the plan was to adjust medications.
Installment 01's other headline was that the pure-code negation rule fired sixteen times and thirteen of those were wrong, most of them on a patient answering "Correct, none of those" to a list of contraindications. Those thirteen are gone. The two flags that survive are exactly the two we read as correct.
That is a real improvement on a fixed set of bytes, and it is checkable line by line in baseline-rerun/engine-comparison.json.
Then, the new way of being wrong
Thirty-five Unsupported verdicts where there were three. Four of them are wrong, and all four are wrong the same way.
Example 1
Source: Omi Health medical-note-eval (MIT), dialogue_24, note by GPT-5.5.
Transcript: "S14: I have your recent lab results here, your A1C has gone up to 8.2% from 7.5% last time."
Note sentence: "Hemoglobin A1C increased to 8.2% from 7.5% previously." The note has it under Exam.
Note Check verdict: Unsupported. Reason given: "The hemoglobin A1C is a lab result, not part of the physical exam."
Our reading: Note Check is wrong. Both numbers and the direction of change are in the transcript verbatim. The reason concedes the fact and objects to which heading it sits under. In this product Unsupported means not supported by the transcript. A clinician reading that verdict will go looking for a fabricated lab value and find a correctly transcribed one under an awkward heading.
Example 2
Source: Omi Health medical-note-eval (MIT), dialogue_48, note by Gemini 3 Pro Preview.
Transcript: "S7: [P]: The irregular cycles are more frustrating, the cramps are manageable with ibuprofen."
Note sentence: "Ibuprofen (used by patient for cramps)"
Note Check verdict: Unsupported. Reason: "The transcript mentions the patient uses ibuprofen for cramps but does not indicate it was part of the plan or prescribed."
Our reading: Note Check is wrong. The note says what the transcript says, and the parenthetical is exactly the qualifier that keeps it honest.
Example 3
Source: Omi Health medical-note-eval (MIT), dialogue_70, note by Gemini 3.1 Pro Preview.
Transcript: "S23: Should I keep a log of when the headaches and numbness happen until then?"
Note sentence: "Patient asked if he should keep a log of when the headaches and numbness happen."
Note Check verdict: Unsupported. Reason: "The patient asked about keeping a log, but the clinician did not confirm or instruct the patient to do so."
Our reading: Note Check is wrong. The note reports a question as a question. In the same dialogue, two other notes asserted the log as an instruction the clinician never gave, and there the Unsupported verdict is right. The judge is not separating a note that fabricates a plan from a note that accurately reports a patient's question in a plan-shaped section.
There is a fourth, on dialogue_24, where a note honestly writes "Left knee examination planned but findings not documented in transcript" and the judge marks it Unsupported with a reason that restates the note back at it.
The pattern is one thing: the judge is grading section placement and reporting the result under a faithfulness label. That is worse than a miss. A miss is silence. This is confident and wrong in the direction that costs a clinician time, and it penalises exactly the careful hedging a good note should contain.
The psychiatric set
24 notes, eight dialogues, three writers. 1,393 statements judged: 992 Supported, 7 Unsupported, 0 Contradicted, 0 Unverified, 394 Scaffolding. 14 rule flags. 3 omissions listed.
Six of the seven Unsupported verdicts land on DeepSeek-V4-Pro notes, which is the direction the corpus authors' own leaderboard would predict. Dialogue_169, the most explicitly psychiatric encounter in the set, with a depression scale, a dementia screen and a memory quotient in the transcript, produced nothing at all across three notes.
Example 4, where the check is right
Source: Omi Health medical-note-eval (MIT), dialogue_87, note by DeepSeek-V4-Pro.
Transcript: "S10: [D]: Last time we spoke, we were working on managing your anxiety and seasonal allergies." and "S22: [D]: Have you been taking the medication we discussed for anxiety?" The words "disorder" and "diagnosis" never appear.
Note sentence: "Anxiety disorder, with intermittent exacerbations characterized by palpitations, chest tightness, and insomnia"
Note Check verdict: Unsupported. Reason: "The transcript does not label the anxiety as a disorder or provide a formal diagnosis."
Our reading: Note Check is right. This is the same shape of statement the old engine passed when a note wrote "Type 2 diabetes mellitus" over "your diabetes". In behavioural health the upgrade from a symptom to a named disorder is not a formatting choice. It follows a person into a record that insurers, employers and courts can reach.
The same note also carries "Seasonal allergic rhinitis, with nasal congestion and itchy eyes, partially controlled on current regimen", also Unsupported. We cannot call that one cleanly. "Partially controlled on current regimen" is a judgement nobody in the visit made, and the verdict is right on that clause. "Seasonal allergic rhinitis" is ordinary clinical translation of "seasonal allergies" with congestion and itchy eyes, and the same verdict is harsh on that clause. One verdict covers the whole line, so a reader cannot tell which half is at issue without opening the reason text.
Example 5, where the check is right about a small invented negative
Source: Omi Health medical-note-eval (MIT), dialogue_38, note by GPT-5.5.
Transcript: "S28: [D]: Do you ever get nausea or vomiting with the headaches? S29: [P]: Yeah, sometimes I feel sick to my stomach."
Note sentence: "Gastrointestinal: Positive for nausea; negative for vomiting."
Note Check verdict: flag neg-001, High severity: "'vomiting' is recorded as denied, but the transcript shows the topic was raised and nobody denied it."
Our reading: Note Check is right. The patient affirmed nausea and never answered on vomiting. The writer completed the pair because completed pairs look thorough. This is the exact thing the rule exists for.
Example 6, where the check is wrong on the cleanest possible input
Source: Omi Health medical-note-eval (MIT), dialogue_62, notes by GPT-5.5 and DeepSeek-V4-Pro.
Transcript: "S35: [D]: Any episodes where you got lost or forgot where you were? S36: [P]: No, nothing like that."
Note sentence (GPT-5.5): "Patient denies episodes of getting lost or forgetting location." DeepSeek-V4-Pro wrote "No episodes of getting lost or forgetting location".
Note Check verdict: flag neg-001, High severity: "'forgetting location' is recorded as denied, but the transcript shows the topic was raised and nobody denied it."
Our reading: Note Check is wrong, twice, and this one has no excuse. Question and denial on adjacent lines, correct speakers, plainest possible phrasing. Both writers recorded the visit correctly and the report tells a clinician at High severity that they did not.
Example 7, a rule that reads digits and not words
Source: Omi Health medical-note-eval (MIT), dialogue_54, note by DeepSeek-V4-Pro.
Transcript: "S59: [P]: My dad had an ulcer in his fifties, and my grandmother had colon cancer."
Note sentence: "Father: ulcer in his 50s"
Note Check verdict: flag num-001, Review severity: "'50' appears in the note but no such value appears in the transcript."
Our reading: Note Check is wrong. The transcript spells the number as a word and the rule compares digits. Ages, doses and frequencies get spoken as words constantly, so this is not an edge case.
Example 8, a fact listed as missing while it sits in the note
Source: Omi Health medical-note-eval (MIT), dialogue_100, note by DeepSeek-V4-Pro.
Transcript: "S75: [D]: If your anxiety worsens or you develop chest pain, shortness of breath, or severe headaches, call right away."
Note sentence: "Provide safety netting: call immediately if anxiety worsens, or if chest pain, shortness of breath, or severe headaches develop"
Note Check verdict: listed as an omission, key fact "Provider advised patient to call immediately if anxiety worsens or if chest pain, shortness of breath, or severe headaches develop."
Our reading: Note Check is wrong. The note carries the fact in the Plan section in near-identical wording. Two of the three omissions in this whole run are of this kind. The third, about a patient who eats worse under stress, is genuinely missing and is right.
The part we got wrong ourselves
Fourteen rule flags in this run, thirteen of them wrong on reading. Nine of the thirteen are our fault.
Three of these eight transcripts pack two speakers onto one line, like this: S37: [P]: No. [D]: Any recent head injuries? with the answer on the next line. Our preparation script, unchanged from installment 01 so the two installments stay comparable, maps only a speaker tag at the start of a line. So the doctor's question stayed attributed to the patient, and the rule had a topic with no visible denial.
We could have fixed the script and reported a cleaner number. Instead we kept the original for the headline run and added a control that renormalises the affected pairs properly, then reran them. Nine flags cleared. Three did not: the two on "forgetting location" above, and the "fifties" one. The correct flag on vomiting also survived.
Headline run: 14 flags, 1 right, 13 wrong. Control run: 4 flags, 1 right, 3 wrong. Both numbers are in the report, because publishing only the flattering one is the thing this series exists to not do.
The finding we did not go looking for
The corpus ships something we had not used before: the output of Omi Guard, its authors' own transcript-grounded safety layer, run over the same writers and dialogues. That puts it in the same job as Note Check, built by different people, on the same 24 notes.
We compared each base note with its Guard version, ignoring citations, bullet characters and headings.
| Across the 24 notes | |
|---|---|
| Content lines Guard removed | 0 |
| Content lines Guard added | 18 |
| Omissions Note Check listed | 3 |
| Of those, naming a fact the note already had | 2 |
| Overlap between Guard's 18 and Note Check's 3 | 0 |
Everything Guard did on this set was recall. Almost everything Note Check did was precision. On what a psychiatric note left out, the two systems agreed on nothing.
One of Guard's 18, from the same MIT corpus, dialogue_54, GPT-5.5 note:
Transcript: "S87: [D]: Are you up to date on your routine screenings, like colon cancer screening? S88: [P]: I haven't had that yet. S89: [D]: Given your family history, we may want to discuss that soon."
Note: nothing. Colon cancer appears only as "Grandmother had colon cancer" under family history.
Note Check verdict: no omissions listed for this note.
Guard added: "Discuss colon cancer screening soon given family history."
That is a screening plan raised because of a family history, and it did not come back to the clinician from us.
Guard is model output, not a clinician's judgement, so none of this proves its 18 are all right. What it says is that two independently built checkers disagree almost completely about what a behavioural-health note dropped, and that the omission side is where to look next.
What happens now
Three defects go into our issue tracker before this publishes, each with the result file that reproduces it:
- The judge returns Unsupported for statements it acknowledges are in the transcript, on note-section grounds. Four instances.
-
neg-001fires on a denial stated on the line immediately after the question, with correct speaker attribution. Two instances, reproducing on renormalised input. -
num-001compares digits only and does not resolve spelled-out numbers. One instance, reproducing on renormalised input.
Carried from installment 01 and still open in reduced form: the omission list names facts the note already contains. Closed and verified closed here: the negation rule on short affirmative denials, the judge accepting inferred assessments and unsaid plan items, and inconsistent treatment of None placeholder lines.
None of this changes what we say Note Check is. It is a review aid that reports what it found, with the clinician reading the whole note. It is not a safety guarantee and we do not sell it as one. We publish counts and examples, never a rate. There is no accuracy figure, no recall figure and no catch rate for Note Check in this piece, in the report behind it, or anywhere else we write, and there will not be one until a clinician-adjudicated reference set exists and is published.
Every input here is regenerable by anyone. The corpus is public and MIT-licensed, and this run is pinned to commit d828ea9562324f51813f3e5bd1f589637efc187d of Omi Health medical-note-eval. Behind the piece sit 67 result files, the run summary for every check, the base-against-Guard comparison, a machine-readable file of every disagreement with its transcript excerpt and note sentence, and every script. Write to info@krasyn.com and we will send that set.
Send us a pair
Every installment runs reader-submitted pairs in the next one, in full, with the result published whether or not it flatters us. Send a de-identified synthetic transcript and the note written from it.
Synthetic only. We do not accept real patient data, de-identified or otherwise, and we never will.
Dialogues and notes: Omi Health medical-note-eval, MIT license, copyright (c) 2025 Omi Health B.V.
Top comments (0)