My MRI interpretation vision model pointed at the right kidney and wrote "left"

I build a tool that reads MRI and CT scans with general vision models and explains the findings in plain language. The most repeated error in that pipeline isn't a missed tumour or an invented measurement. It's left and right.

We ran eight real studies through four readers (Claude, Gemini, Grok and Google's MedGemma), with the radiologist's signed report as ground truth. Here's what came back:

• a 5 cm cyst in the right kidney, placed in the left by two of the four readers
• a medial meniscus tear, reported in the lateral meniscus
• a left paracentral disc extrusion, called right-sided
• one reader that gave two different sides for the same finding in two acquisitions of the same study

In medicine a left/right swap is a "never event". You don't ship a report that does this, and a better prompt doesn't fix it. So here is where the error comes from, and the small, model-free check that now catches it.

The model gets the coordinates right and the word wrong

The detail that cracked it came from MedGemma. We ask each reader for two things per finding: a sentence, and a location on a specific slice. On one study MedGemma put its crosshair 17–26 mm to the right of the midline, which matched the radiologist. Then it wrote that the finding was "slightly to the left".

The crosshair was correct. The sentence described the side of the picture, not the side of the patient.

That's the radiological convention catching the model out. An axial scan is displayed as if you're standing at the patient's feet looking up, so the patient's right is on the left of the screen. A model trained mostly on ordinary photos describes what it sees: "on the left of the image". In a report, that's the wrong side.

This failure turns out to be useful. When the coordinates are right and only the sentence is wrong, you can check one against the other without involving a model at all.

World space already knows which side is which

Our volumes are converted from DICOM to NIfTI in the browser (dcm2niix compiled to WebAssembly). A NIfTI file carries an affine: a 4×4 matrix mapping voxel indices (i, j, k) to millimetres in world space. That space is RAS: +x is the patient's Right, +y Anterior, +z Superior.

So once a finding has a voxel location, the sign of its world-x gives a second, independent answer about which side it's on:

export function voxToMm(affine: Affine, [i, j, k]: [number, number, number]) {
  const m = (r: number) =>
    affine[r][0] * i + affine[r][1] * j + affine[r][2] * k + affine[r][3]
  return [m(0), m(1), m(2)] as [number, number, number]
}

const OFF_MIDLINE_MM = 8

export function sideOfPoint(mm: [number, number, number]): Side | null {
  const x = mm[0]
  if (!Number.isFinite(x) || Math.abs(x) < OFF_MIDLINE_MM) return null
  return x > 0 ? 'right' : 'left' // RAS: +x is the patient's RIGHT
}

The 8 mm dead zone matters. A central disc extrusion sits at x ≈ 0 and belongs to neither side, and floating-point noise shouldn't get read as a side.

Reading the side out of the sentence, in nine languages

Reports come back in the user's language, so an English-only regex would silently check nothing for most users. The side words are prefix patterns, because almost every one of these languages inflects them (destro/destra, rechts/rechten, prawy/prawa):

const RIGHT_WORDS =
  /\bright\w*|\bdestr[aeio]\w*|\bderech\w*|\bdroit\w*|\brecht[aensr]\w*|\bdireit\w*|\bpraw(?!d)\w*|\bsağ\w*|يمين|أيمن|اليمنى/iu
const LEFT_WORDS =
  /\bleft\w*|\bsinistr\w*|\bizquierd\w*|\bgauche\w*|\blink[aensr]\w*|\besquerd\w*|\blew\w*|\bsol\b|يسار|أيسر|اليسرى/iu
const BOTH_SIDES =
  /\bbilateral\w*|\bbilaterale?\w*|\bbeidseit\w*|\bobustronn\w*|\bambos\b|\bambas\b|\bdes deux côtés\b|ثنائي/iu

Two traps are handled on purpose:

• "bright" must not match "right". It doesn't, because \b needs a non-word character before the r, and b is a word character. MRI reports say "bright" constantly.
• Polish prawdopodobnie ("probably") must not match praw. Hence the (?!d).

Then there's the rule that took longest to get right: a sentence that names both sides claims neither.

export function sideClaimed(text: string): Side | null {
  if (!text) return null
  if (BOTH_SIDES.test(text)) return null
  const right = RIGHT_WORDS.test(text)
  const left = LEFT_WORDS.test(text)
  if (right === left) return null // none, or one of each
  return right ? 'right' : 'left'
}

"The left L5 root is displaced; the right is not" mentions both words. If you picked either one, you'd be inventing a claim the sentence never made.

The check only works when the scan crosses the midline

This is the part I got wrong first. A left knee MRI sits entirely at negative x. Every finding in it is "on the left", while the text is talking about the medial and lateral compartments of that one knee. Run the check there and every knee report gets flagged.

So the sign of x only means a body side when the field of view reaches well past the midline in both directions. You can't read that off the origin, because the affine can rotate the grid. You have to check all eight corners:

const MIDLINE_MARGIN_MM = 40

function worldXRange(affine: Affine, [nx, ny, nz]: [number, number, number]) {
  let lo = Infinity, hi = -Infinity
  for (const i of [0, nx - 1])
    for (const j of [0, ny - 1])
      for (const k of [0, nz - 1]) {
        const x = voxToMm(affine, [i, j, k])[0]
        lo = Math.min(lo, x); hi = Math.max(hi, x)
      }
  return [lo, hi]
}

export function straddlesMidline(affine?: Affine, dims?: [number, number, number]) {
  if (!affine || !dims) return false
  const [lo, hi] = worldXRange(affine, dims)
  return lo <= -MIDLINE_MARGIN_MM && hi >= MIDLINE_MARGIN_MM
}

The 40 mm margin is generous on purpose. A spine or abdomen study clears it easily, and a limb never does.

Putting it together: only report a conflict when you're sure

export function lateralityConflict(opts: {
  text: string
  locations?: { mm?: [number, number, number] }[]
  affine?: Affine
  dims?: [number, number, number]
}): Side | null {
  const claimed = sideClaimed(opts.text)
  if (!claimed) return null
  if (!straddlesMidline(opts.affine, opts.dims)) return null

  // Several points must agree with each other before they outvote the sentence.
  const sides = (opts.locations ?? [])
    .map((l) => (l.mm ? sideOfPoint(l.mm) : null))
    .filter((s): s is Side => s !== null)
  if (sides.length === 0 || !sides.every((s) => s === sides[0])) return null

  return sides[0] === claimed ? null : sides[0]
}

Count the return nulls. There are four ways for this function to say nothing, and one way for it to speak. That's deliberate. It isn't a classifier trying to arbitrate uncertain cases. It exists to catch the contradictions that are certain: the text names one side, the geometry clearly and consistently shows the other, and the scan covers both sides of the body.

What we do when it fires

We don't silently rewrite the sentence. The crosshair is usually right (that's the failure we measured), but the sentence is the part a person reads and acts on. So the conflict is stored on the finding (sideConflict: 'right') and shown, instead of being quietly "fixed" by code that could itself be wrong.

What this doesn't fix

• Extremities. Knees, shoulders and wrists get no check at all. Medial vs lateral inside one joint needs a different reference: the joint's own axes, not the body midline.
• Findings with no location. No coordinates, no second opinion.
• Errors both halves share. If the model puts the crosshair on the wrong kidney and names that kidney, the two agree and nothing fires. That's also why having two models agree doesn't prove much: independent readers catch errors that differ, but they're blind to errors they share.

The general lesson

When a multimodal model gives you structured output next to prose, the two halves often fail independently. Coordinates, slice indices, bounding boxes and counts come from a different part of the model's behaviour than the sentence does. Whenever the domain gives you a deterministic way to relate the two (here: an affine matrix and a coordinate convention), you get a check that costs microseconds, needs no second model, and can't hallucinate.

It's worth finding those checks before reaching for another prompt.

---

I'm Lorenzo, and I build ReadYourScan, which opens DICOM/NIfTI studies in the browser and explains them in plain language. It's not a diagnosis. The browser viewer is open source (MIT): lgdesignee/readyourscan-viewer.

Comments

[Jo Do]

The midline eligibility check is what makes this more than a clever regex. It first proves the geometry can answer the question, then compares that answer with the language. I would log three outcomes separately: eligible and consistent, eligible and contradictory, and ineligible. Otherwise a dashboard full of no alerts can hide that most studies never ran the check. In a medical pipeline, check coverage is as important as check accuracy.