DEV Community

Selva
Selva

Posted on

I shipped an npm package that fails your build when an LLM leaks PHI

#ai #phi #typescript #hippa

The problem nobody had a clean answer to

I build LLM features over clinical text. Before shipping any of them, I wanted one boring guarantee: if the model ever puts a patient identifier in its output, the build fails.

I went looking for a tool. Every LLM-eval library I found (autoevals, promptfoo, vitest-evals, evalite) fell into one of two camps: no concept of PHI at all, or it graded output by sending it to a hosted API. The second is a non-starter — when the thing you're evaluating is patient data, mailing it to a third party is the exact risk you're trying to avoid.

So I built phi-leak-guard.

What it is

A zero-dependency TypeScript library that detects PHI/PII in text and fails your test suite when it finds any. It runs locally in Vitest/Jest — no network, no model, nothing leaves the process.

import 'phi-leak-guard/vitest';

 test('clinical summary never leaks PHI', () => {
   expect(summarize(patientNote)).toContainNoPHI();
});
Enter fullscreen mode Exit fullscreen mode

When it fails, it tells you exactly what leaked:

Expected output to contain no PHI, but found 2:

  • [name] "John Smith" (pattern)
  • [nhs-number] "943 476 5919" (validated)

The interesting part: precision without a model

The hard problem with deterministic PHI detection is false positives. A naive \d{10} regex flags every order number as an NHS number and gets disabled within a day. The trick is to validate, not just pattern-match:

  • NHS numbers → Modulus-11 checksum
  • Vehicle VINs → ISO-3779 check digit
  • IPv4 → octet range validation
  • SSN / NINO → structural + prefix rules

A random 10-digit number fails the checksum, so it isn't flagged. Matches are tagged validated (checksum-backed) or pattern (regex/context) so you can see how much to trust each hit.

Staying honest about coverage

HIPAA Safe Harbor is a finite list of 18 identifier categories — a completable target. UK GDPR "personal data" is open-ended, so the library covers common direct identifiers, not "everything." A coverageReport() prints exactly what is and isn't checked per standard, and three categories (biometrics, photos, "any other identifier") are reported as not detectable in text rather than pretended-covered.

It's not a compliance certification — it's a regression gate that reduces risk. And deterministic matching can't catch paraphrased re-identification or every name, so there's a pluggable seam to drop in an NER model.

Try it

npm install --save-dev phi-leak-guard

It's MIT, ships ESM + CJS + types, and has a synthetic benchmark (precision 1.00, recall 0.97). Repo: https://github.com/selvassn/phi-leak-guard — I'd genuinely love feedback on which identifiers to add next.

Top comments (0)