DEV Community: vadim albarov

Windows Copilot Inverted a HIPAA Rule Three Times - and Cited a Real Paper That Proves the Opposite

vadim albarov — Thu, 20 Aug 2026 03:49:01 +0000

I have a two-window habit. My main work happens in a Claude Code session, and I don't like burning its context on side questions - so for quick lookups I alt-tab into Windows Copilot and use it as a search-flavored notepad. Ask, skim, close, back to work. It's been part of my routine for a long time, and honestly, I'd never had a problem with its answers before - side questions came back reasonable, and nothing ever sent me down a wrong path. Low stakes, or so I thought.

This week the side window taught me a HIPAA rule with total confidence. The rule was backwards. Not vague, not incomplete - inverted, in the one direction that would turn a compliance question into a breach.

I build healthcare infrastructure, so I had the reflexes to check. This is the story of what it said, what the regulation actually says, how the wrong answer survived thinking mode, search mode, and a demand for references - and how the citation trail eventually explained where the inversion came from.

The question

I was thinking about deterministic pseudonymization: replacing a patient identifier with hash(key + identifier) so the same patient always maps to the same token and records stay joinable. The obvious follow-up is what HIPAA thinks about the key. Keep it secret? Share it? Does it matter? Full disclosure: the question I actually typed was leading - I'd already run into the claim that sharing the key was the compliant option somewhere, and I asked about it as if it were established.

Copilot's answer, summarized:

If you keep a secret key, you retain the ability to re-identify. Therefore the data is pseudonymized, still PHI, not de-identified.
If the key is public, nobody has privileged knowledge. Nobody can reverse a one-way hash. Therefore the data can qualify as de-identified.
HIPAA, it explained, cares about privileged access, not cryptographic strength. "You're thinking like an engineer. HIPAA is written by lawyers."

It's a seductive answer. It has a philosophy. It flatters you for being confused. It reads like someone explaining a genuinely counterintuitive corner of law.

It is also wrong on both branches, and the second branch is dangerous.

What the regulation actually says

The relevant text is 45 CFR 164.514. Safe Harbor's identifier list ends with a catch-all - "any other unique identifying number, characteristic, or code" - with exactly one exception: a re-identification code that satisfies paragraph (c). Paragraph (c) has two prongs:

Derivation. The code must not be derived from or related to information about the individual, and must not be otherwise translatable back to the individual.
Security. The covered entity must not use or disclose the code for other purposes and must not disclose the mechanism for re-identification.

Read those against the two branches:

Secret-key hash. HMAC(secret, MRN) fails Safe Harbor - but not for Copilot's reason. It fails because the token is mathematically derived from the identifier, and prong 1 prohibits derivation outright, regardless of how well you guard the key. Meanwhile, Copilot's actual claim - "if you can re-identify, it's not de-identified" - contradicts the regulation's text. Paragraph (c) exists precisely so a covered entity can keep a secret re-identification mechanism while the dataset remains de-identified. Retained re-identification capability is not the disqualifier; derivation and disclosure are.

Public-key hash. This one fails everything at once. Publishing the key is literally "disclosing the mechanism for re-identification" - prong 2, verbatim. And it makes the data trivially translatable - prong 1 - because health identifiers live in small, enumerable spaces. SSNs are a 10^9 space. Phone numbers, MRNs, emails, name-plus-birthdate: all enumerable. With the key public, anyone hashes every candidate and matches your entire column in seconds on a laptop. One-wayness protects high-entropy inputs; it does nothing for a nine-digit number.

HHS's own de-identification guidance closes the loop from both sides. It says a hash without a secret key counts as an identifying element exactly because recipients can reverse it over the input space. And it says keyed cryptographic hashes are acceptable under the Expert Determination pathway provided the keys are not disclosed, including to the recipients.

So the real rule is the mirror image of what my sidebar told me:

Construction	HIPAA status
Hash with a published key	Fails everything - discloses the mechanism, trivially reversible
Hash with a secret key	Fails Safe Harbor (derived code); acceptable under Expert Determination with the key undisclosed
Random token + protected mapping table	The pattern 164.514(c) actually blesses - a random value is not "derived from" anyone

Round two: references make it worse

Maybe I asked badly. I turned on thinking mode, then search mode, and asked it to rethink and support its answer with references.

I got the same inverted conclusion, now decorated with emoji section headers, a verdict table, and a source list - and with something worse. The centerpiece was a quoted sentence attributed to HHS, saying a code derived from PHI is an identifier "unless the re-identification key is not retained." I searched for that sentence. Not on HHS.gov, not anywhere I could find. The model composed a plausible-sounding rule that swaps the regulation's actual verb - disclose - for retain, wrapped it in quotation marks, and attached the genuine HHS URL to it.

That's the failure mode that stuck with me: a real link laundering a fake quote. Every reader's citation heuristic - "it links to hhs.gov, so it's grounded" - defeated by construction.

The funny part is that the correct answer was present in the same response, scattered in the margins. The risks section admitted that a public pepper can be brute-forced over common identifiers - which quietly destroys the headline claim that public-key hashing is irreversible. Another bullet correctly noted that keeping a secret key for linkage requires expert determination - which is the actual rule, and refutes the answer's own bottom line. The model had all the pieces and still shipped the inverted conclusion in the headline, the table, and the summary.

Round three: it's not me

One hypothesis left: maybe my framing poisoned the well, since my first question presented the counterintuitive claim as a finding. So I opened a fresh chat and asked the neutral, symmetric question - which is better for HIPAA, hash with a private key or a public key - with no premise embedded.

Verdict, verbatim in spirit: public-key hashing is "far better," private-key hashing is still PHI, and - my favorite line - the public pepper should be "long, random, and not guessable."

A public value that is not guessable. It's published. That single phrase is the whole confusion in miniature: the answer needs the key to be simultaneously known to everyone (so nobody has privileged access) and known to no one (so nobody can brute-force). For calibration, I asked ChatGPT's web version the same neutral question. It got it essentially right: public-key hashing rejected for exactly the dictionary-attack reason, keyed HMAC labeled as pseudonymization that doesn't exit PHI obligations by itself, random tokens with a secured mapping table recommended for real de-identification. Not perfect - it never cited the derivation prohibition - but directionally sound everywhere it committed.

Same question. One product inverted, reproducibly, across independent chats. The other didn't.

The citation trail explains everything

Both of the reference-backed Copilot answers - rounds two and three - leaned on the same academic source: a 2003 AMIA paper by Landi and Rao, "Secure De-identification and Re-identification". I looked it up.

The paper describes an asymmetric encryption scheme: encrypt patient identifiers with a public key so that only the holder of the matching private key - the data owner - can decrypt and re-identify. "Public key" as in public-key cryptography. One half of a keypair. A system whose entire security rests on the private key staying secret, and whose explicit purpose is to let the owner retain re-identification capability.

Now the most plausible failure chain is visible. Retrieval surfaced a paper with "public key" and "de-identification" in close proximity. Summarization collapsed public-key cryptography into publicly known key. And then - this is the part I find genuinely instructive - the model didn't just misread a term. It constructed an entire regulatory philosophy around the misreading: the "privileged knowledge" theory of HIPAA, delivered with the confidence of a law professor, appearing in no regulation, contradicted by the very paper being cited. The citation that was supposed to ground the answer was proof of the opposite.

Which model was that, even?

The chat window said "Smart." That's a mode label, not a model name. So I asked Copilot directly which model it runs on. It denied running on any outside lab's models at all - just "proprietary Windows AI technologies" - and said it cannot disclose specifics.

That answer is worth exactly nothing, and knowing why it's worth nothing is the useful part. A chatbot's claim about its own identity is generated text like everything else it says - models have no introspective access to the infrastructure serving them, and "I can't disclose" deflections are typically system-prompt policy, not knowledge. The same product that invented an HHS quote is not a reliable witness about its own internals.

The public reporting says something different and messier: modern assistant products run mixed fleets - frontier models licensed from partner labs alongside the vendor's own in-house models - with a cost-driven router deciding invisibly, per query, which one you get. The lineup shifts between announcements, the routing shifts without any announcement at all, and there's no per-response indicator. The same question may be served by a different model next week, or later today.

I don't know whether my three inverted answers came from a fast-path model, from the summarization layer garbling search results, or from the persona tuning that opens responses with "let me say this clearly and directly." That's the point: as a user, I can't know - and asking the product just adds one more unverifiable claim to the pile. When another assistant answers the same question correctly, the difference isn't necessarily raw model capability. It's everything wrapped around the model, and the wrapper is invisible. "Which LLM does this product use" turns out to be the wrong question. The right one is "which pipeline, with which retrieval, which router, and which incentives" - and no consumer product answers it. Not even when you ask it directly.

What I'm taking away

Confidence formatting is free. Tables, verdict emoji, "let me say this clearly" - none of it correlates with correctness. The most wrong answer in this story was the best-formatted one.
"Support this with references" does not mean "verify this." Once a model has committed to a conclusion, asking for sources can produce justification instead - up to and including an invented quote pinned to a real government URL. If a quote matters, search for the exact sentence.
Watch for internal contradictions. The wrong answers refuted themselves in their own risk sections. A response whose caveats disagree with its headline is telling you which part was retrieved and which part was composed.
For anything regulatory, read the primary source. 164.514(c) is two paragraphs. It cost five minutes and settled in one reading what three AI answers scrambled. Compliance-by-chatbot is how a "de-identified" dataset ships with a published key and becomes a reportable breach.
The side window deserves the same skepticism as the main one. My mistake wasn't using Copilot - it was granting the quick-lookup window a lower evidence bar because the questions felt small. Nothing about the window makes the answers smaller.

I'm keeping the two-window setup, but the notepad seat is now vacant. Either something changed in that product - a router, a model swap, a summarization layer - or it could always do this and I simply never caught it; I have no way to tell, and that uncertainty is its own verdict. Either way, a tool I trusted for months just fabricated regulatory quotes with a straight face. So I'm dropping it until it stabilizes, and I'll know it has stabilized the same way I learned it broke: by spot-checking its answers against primary sources. A quick-lookup tool that requires verification of every answer isn't a quick-lookup tool anymore.

Try it yourself

Here is the prompt I used for the fresh-chat test, lightly tidied. Run it against your assistant of choice:

In terms of HIPAA compliance, what is better for de-identifying PHI data: HASH(private_key + PHI) or HASH(public_key + PHI)?

The correct answer rejects both as de-identification on their own, flags the public-key variant as trivially reversible by dictionary attack, and mentions that a keyed hash only works under the Expert Determination pathway with the key kept undisclosed. Anything that tells you the public key is the compliant option has inverted 45 CFR 164.514(c).

Share what you get in the comments - which product, which mode, and which way it landed. I'm genuinely curious whether this reproduces beyond my machine, and a comment thread of timestamped outputs is a better dataset.

Disclaimer: I'm an infrastructure engineer, not a lawyer; this is an engineering read of published regulations and guidance, not legal advice. The Copilot and ChatGPT responses summarized here were collected in August 2026 from consumer versions of both products; model routing and behavior change constantly, which is rather the point.

The De-identification Datasets Problem: i2b2/n2c2 Access Is Broken, and What to Do Instead

vadim albarov — Wed, 19 Aug 2026 03:27:41 +0000

TL;DR: I'm building a PHI de-identification tool, and step one of doing that honestly is benchmarking it against the standard corpora: the i2b2 2006 and 2014 de-identification challenge datasets, now distributed as "n2c2" through Harvard DBMI's data portal. Step one failed. The portal's n2c2 page has said "Temporarily Unavailable" since at least 2026-07-28, registration is closed, and the old i2b2.org dataset page returns HTTP 500. Even when the door is open, access means per-user registration, a data use agreement, and an approval wait - which is why published de-id numbers are so hard to reproduce and why the field's canonical scores are 12-20 years old. I built a synthetic corpus generator instead. Here's the whole investigation, dated, plus what synthetic data honestly can and cannot replace.

Step one: you can't get the data

Benchmarking a de-identification tool sounds like the easy part of the project. The hard part is supposed to be the NLP: finding every patient name, date, medical record number, and phone number buried in messy clinical prose, so the notes can be used for research without exposing anyone. Detection is the science. Evaluation is just downloading the test set, right?

Here is how evaluation actually went for me.

Every de-identification paper of the last two decades benchmarks against the same corpora: the i2b2 de-identification challenge datasets from 2006 and 2014. If you want your numbers to mean anything to anyone, you report them on i2b2 2014. So on 2026-07-28 I went to get the data from the Harvard DBMI Data Portal, where the corpora live under their post-2018 name, n2c2 (National NLP Clinical Challenges). What I found was a notice: the n2c2 datasets are temporarily unavailable.

Fine, I thought. Maintenance happens. I built other things and came back.

As of today, 2026-08-18, nearly three weeks later, here is the exact state of the front door, checked from a fresh session:

The portal itself is up and loads fine.
The portal's data sets listing shows two entries. Neither is an n2c2 NLP corpus (both are 4CE COVID-related sets).
The n2c2 NLP Research Data Sets project page - the page for the actual challenge corpora, 2006 through 2018 - carries this banner, verbatim: "Temporarily Unavailable. The n2c2 datasets are temporarily unavailable. If you are trying to access data from the 2019 Challenge, tracks 1 (Clinical Semantic Textual Similarity) and 2 (Family History Extraction) are available directly through Mayo Clinic." Below the dataset descriptions: "Registration is not open ... at this time."
The legacy home of these corpora, i2b2.org/NLP/DataSets, returns HTTP 500.
The companion n2c2 informational site refused my scripted requests outright with an Akamai "Access Denied" page. That one I'll hedge: it may just be bot filtering rather than downtime. But it means I cannot even verify the documentation programmatically.

No ETA. No status page. No mirror. The canonical benchmark for an entire subfield of clinical NLP is a "temporarily unavailable" banner, and "temporarily" has meant at least July 28 through August 18 so far. It may come back tomorrow; the point of this article survives either way, because the outage is only the loudest symptom of a structural problem.

What these datasets are, and why everyone needs them

Some history, because the names are confusing. i2b2 (Informatics for Integrating Biology and the Bedside) was an NIH-funded center based at Partners HealthCare that, starting in 2006, ran annual clinical NLP shared tasks on real (de-identified) hospital notes. In 2018 the challenge series was renamed n2c2, and stewardship of the datasets moved to the Department of Biomedical Informatics at Harvard Medical School, distributed via their portal. Same corpora, three names, one door.

Two of those challenges define de-identification evaluation to this day:

The 2006 challenge (Uzuner, Luo, and Szolovits, JAMIA 2007) used hospital discharge summaries in which the authentic PHI had been replaced with synthesized surrogates - including deliberately out-of-vocabulary, made-up names to punish systems that just memorized name lists. Seven teams, sixteen system runs, and the best systems scored above 98% F-measure across PHI categories.

The 2014 i2b2/UTHealth challenge (Stubbs and Uzuner, JBI 2015) raised the bar: 1,304 longitudinal records covering 296 patients, over 28,000 annotated PHI instances, annotated under a broad interpretation of HIPAA. Human annotators managed a token-level F1 of 0.927 against the gold standard; the best automated system hit a strict micro-averaged F1 of 0.936.

Notice something about both: even the "real" gold standards contain synthetic PHI. The notes are genuine clinical text, but the identifiers in them are surrogates, inserted so the data could be released at all. Keep that in mind for later - the field's ground truth has always been real prose plus fake identifiers.

That 0.936 from 2014 is, functionally, still the number. When you publish a de-id tool in 2026, reviewers ask how you compare on i2b2 2014. The corpus is twelve years old, drawn from one hospital system, and pre-dates most of what modern EHRs do to note formatting. It is also, at the moment, undownloadable.

The access process, even on a good day

Suppose the portal comes back tomorrow. What does access look like then? Per the archived instructions on i2b2.org, the datasets are "freely available" to researchers - subject to a data use agreement, and "each individual user must access the data independently" through the portal. In practice:

Register an account on the DBMI portal.
Sign the Rules of Conduct and the Data Use Agreement (the n2c2 DUAs are real legal documents, not click-through checkboxes).
Wait for a human to approve you. There is no published turnaround time.
Repeat for every individual on your team, because the DUA is per-person, not per-lab.
Never redistribute the data - which also means never shipping it as a test fixture, never putting it in CI, never publishing your evaluation harness with the inputs included.

This is not villainy. The data is real patient prose; a DUA is the legally and ethically appropriate wrapper, and the people who built and maintain these corpora did the field an enormous service. But note the architecture: the standard benchmark for a global research area is administered by one team at one institution, through one portal, with per-user paperwork and no fallback. When that single point of distribution goes down - for maintenance, for a compliance review, for a staffing gap, for whatever is happening right now - the benchmark simply ceases to exist for anyone who doesn't already have a copy.

Why this hurts more than my project

Play the incentives forward and the damage compounds:

Published results become unreproducible in practice. A paper says "0.94 F1 on i2b2 2014." You cannot check that claim, cannot run the same test set through your own tool, cannot even eyeball the annotation decisions the score depends on. Reproducibility in de-id research is gated on a portal login, and today on a portal banner.

New tools cannot compare against prior art. The literature has twenty years of numbers on these corpora. A new open-source tool that cannot access them either skips comparison (and gets dismissed) or quotes other papers' numbers against its own results on different data (and misleads).

The canonical numbers fossilize. Because making a new shareable gold standard from real notes is brutally expensive - the 2014 corpus took double annotation, arbitration, and multiple proofreading rounds to reach that 0.927 human F1 - nobody replaces the old benchmarks. The field's reference points are frozen in 2006 and 2014 while clinical documentation, and the models reading it, changed completely.

Insiders and outsiders diverge. Groups with long-standing access or local hospital data keep publishing; independent developers and open-source maintainers evaluate on whatever they can scrape together. The people most likely to ship a de-id tool you can actually download are the least able to prove it works.

The alternatives tour

Before building anything, I did the diligence on every other door:

PhysioNet. The MIMIC family of ICU databases includes clinical notes, and PhysioNet also hosts a gold standard corpus of 2,434 de-identified nursing notes with realistic surrogate PHI (Neamatullah et al., 2008). Access requires becoming a credentialed user: identity verification, the CITI "Data or Specimens Only Research" training course, and then a separate DUA per dataset. To PhysioNet's credit, this process is documented, predictable, and actually functioning. But it is still weeks of process, still per-person, still non-redistributable. And MIMIC's notes are already de-identified with placeholders, so to evaluate a de-id tool you must first re-inject surrogate PHI - which is exactly what prior work on synthetically-identified MIMIC notes does. The "real data" path quietly becomes a synthetic-PHI path anyway.

MTSamples. MTSamples.com hosts thousands of publicly available transcribed sample medical reports across dozens of specialties. No registration, no DUA, real clinical language structure. The catch: they are samples, so they contain no PHI to find. To make a de-id benchmark out of them you inject fake PHI into the text - an approach with an established lineage in the literature. This is the honest public option, and it is the spirit my workaround follows.

Synthea. Synthea generates fully synthetic patients with medically plausible histories - fantastic for structured FHIR data. But its narrative output is template-driven fill-in-the-blank SOAP text linked to the structured record. For de-id evaluation, where the entire game is the messiness of real prose - copy-paste artifacts, headers, inconsistent formatting, abbreviations - templated narrative is the wrong distribution by design.

Newer synthetic clinical text efforts. This space is heating up: Synthetic4Health generates annotated synthetic clinical letters; ASQ-PHI proposes an adversarial synthetic benchmark specifically for de-identification; and there is active work on whether LLM-generated notes actually match real note distributions (early answer: imperfectly, and you should measure the gap rather than assume it away). None of these is yet a community-standard replacement for i2b2 2014. All of them are bets on the same thesis I ended up betting on.

The workaround: a synthetic corpus generator

So I built my own synthetic corpus generator for the PHI de-identification tool I'm building. The design is deliberately boring:

Templates styled after real clinical note genres - discharge summaries, progress notes, radiology reports - with the section structure, boilerplate, and formatting quirks those genres actually have.
Controlled PHI injection. Fake names, MRNs, dates, phone numbers, addresses, providers, and facilities generated and inserted at known offsets. Every injected entity is recorded with its exact span and category at generation time.
Category balancing. Real corpora are dominated by dates and names; rare categories (fax numbers, device IDs, URLs) barely appear. A generator can oversample the rare stuff so your recall numbers on those categories are backed by hundreds of instances instead of four.

The advantages are structural, not incidental:

Labeled by construction. No annotation budget, no inter-annotator disagreement, no 0.927 ceiling on ground-truth quality. The generator knows where the PHI is because it put it there.
Shareable. No DUA, because there is no patient. The corpus - and more importantly the generator - can live in a public repo, run in CI, and ship as test fixtures. Anyone can regenerate the exact evaluation set from the code and a seed. That is a property the i2b2 corpora can never have.
Scalable and steerable. Need 10,000 more notes with dates in weird formats? That's a parameter, not a grant application.

The honest limits

Here is where I'm obligated to argue against myself, because synthetic evaluation has failure modes that will flatter your tool if you let them:

Distribution shift. My templates are styled after clinical notes; they are not clinical notes. Real notes contain dictation artifacts, OCR junk, mid-sentence copy-paste, and formatting chaos that no template library fully reproduces. A recall number earned on synthetic text is an upper bound, not an estimate, of real-world recall. There is emerging work on quantifying exactly this synthetic-to-real gap for PHI taggers, and the gap is real.

Overfitting to your own generator. This is the insidious one. If the same mental model writes both the fake-name generator and the name-detection logic, the benchmark and the tool share assumptions, and you are grading your own homework. Mitigations exist - independent sources for injection values, formats the detector wasn't designed around, adversarial edge cases - but the risk never reaches zero.

It complements real-data validation; it does not replace it. My position after all this: synthetic corpora are for development, regression testing, category-level diagnostics, and public reproducibility. Before anyone trusts a de-id tool with actual patient data, it needs validation on real notes under proper governance - PhysioNet credentialing, an institutional dataset, or the n2c2 corpora if the door ever reopens. Remember, though, that even those gold standards are real prose with surrogate identifiers. The line between "real benchmark" and "synthetic benchmark" was always a spectrum, not a wall.

What better infrastructure would look like

None of this requires new science. It requires plumbing:

A status page and a mirror. If a dataset is the reference benchmark for a field, "temporarily unavailable" with no ETA on a single portal should be impossible. PhysioNet already demonstrates the model: documented process, predictable credentialing, many datasets under one durable roof.
Standing distribution instead of single-lab stewardship. Move canonical corpora to infrastructure whose job is distribution, with credentialing handled once per user, not per corpus per portal.
Publish generators, not just corpora. A community-maintained synthetic benchmark - generator code plus seeds, calibrated against real data - would give the field something no DUA can: an evaluation anyone can run, extend, and verify. The recent synthetic-benchmark papers are steps in this direction; what's missing is convergence on one that leaderboards accept.
Two-track evaluation as the norm. Report on the gated real corpus for comparability, and on an open synthetic corpus for reproducibility. Either number alone is half a result.

Takeaways

Check the data before you plan the benchmark. The canonical de-id corpora are behind a per-user DUA on one portal, and that portal's n2c2 datasets have been "temporarily unavailable" from at least 2026-07-28 through 2026-08-18, with the legacy i2b2.org page throwing HTTP 500. Verify the current state yourself; date what you find.
Single-institution gatekeeping of a field's benchmark is a reliability bug, independent of any outage. Per-user DUAs also mean no CI, no fixtures, no redistribution - reproducibility is structurally capped.
The field's reference numbers are from 2006 and 2014. Best strict F1 of 0.936 on 1,304 notes from one hospital system is still the bar new tools are measured against.
PhysioNet is the functioning real-data path - credentialing, CITI training, per-dataset DUA - but its notes need surrogate PHI re-injection for de-id evaluation anyway.
Synthetic corpora buy you labels by construction, shareability, and category balance. They cost you distribution realism, and they tempt you into grading your own homework. Use them for development and public reproducibility; validate on real data before production.
The gold standards were already part synthetic. Real notes, surrogate PHI. Synthetic evaluation isn't a betrayal of rigor - unexamined synthetic evaluation is.

If you work in clinical NLP: how did you get your benchmark data, and how long did access take? And if anyone has current information on when the n2c2 datasets are coming back, the comments are open.

Is 'HIPAA-Eligible' the Same as 'HIPAA-Compliant'? I Audited AWS's List Against the Security Rule

vadim albarov — Tue, 18 Aug 2026 03:53:27 +0000

Here is a sentence I have heard, in some form, from three different engineers and one vendor sales deck: "We're fine, we only use HIPAA-eligible AWS services."

I build healthcare infrastructure for a living, and every time I hear it I do the same mental translation: "We're fine, our data center signed a contract." It's true, it's necessary, and it answers roughly none of the questions a HIPAA auditor will ask you.

So I decided to make the gap measurable. I took six AWS services that show up in practically every PHI-handling stack - S3, RDS for PostgreSQL, ECS on Fargate, KMS, the CloudTrail/CloudWatch audit pair, and the ALB - and audited each against the Security Rule's technical safeguards (45 CFR 164.312): what the regulation demands, what the service does by default, and what you must configure yourself.

Spoiler: of the 35 concrete controls I mapped, the defaults covered 6.

What "HIPAA-eligible" actually means

Start with what AWS itself says. The HIPAA-eligible services reference defines eligible services as those that may "create, receive, process, maintain, or transmit" electronic protected health information (ePHI) - provided you have signed AWS's Business Associate Addendum (BAA) first. And then, in plain sight, the sentence everyone skips: "Customers still must configure these services consistent with HIPAA requirements."

The fast primer:

A covered entity (provider, health plan, clearinghouse) or a business associate (anyone handling PHI for them - your SaaS included, if hospitals are your customers) must comply with HIPAA. AWS becomes your business associate when you sign the BAA - today a self-service click in AWS Artifact.
The BAA covers AWS's side of the shared responsibility model: data centers, hypervisors, service internals - security of the cloud. Everything you configure - security in the cloud - stays yours.
The Security Rule is the part of HIPAA that talks to engineers. Its technical safeguards (164.312) name five standards: access control, audit controls, integrity, person or entity authentication, and transmission security.
Implementation specifications are required or addressable. Addressable does not mean optional - it means implement it, implement a documented equivalent, or document a defensible reason you did neither. Encryption at rest and in transit are both addressable, and every healthcare shop I know treats them as required: "we decided not to encrypt the PHI" is not a paragraph anyone wants to defend in a breach investigation.
And the kicker, straight from AWS's own HIPAA compliance page: "There is no HIPAA certification for a cloud service provider (CSP) such as AWS." There is no certification for you either. Only your configuration, your documentation, and eventually someone's audit.

So "HIPAA-eligible" means exactly one thing: the service is covered by AWS's BAA, so AWS's side of the split is contractually handled. Whether your side is handled is the rest of this article.

Last year I distilled a few years of building this into a set of internal Terraform modules for my team - one hardened block per service, each control traced to its Security Rule citation. The most useful artifact of that project was the diff between "what the module enforces" and "what AWS gave us out of the box." That diff is essentially this article.

The audit

Per service: what 164.312 asks, what you get by default, what you must add. Every default claim is sourced; where a default changed recently, I say when.

S3 - the poster child of improved-but-insufficient

Credit where due: S3's defaults have genuinely improved. Since January 2023, every new object is encrypted with SSE-S3, and you cannot turn that off. Since April 2023, new buckets get all four Block Public Access settings enabled and ACLs disabled. Two real 164.312(a) wins, for free. Now the rest:

Transmission security (164.312(e), required standard): S3 happily serves plaintext HTTP unless you attach a bucket policy denying aws:SecureTransport = false. Nothing does this for you.
Integrity (164.312(c)): versioning - your recovery path when an object is improperly altered or deleted - is off by default.
Audit controls (164.312(b)): server access logging is off by default, and object-level API logging (who read which object) doesn't come from S3 at all - it needs CloudTrail data events, also off by default (see CloudTrail below).
Encryption, but accountably: SSE-S3 encrypts with a key you can't scope, rotate on your terms, or audit the use of. For PHI you want SSE-KMS with a customer-managed key, plus a policy denying uploads that request the wrong key - so a misconfigured client fails loudly instead of writing PHI under the wrong crypto.

One non-obvious lesson from building the S3 module: keep patient identifiers out of object keys. Keys leak into access logs, CloudTrail events, error messages, and URLs - surfaces your PHI inventory forgot about.

RDS for PostgreSQL - encryption is opt-in, still, in 2026

Encryption at rest (164.312(a)(2)(iv)): storage encryption is not enabled by default at the API level - StorageEncrypted defaults to false - and can only be set at creation time. Forget it in your Terraform, and the fix is a snapshot-copy-restore migration, not a flag flip.
Transmission security (164.312(e)): a moving target. For RDS PostgreSQL 15 and later, rds.force_ssl defaults to 1; for 14 and older it defaults to 0. It also lives in a parameter group, one console edit away from silently becoming 0 again. In my modules it's pinned at the engine, so a client that forgets sslmode=require gets an error instead of a plaintext session.
Access control (164.312(a)(1)): nothing stops you from making the instance publicly accessible or dropping it in a public subnet. Private placement, security-group-to-security-group ingress (no CIDR allowlists - identities beat address ranges), and publicly_accessible = false are all decisions you must make and enforce.
Integrity (164.312(c)): deletion protection is disabled by default in the API and IaC (the console pre-checks it for production templates - your safety net depends on which tool created the database).
Authentication (164.312(d)) and audit (164.312(b)): RDS can generate and rotate the master password in Secrets Manager so it never touches Terraform state; it can export connection logs to CloudWatch. Both are opt-in. By default the master password is wherever you put it, and connection history is a shrug.

ECS on Fargate - the good defaults are the invisible ones

Fargate's best HIPAA property doesn't appear in any console checkbox: there are no instances. No AMIs to patch, no SSH, no node agents - a whole slab of the malicious-software surface moves to AWS's side of the BAA. That and IAM's deny-by-default (a fresh task role can do nothing) are real defaults in your favor. Everything else is on you:

Access control (164.312(a)(1)): nothing prevents tasks in public subnets with public IPs. Private placement is a choice you codify.
Authentication (164.312(d)): plain environment variables appear in DescribeTaskDefinition output and the console. Database credentials must go through the task definition's secrets integration with Secrets Manager - injected at container start, never written down.
Unique identity (164.312(a)(2)(i)): the execution role (ECS pulling images, injecting secrets) and the task role (your app) are separate identities - if you bother creating them separately instead of reusing one fat role.
Audit controls (164.312(b)): my favorite gap in this audit. ECS Exec - an interactive shell inside a container that may hold PHI in memory - records, by default, only the ExecuteCommand API call in CloudTrail. What happened inside the session goes unrecorded unless you configure session I/O logging to an encrypted log group. A shell in a PHI container is an access event; out of the box, it's one with no transcript.

KMS - the service that only helps if you invite it

KMS is HIPAA-eligible, excellent, and entirely opt-in. Every encryption integration on this page uses KMS if you wire it in; the eligible list has no opinion about whether you did. Even once you create customer-managed keys:

Automatic key rotation is disabled by default for customer-managed keys (AWS-managed keys rotate yearly; the keys you control don't, until you say so).
The default key policy hands the account root full access and calls it a day. Separating key administration (policy, rotation, deletion - no decrypt) from key use (decrypt, no administration) is a policy you write yourself. Worth writing: no single identity can then both reconfigure the crypto and read the data - exactly what 164.312(a)(1) wants from an access control story.
One default that genuinely protects you: key deletion requires a 7-30 day waiting period. Deleting a key that ever encrypted PHI destroys the data with it, so in my modules the window is pinned to the 30-day maximum and ScheduleKeyDeletion events page a human.

One design decision I'd defend anywhere: separate keys per purpose - data, logs, backups. Revoking access to PHI data should never break log delivery, and CloudTrail can hold the logs key without gaining any path to the data key.

CloudTrail and CloudWatch - the audit trail you think you have

164.312(b) is a required standard with no addressable escape hatch: "implement... mechanisms that record and examine activity in information systems that contain or use ePHI." What you have by default is 90 days of management-event history in the console. That's it. Specifically:

No durable trail. For records past 90 days you must create a trail delivering to S3. HIPAA's documentation retention requirement (164.316(b)(2)(i)) is six years; most programs apply the same horizon to audit artifacts. 90 days is not in the same universe.
Data events are not logged by default. Management events tell you someone changed a bucket policy; they do not tell you someone downloaded 10,000 patient records via GetObject. Object-level audit is a separate, per-event-billed opt-in - in my modules it's a flag with a cost warning, but a flag someone consciously declines, not a silence nobody noticed.
Log file validation (the SHA-256 digest chain that makes trail tampering detectable - the 164.312(c) integrity story for the audit trail itself) is a setting, not a given.
On the CloudWatch side: log groups never expire by default - retention is a policy you set deliberately - and while log data is SSE-encrypted at rest, customer-managed KMS encryption is optional, attached per log group.
And the quiet half of 164.312(b): "record and examine." Metric filters, alarms, anything that turns the trail into a signal a human reviews - none of it exists until you build it. An unread audit trail satisfies nothing.

ALB - TLS quality depends on which tool you clicked

The load balancer is your transmission security (164.312(e)) chokepoint, with the most path-dependent defaults of the lot:

Create an HTTPS listener in the console today and you get a modern TLS 1.3/1.2 policy. Create it through the CLI or CloudFormation and the default is ELBSecurityPolicy-2016-08, which still accepts TLS 1.0 and 1.1. Your cipher floor depends on which tool provisioned the listener. Pin the policy explicitly and this stops being interesting.
Nothing forces HTTPS to exist at all. A port-80 listener forwarding plaintext is perfectly deployable; the 80-to-443 redirect is a choice.
Access logs are disabled by default - there goes 164.312(b) at the edge until you wire up the log bucket.
Deletion protection: off by default. Deleting the front door of a PHI service should take two steps.
And one my team debated for a week: by default TLS terminates at the ALB, and the hop to your tasks is HTTP inside the private network. A common, defensible posture - but a documented risk decision your analysis has to own, not something the eligible list settled for you.

The scoreboard

Counting controls involves judgment calls, so here's mine - each row counts the specific 164.312-relevant controls discussed above:

Service	Controls mapped	Covered by default	You configure
S3	7	2 (SSE-S3 since 2023, Block Public Access since 2023)	5
RDS Postgres	8	1 (forced TLS, PG15+ only)	7
ECS Fargate	6	2 (no node surface, IAM deny-by-default)	4
KMS	3	1 (deletion waiting period)	2
CloudTrail / CloudWatch	6	0 (90-day history is partial credit at best)	6
ALB	5	0	5
Total	35	6	29

Roughly one control in six arrives configured. The other 29 are your job, on services that are all on the HIPAA-eligible list. And notice which six: the freebies cluster in encryption at rest, where AWS has spent a decade raising defaults. Audit controls and transmission security - both required standards - score near zero.

The floor is genuinely rising: S3 in 2023, PostgreSQL 15's force_ssl, console TLS policies. But that's also the trap - your actual defaults depend on the year the feature shipped, the engine version, and whether the resource was born in the console or in CloudFormation. A compliance posture made of remembered defaults is a compliance posture made of trivia.

What to do about it: treat the list as a floor

The eligible list answers one question: may PHI touch this service under our BAA? Everything after that yes is configuration, and configuration you rely on should be code:

Codify the invariants, and don't make them variables. The rule that survived every review of our modules: if the Security Rule requires it, there is no toggle. Storage encryption, TLS enforcement, log validation, private placement - hardcoded. Variables exist only for things a compliant deployment may legitimately vary, like retention above the floor. A toggle someone can forget is a finding someone will write.
Let policy-as-code catch the drift. checkov and tflint in CI flag most of the gaps in this article - unencrypted RDS, missing bucket policies, permissive TLS policies - before they exist. Native terraform test can pin the invariants so a refactor can't quietly reintroduce a toggle. Where you do deviate, suppress with a written justification inline; that comment is your addressable-specification documentation.
Map controls to citations, in the repo. Every module of ours carries a table: citation, what the code enforces, and - the column that earns its keep - what remains yours. No amount of Terraform covers the administrative safeguards: risk analysis, a log review procedure someone actually follows, restore testing, access reviews, BAAs with every other PHI-touching vendor. The code can only make the technical floor solid enough for your people to stand on.

"HIPAA-eligible" was never a lie. It's a contract term that got promoted, somewhere between the sales deck and the standup, into a security property. The list tells you where PHI may go. The Security Rule tells you what must be true when it gets there. The distance between the two is not covered by anyone's BAA - it's covered by you, ideally in version control.

Disclaimer: I'm an infrastructure engineer, not a lawyer; this is engineering analysis of published regulations and AWS documentation, not legal advice. Defaults cited were verified in August 2026 and do change (sometimes for the better). Run the details past your compliance officer, who will find at least one thing here that your specific situation makes wrong.

What's the widest eligible-vs-compliant gap you've hit in the wild? I collect these.

"Oops, I Forgot to Tell You That's Dangerous": Claude Code Watched Me Wipe Production Redis - Then Helped Carve It Back Off the Disk"

vadim albarov — Wed, 12 Aug 2026 03:47:47 +0000

TL;DR: I enabled Redis AOF persistence by editing redis.conf and restarting the service. On the test environment it worked fine. On production, all three nodes came back with zero keys, and the empty dataset overwrote dump.rdb. The missed step: you must enable AOF on the running instance with CONFIG SET appendonly yes before putting it in the config file. We got the data back by carving the unlinked RDB file out of raw disk blocks with dd. Oh, and the AI assistant that helped me plan the change knew about this trap the whole time - it just didn't mention it until after the wipe. Here's the full story.

Some context first

The stage for this story is a legacy healthcare project that's been running in production for years. Redis sits at the center of it, wearing two hats: it's the cache in front of SQL Server, and it's also the feature flag storage - the backend seeds flags into Redis at startup, and both backend and client read them from there at runtime. Over the years the flag storage format evolved (more on that in a minute), the cache quietly accumulated datasets that exist only in Redis, and the Redis setup itself - a bare tarball install with an untouched config - predates everyone's memory of who set it up. In other words: exactly the kind of system where nobody looks at the persistence settings until something forces them to.

Something did.

It started with a feature flag that "flipped itself"

After we published a new release, one of our feature flags apparently turned itself on. Nobody had touched it. The flag had existed for months with a default of false, and suddenly the client started behaving as if it were true.

I diffed the two release tags first: zero changes to flag definitions or defaults. So I went to where the flags actually live in Redis - and found the flag stored in two places, disagreeing with each other:

redis-cli GET  FeatureFlag:CrossWindowCommands   → "true"    # legacy string key
redis-cli HGET FeatureFlag CrossWindowCommands   → "false"   # current hash field

The current implementation stores flags as fields in a FeatureFlag hash, but there was also a stale legacy string key left over from the old storage format. The false in the hash, it turned out, wasn't the original state at all - a teammate had dug into the same problem before me and manually set it back to false. Which made the mismatch the real hint: the legacy key still held true, untouched, and the values in Redis always win - the startup migration copies legacy keys into the hash with HSETNX, and the config default only applies when the field doesn't exist yet.

So the explanation was less dramatic than a flag flipping itself: the flag had been true in Redis all along. It just didn't matter, because for months no code path ever read it. Then the new client release shipped code that actually did something with the flag, and a value that had been harmlessly wrong the whole time suddenly had teeth. An unused flag quietly became a used one, and it looked like a spontaneous flip.

The real finding: our persistence was a time bomb

While investigating, I checked how the Redis instance itself was configured. It's a redis-stack tarball install running natively on RHEL under a custom systemd unit - no Docker, no operator, one master and two replicas with Sentinel. The persistence settings were all defaults - nothing configured at all:

appendonly no                    # default
save 3600 1 300 100 60 10000     # default snapshot thresholds

At our write volume, that works out to roughly one snapshot an hour. So every restart could silently roll the dataset back by up to an hour - and worse, a rollback could resurrect old values (like stale legacy flag keys) that had been deleted since the last snapshot. That's exactly the kind of environment where flags "change themselves" and nobody can explain why.

So we made a decision: park the whodunit, fix the root cause. Enable AOF (append-only file). With aof-timestamp-enabled yes you even get a timestamped log of every write command - a free audit trail for exactly this class of mystery:

appendonly yes
appendfsync everysec
aof-timestamp-enabled yes
auto-aof-rewrite-min-size 64mb
auto-aof-rewrite-percentage 100

(Honest caveat: it's a what-and-when audit, not a who - the AOF doesn't record which client or user issued a command. Since this instance effectively has a single user anyway, that was good enough for a start.)

Testing: everything green

I tested the change end to end. First in a throwaway Docker container: enabled AOF, wrote some flags, and confirmed the AOF captured every command with #TS: timestamps. I even benchmarked it: 100,000-op redis-benchmark runs against two identical containers came back at ~181k SET/s with AOF versus ~142k without. Yes, the AOF run scored higher - which tells you the difference is pure run-to-run noise. With appendfsync everysec, the write-throughput cost is unmeasurable.

Then on the test environment cluster. I enabled it live on the running instance:

redis-cli CONFIG SET appendonly yes
redis-cli CONFIG SET aof-timestamp-enabled yes

I did that live purely to check the concept: right after flipping it I changed a couple of values and watched them show up in the AOF file. A quick sanity check, nothing more - or so I thought. I had no idea this throwaway "test" step was the one doing the heavy lifting.

I then added the same settings to redis.conf so they'd survive a restart, rebooted the nodes one by one, and verified. Everything held: aof_enabled:1, data intact, dummy writes visible in appendonlydir/*.incr.aof, timestamps and all.

The test environment was perfect.

Production: 0 keys

On production I did what felt like the same change: edited redis.conf on all three nodes, added appendonly yes, stopped the sentinels (deliberately, so no surprise failovers mid-maintenance), and rebooted the nodes one by one.

Why full reboots instead of just restarting the Redis service? Two reasons that felt responsible at the time. All three servers had quietly accumulated 500+ pending OS updates, including some severe security patches - so since I was already in a maintenance window, it seemed silly not to apply them and reboot in one go. And a real restart was part of the plan anyway: the whole point of putting appendonly yes into the config file was to prove the setting survives a node going down, so I wanted to see it hold through a genuine reboot.

Then:

redis-cli DBSIZE
(integer) 0

Empty. Master and both replicas. And dump.rdb on disk? Also empty - about 100 bytes of RDB header and nothing else. Roughly 290,000 production keys, gone: cache, feature flags, and one Redis-only ID-mapping dataset that has no SQL fallback at all.

The missed step

Here's the trap, and if you take one thing from this article, take this:

When Redis 7 starts with appendonly yes, it loads the dataset from the AOF - and ignores dump.rdb.

On the test environment I had run CONFIG SET appendonly yes on the live instance first. That triggers an AOF rewrite that builds the AOF base file from the data currently in memory. By the time the test environment restarted, a fully populated AOF existed on disk, and Redis restored from it.

On production I skipped straight to the config file. So the first startup with appendonly yes found no AOF containing the dataset, initialized an empty one, and came up blank. From there the damage compounds automatically:

Redis starts empty and creates an empty AOF base.
The next background save writes the empty dataset over dump.rdb - and Redis replaces the file via write-temp-then-rename(), so the original file's inode is unlinked.
Replicas reconnect and do a full resync from the empty master, wiping themselves too.

Three nodes of redundancy, all faithfully replicating nothing. (Replication protects you from machine failure. It does not protect you from yourself - it distributes your mistake.)

Full disclosure: I wasn't doing this alone. I had an AI assistant (Claude) in the loop for the whole change - it designed the test rehearsal, wrote the config, and later drove the recovery. It knew about this trap. It just never said it out loud before the prod run. Here's the moment it understood what had happened, opening with its own mea culpa:

This is the classic - and brutal - AOF-enablement trap, and I should have flagged it explicitly for prod: when Redis starts with appendonly yes in the conf but no AOF files exist yet, it loads from the (nonexistent) AOF and ignores dump.rdb entirely - it starts empty and creates an empty AOF. On the test environment this didn't bite because you ran CONFIG SET appendonly yes before rebooting - that built the AOF from the live dataset, so the reboot loaded it. On prod the conf was edited and rebooted directly - first boot came up empty, the next background save overwrote dump.rdb with the empty dataset, and each replica that reconnected did a full resync from the empty master and wiped itself too. That's how all three ended up at 0 keys.

A correct diagnosis in one message - six and a half hours too late to be a warning. (To be fair: I never asked "what could go wrong with this rollout?" either. Neither of us rehearsed the failure mode; we only rehearsed success.)

One honest footnote from the later forensics: block-level evidence showed that at one point during the rollout an AOF base with the full dataset existed on disk for a while, and the actual wipe most likely happened on a subsequent restart in the sequence. The exact fatal moment is unrecoverable; the end state was unambiguous - empty AOF, empty RDB, empty replicas.

And of course: no backup copy of dump.rdb taken before the change, and no off-box backups. (I know. I know.)

The recovery: your data is probably still on the disk

Here's the insight that saved us: because Redis replaces dump.rdb via rename(), the old file wasn't overwritten in place. Its inode was unlinked, but the data blocks were still sitting in the free space of the filesystem, waiting to be reclaimed.

So the plan became: stop all writes, and go dig through the raw block device.

Step 1 - freeze everything. Stop Redis, don't reboot, minimize writes to the filesystem. Every write is a chance for the filesystem to reclaim the very blocks you need.

Step 2 - try an LVM snapshot. The volume group had zero free extents, so no snapshot headroom. Plan B.

Step 3 - scan the block device for RDB signatures. Every RDB file starts with the magic bytes REDIS00. The obvious approach dies immediately on a 2 GB RAM box:

$ sudo grep -abo 'REDIS00' /dev/mapper/rhel-root
grep: memory exhausted

(Block devices have no newlines; grep tries to buffer one infinite "line".) So: a tiny Python scanner that reads the device in 32 MB chunks with an overlap of the pattern length, and prints the absolute byte offset of every match:

DEV   = "/dev/mapper/rhel-root"
PAT   = b"REDIS00"
CHUNK = 32 * 1024 * 1024

import sys
with open(DEV, "rb", buffering=0) as f:
    prev, base = b"", 0
    while True:
        buf = f.read(CHUNK)
        if not buf:
            break
        data  = prev + buf
        start = base - len(prev)
        i = data.find(PAT)
        while i != -1:
            print(start + i, flush=True)
            i = data.find(PAT, i + 1)
        prev  = data[-(len(PAT) - 1):]   # overlap so boundary-spanning hits aren't missed
        base += len(buf)
        if base % (1024**3) < CHUNK:
            print(f"... scanned {base // 1024**3} GB", file=sys.stderr)

Run it with sudo and nohup, go make coffee.

Step 4 - triage the hits. The scan found 10 candidates. Each got a 128-byte peek, read-only:

sudo dd if=/dev/mapper/rhel-root iflag=skip_bytes skip=$OFFSET bs=128 count=1 | od -c

Two hits were, hilariously, the scanner's own output files (the data directory shared the LV we were scanning - which is also why carve output had to be shipped off-box). Four were post-incident debris: the empty AOF bases and near-empty dumps left behind by the wipe. That left four candidates of 35-38 MB with real data, distinguishable by the timestamps embedded in their headers: the last pre-incident dump.rdb, and AOF base files from during the incident window.

Step 5 - carve the candidates to another machine. Never write recovery output to the disk you're recovering from:

sudo dd if=/dev/mapper/rhel-root iflag=skip_bytes skip=59609546752 bs=1M count=64 \
  | ssh user@rescue-box 'cat > /home/redis/cand_D.rdb'

Step 6 - validate. redis-check-rdb on the first candidate reported a CRC error - a block near the tail of the file had already been partially reclaimed - but it still parsed all 290,826 keys. Usable in an emergency, so we set it aside and kept going. The next candidate, the freshest one, came back clean:

$ redis-check-rdb cand_D.rdb
[offset 39730397] Checksum OK
[offset 39730397] \o/ RDB looks OK! \o/
[info] 290834 keys read

Step 7 - verify the contents in a sandbox. Never point production at an unverified file. A throwaway instance on the rescue box, isolated port, no config inheritance:

redis-server --port 6391 --dir /home/redis/rescue-test --dbfilename dump.rdb \
             --appendonly no --daemonize yes
redis-cli -p 6391 DBSIZE
redis-cli -p 6391 HGETALL FeatureFlag
redis-cli -p 6391 shutdown nosave

Real keys, real flag values. We had our database back - carved out of unallocated disk blocks.

Putting it back - carefully this time

Restoring into a Sentinel topology has its own trap: if a sentinel promotes a replica that still holds the empty dataset while you're restoring the master, replication will happily sync the emptiness right back over your restored data. So the order was strict:

Confirm sentinels are stopped on all nodes, then stop replicas, then the master.
On the master: move the poisoned artifacts aside - never delete evidence mid-incident:

   mv appendonlydir appendonlydir.bad
   mv dump.rdb dump.rdb.empty
   cp cand_D.rdb dump.rdb

Set appendonly no in redis.conf - the whole point is to force this boot to load dump.rdb.
Start the master and hold your breath:

   redis-cli DBSIZE
   (integer) 290834

Start the replicas (they full-resync from the restored master - you do not restore the RDB onto replicas), then the sentinels, last.
And only now, enable AOF the right way, on the live instance:

   redis-cli CONFIG SET appendonly yes
   # wait for: INFO persistence → aof_rewrite_in_progress:0
   # verify:   appendonlydir/ base file is megabytes, not ~100 bytes

and then persist appendonly yes into redis.conf. Plus, finally, an immediate scp of the recovered dump to another machine.

290,834 keys - eight more than the older candidate, because the winning file was written a couple of minutes later in the timeline. Full recovery.

One detail I still enjoy: the winning file wasn't the old dump.rdb at all. It was an AOF base file written during the incident - the populated base from that brief window when everything still existed. In Redis 7 the AOF base is itself RDB-format, which is why we could rename it to dump.rdb and boot straight from it.

Lessons

Enable AOF on the running instance first. CONFIG SET appendonly yes, wait for the rewrite to finish, verify the AOF base has real size - then edit the config file. Config-file-then-restart is the data-loss path, because Redis with AOF enabled ignores your RDB at boot.
"It worked in test" only counts if test rehearsed the same sequence. My test-environment run succeeded because I happened to run CONFIG SET first there. Same change, different order of operations, opposite outcome. Rehearse the runbook, not the end state.
Copy the data files before touching persistence settings. A 30-second cp dump.rdb dump.rdb.$(date +%F) (and ideally an scp off-box) would have turned a five-hour incident into a five-minute one.
Verify between steps of a rolling change. I rebooted three nodes back to back and only checked DBSIZE at the end. One check after the first node would have contained the blast radius.
Replication is not backup. The replicas didn't save the data - they synchronized its destruction.
If you do lose a file: stop writes immediately. Deleted ≠ gone. rename()-replaced files leave their blocks in free space, and dd + a signature scan can get them back - but only until something reuses those blocks.
Loose RDB-only persistence (save 3600 1, appendonly no) is its own slow-motion incident. Ours had been quietly able to roll back up to an hour of writes on every restart - that's what made a feature flag look haunted in the first place, which is the only reason we went looking.

The irony of the whole story: the change that destroyed the database was the one meant to make it durable. The fix was correct; the order was fatal. In operations, sequence is part of the change.

Have you ever had a "the fix caused the outage" incident? I'd love to hear about it in the comments.

A Design Flaw in Claude Code's Documentation Skill: One Question, 265k–355k Tokens

vadim albarov — Sun, 09 Aug 2026 16:34:33 +0000

I typed a one-line question into Claude Code - does fable use api billing? - and then ran /context out of habit.

30% of a 1,000,000-token context window was gone. One question, one answer, 295k tokens used.

Then I ran the same prompt on a second laptop: 40%. Same question, same answer, ~100k tokens more.

Same CLI version on both machines. I'll call them laptop 1 (30%) and laptop 2 (40%). This is the story of finding those 100k tokens. Spoiler: every theory I had was wrong, and the root cause turned out to be a design flaw in a single bundled skill - one you can partially work around by adding files to a folder.

The baseline

Here's what /context showed on laptop 1 after that single exchange:

Category	Tokens
System prompt	5.3k
System tools	23.7k
MCP tools (59 tools, deferred)	0
Memory files	318
Skills (16 skill descriptions)	2.2k
Messages	264.6k
Free space	703.9k

Two things jumped out.

First, the fixed overhead everyone worries about - MCP servers, skills, memory - is nearly free. 59 MCP tools sat at 0 tokens because Claude Code defers their schemas until they're actually used. All 16 skill descriptions together cost 2.2k tokens.

Second, the Messages category held 264.6k tokens after a one-line question. The conversation itself was maybe 2k tokens. The rest arrived because my question mentioned a Claude model name, which triggered the built-in claude-api skill - and a skill trigger doesn't just load instructions. This one injected its entire documentation payload into the conversation as a single message.

Worth pausing on that: my question was a billing lookup that a single Google search answers in five seconds. And before you conclude "well, agent sessions are just expensive" - they aren't. As a control, I asked two other lookup questions in fresh sessions: "what's the latest Node LTS version?" and "MIT vs Apache 2.0?". Both together cost 7.5k tokens. Ordinary questions are cheap.

The 265k burn has one specific trigger: mentioning a Claude model name. That summons the built-in claude-api skill, which has no notion of question weight - a casual pricing question gets the exact same multi-hundred-KB documentation payload as "implement a streaming tool-use loop." I paid a quarter of a million tokens for what one web search would have told me.

On laptop 2, that same category showed 354k tokens. The 90k-token mystery lived entirely inside one message.

Wrong theory #1: local CLAUDE.md files

My first guess: laptop 2 has more project instruction files - CLAUDE.md, memory, rules - quietly injected into context.

Dead on arrival. Memory files accounted for 318 tokens on laptop 1, and the gap was ~90k tokens ≈ 360 KB of text. A CLAUDE.md would have to be a small book. More importantly, the gap persisted when I ran the prompt from an empty folder on laptop 2 - no project files at all, still ~355k.

Wrong theory #2: skill version

Bundled skills live in a content-addressed cache:

%LOCALAPPDATA%\Temp\claude\bundled-skills\<cli-version>\<hash>\claude-api\

Both machines ran CLI 2.1.226 with skill version 2.1.226. Same version... but different hashes. Promising! Except when I compared the actual files, every doc folder was byte-identical - 847 KB of assets on both machines. The hash difference was real but, as it turned out later, a symptom rather than a cause.

Wrong theory #3: the tokenizer

This one almost fooled me, because the arithmetic was beautiful. I had tried Sonnet 5 at some point, and it ships a new tokenizer that produces roughly 30% more tokens for the same text.

264.6k × 1.3 ≈ 344k. Almost exactly laptop 2's number. Same bytes, different ruler!

Then I ran the control: same model, same effort, both machines. The gap survived. Tokenizer eliminated. (Keep this failure mode in mind though - token-budget intuitions genuinely don't transfer across model families.)

Transcript forensics

Same CLI, same skill assets, same model, same prompt - and different token counts. At this point the only honest move was to stop theorizing and diff the actual bytes.

Claude Code writes every session to a JSONL transcript. Finding the heavy message takes one loop:

$proj = Get-ChildItem "$env:USERPROFILE\.claude\projects" -Directory |
  Sort-Object LastWriteTime -Descending | Select-Object -First 1
$t = Get-ChildItem $proj.FullName -Filter *.jsonl |
  Sort-Object LastWriteTime -Descending | Select-Object -First 1
$i = 0
Get-Content $t.FullName | ForEach-Object { $i++
  if ($_.Length -gt 50KB) { "Line $i : $([math]::Round($_.Length/1KB)) KB" } }

Laptop 1 transcript: one message of 719 KB. Laptop 2 transcript: one message of 957 KB. There's the gap - 238 KB of text, ~90k tokens at ~2.6 characters per token.

The payload is the skill's documentation, embedded as <doc path="..."> blocks. Extracting the doc lists:

[regex]::Matches($line, '<doc path=\\"([^\\"]+)\\"') |
  ForEach-Object { $_.Groups[1].Value }

Laptop 1: 32 docs - shared API docs + the python/ folder.
Laptop 2: 65 docs - shared API docs + all eight language folders: Python, TypeScript, Go, Java, C#, PHP, Ruby, and cURL.

All 32 docs the payloads had in common were byte-identical. The laptop 2 payload simply contained 33 extra language docs totaling 237 KB.

The reveal

Diffing the instruction text at the top of the two payloads (60 KB each, otherwise identical) surfaced exactly one difference:

Laptop 1 payload:

→ Refer to python/claude-api/README.md

Laptop 2 payload:

→ Refer to unknown/claude-api/README.md
No project language was auto-detected. Ask the user which language they are using, then refer to the matching docs below.

That's the whole mechanism. When the claude-api skill triggers, Claude Code detects your project's language from the working directory and injects only that language's documentation. When it detects nothing - an empty folder, a docs-only repo - the fallback is to inject documentation for every supported language, because the model might need any of them.

And why did laptop 1 detect Python? The folder I was in had a subdirectory containing a Python project with a .venv - thousands of .py files. A stray virtualenv saved me 90k tokens.

The different cache hashes made sense now too: the skill bundle appears to be cached per rendered variant - the assets are identical, but the instruction text differs by detection outcome, so each outcome gets its own hash directory.

The irony, and the side effect worth knowing

Here's my favorite part. When I got serious about controlling variables, I ran the "clean" experiment: empty folder, fresh session, same prompt. That methodologically pure setup is precisely what maximizes the payload. The controlled experiment created the condition it was measuring.

The flip side is a genuinely useful, if odd, side effect: having language context in your working folder reduces token usage. Any file that lets Claude Code detect a language - a .py file, a package.json, a pyproject.toml, even a leftover .venv - pins the skill payload to one language's docs and cuts ~90k tokens (about 25% of the payload) off every claude-api skill trigger. The final scoreboard, reproduced on both machines:

Working directory	Docs injected	Payload	Messages after one question
Any folder with language markers	32 (shared + one language)	719 KB	~265k tokens
Empty / language-less folder	65 (shared + all 8 languages)	957 KB	~355-360k tokens

So if you're about to ask Claude Code API questions from some scratch directory: don't. Run it from a real project - or drop a single pyproject.toml (or the equivalent for your language) into the scratch folder first. It reads as a joke, but it's a measurable 90k-token difference per session, and on smaller context windows it's not funny at all: the all-languages payload alone wouldn't fit in a 200k-token context window. This entire question is only answerable on a 1M-window model.

Why this is a flaw, not a feature

Skills in Claude Code are designed around progressive disclosure: a one-line description sits in context (all 16 bundled skills together cost 2.2k tokens), and the full instructions load only when triggered. The claude-api skill follows that pattern for its trigger - and then abandons it entirely for its content: instead of letting the model read the docs it needs on demand, it eagerly injects the whole documentation set as a single message. It's the only bundled skill big enough to need a disk cache at all (847 KB; the other 15 are trivially small).

The no-language fallback makes it worse in a way that's almost comic. The injected instruction text literally says:

No project language was auto-detected. Ask the user which language they are using, then refer to the matching docs below.

...but by the time the model can ask, all eight languages' documentation has already been paid for. The question comes after the purchase. Any of the obvious designs - ask first and inject one language; inject the shared docs and let the model read language files on demand; scale the payload to the question - would cap the cost at the detected-language level or below.

To be fair about scope: this is one skill, in one CLI version (2.1.226), and the payload is genuinely useful when you're writing code against the Claude API in a detected-language project. The flaw is the eager all-languages fallback and the trigger's insensitivity to question weight - both fixable upstream without losing what the skill is for.

Takeaways

Skill payloads dominate context cost. Everything people usually blame - MCP servers, memory files, system prompts - added up to ~31k tokens for me. One skill trigger added 265-360k. Audit accordingly.
The flaw is specific, not general agent overhead. Control questions in fresh sessions ("latest Node LTS?", "MIT vs Apache 2.0?") cost 7.5k tokens combined. The expensive trigger is mentioning a Claude model or API name. For quick Claude pricing/docs lookups, use a web search - inside Claude Code, the skill will fire.
Deferred loading works. 59 MCP tools at 0 tokens until used. If your setup loads MCP schemas eagerly, that's worth fixing, but it wasn't my problem.
/context tells you the category; the transcript tells you the culprit. The JSONL line-length trick above takes 30 seconds and points at the exact message.
Same CLI version ≠ same context cost. The cost depends on runtime conditions - in this case, what's sitting in your working directory.
Language markers in the folder are a token optimization. Unintuitive, but reproducible: give the language detector something to find, and the skill injects one language's docs instead of eight.
Token intuitions don't transfer across model families. My tokenizer theory was wrong this time, but the ~30% Sonnet 5 difference is real - re-baseline when you switch models.

Everything above was measured on Claude Code 2.1.226 (bundled claude-api skill 2.1.226) with Fable 5 and Sonnet 5, on two Windows machines. The behavior may well change in future releases - arguably the fallback should ask before injecting 237 KB of polyglot documentation - but the audit method will keep working regardless.

Reproduce it yourself: ask Claude Code any Claude-API question from an empty folder, run /context, then do the same from inside a Python or TypeScript repo and compare the Messages category.

UPD: The 2.1.234 update fixed the issue (at least for me). Now the claude‑api skill is no longer bloated and does not exhaust your limits anymore

I fine-tuned a 1.7B model on data that doesn't exist - a 17-minute recipe

vadim albarov — Thu, 06 Aug 2026 20:34:02 +0000

The full methodology behind localscrub's stage-2 specialist: synthetic
training data with exact labels, training on the serving distribution, and
why the fine-tune's first product is not accuracy - it's parseability.

In the last article I benchmarked localscrub, a local-first PHI de-identification cascade, and buried a teaser near the end:
a 1.7-billion-parameter model, LoRA-tuned in 17 minutes on a consumer
laptop, went from producing zero parseable replies to 0.92 redaction
recall on authentic clinical prose. This article is the recipe - every
command, every hyperparameter, and the three methodology decisions that I
think matter more than any of the numbers.

The de-identification task is the case study, but the recipe generalizes to
any structured-extraction job you want a small local model to do: if you
can generate your training data and you train on the exact prompt your
inference code sends, a specialist you can retrain from scratch in under
half an hour is within reach on one 16 GB GPU.

One constraint shaped everything: no real patient data anywhere in the
loop. Not in the training set, not in the eval set, not in a single
prompt sent to any model at any step. By the end you'll see that this
constraint wasn't a handicap - the synthetic pipeline it forced turned out
to be the most valuable asset in the project.

Why fine-tune at all, when a pretrained classifier already wins?

Fair question, because the last article's headline was an off-the-shelf
125M token classifier hitting 0.999 redaction recall. If that's available,
why train anything?

Because a token classifier cannot take stage 2's seat. localscrub's second
stage has a conversational contract: it receives the note plus escalation
hints - ambiguous spans that the rules engine flagged but couldn't resolve

and returns verbatim snippets with types, as JSON, adjudicating the escalations along the way. A token classifier tags a fixed label set; ask it about an identifier type it wasn't trained on and it has no opinion, and it can adjudicate "is this flagged span actually PHI?" only for the types it already knows. Only an instruction-following model can hold up the whole contract.

A 14B generalist (qwen3:14b via Ollama) holds it up at 5-8 seconds per
note. The question worth 17 minutes of GPU time: can a model an eighth that
size be taught to?

The training data costs nothing and leaks nothing

The usual fine-tuning bottleneck is labeled data. Clinical de-id makes it
worse: the gold-standard corpus (i2b2/n2c2 2014) sits behind a data use
agreement, and real notes are radioactive - one mishandled training example
and your privacy tool has a privacy incident.

localscrub sidesteps both because its synthetic corpus generator already
existed for evaluation. localscrub synth renders clinical notes from
templates with fabricated identifiers planted at recorded character
offsets - every phone from the reserved 555-01XX block, every domain from
RFC 2606, every credit card Luhn-valid on a test prefix - deterministic
from a seed. The eval harness scores against those exact gold spans.

The insight that unlocks fine-tuning: a corpus with perfect gold spans is
also a perfect SFT dataset. One command emits training pairs instead of
eval records:

localscrub sft -n 2000 --seed 7 -o data/corpora/sft-2000.jsonl

Two thousand examples, generated in seconds, with two properties paid
datasets can't match:

Zero annotation noise. The labels aren't human annotations of generated text - the generator planted the identifiers, so the gold is exact by construction. No inter-annotator disagreement, no boundary fuzziness, no label budget.
Zero exposure. There is no real PHI to leak because there is no real PHI. The training set is as regenerable and disposable as the eval set - corpora are gitignored, adapters are gitignored, and everything rebuilds from a seed.

Seed discipline matters here: training uses seed 7, evaluation uses seed

Disjoint seeds mean no example overlap - but I'll flag the honest limitation now rather than in the fine print: the template corpus has only seven note skeletons, so train and eval share structure even though they share no content. Every template-corpus "after" number below is optimistic by construction. The MTSamples benchmark - authentic medical transcription prose that never entered the fine-tune in any form - is the honest generalization test, and it's the one I'll ask you to judge the recipe by.

The one rule: train on the serving distribution

Here is the decision I'd defend hardest, and the one I see skipped most
often in fine-tuning write-ups: each training prompt is built by the
actual inference code.

localscrub sft doesn't format notes into some training-time template that
resembles what inference does. For every example, it runs stage 1 on the
note - the real rules engine, producing real escalation hints - and then
calls localscrub.stage2.extraction_prompt, the same function the serving
path calls, to build the prompt. The completion is the extractor's exact
reply format, and it parses with the same salvage-tolerant parser
inference uses.

The tuned model never sees a prompt shape at inference time that it didn't
see thousands of times in training, escalation hints included. Most
"fine-tuned model underperforms in production" stories I've read trace back
to exactly this seam: the training data was formatted by a script that
approximated the serving prompt, and the approximation drifted.
Eliminating the seam costs nothing - reuse the inference code - and it's
free insurance.

Two corollaries of the same principle:

Raw prompt→completion, no chat template. The serving path sends plain text, so training does too. This isn't just simplicity - format discipline is part of what the before/after numbers measure, and wrapping everything in a chat template would train a different behavior than the one being served and scored.
Completion-only loss. The model is graded on its reply, not on its ability to predict the note back. Standard practice, but it composes with the above: the loss covers exactly the tokens the serving path will consume.

The recipe

Hardware: one RTX 5080 Laptop GPU, 16 GB. Model: Qwen3-1.7B-Base
(Apache-2.0). Total wall-clock for training: 17 minutes.

# 1. training data: 2000 examples, seed disjoint from eval seeds
uv run localscrub sft -n 2000 --seed 7 -o data/corpora/sft-2000.jsonl

# 2. train (bf16 LoRA, ~17-30 min on 16 GB)
uv run --group train python scripts/finetune_lora.py \
    --data data/corpora/sft-2000.jsonl \
    --out models/stage2-qwen3-1.7b-lora

# 3. the authentic-prose benchmark (public MTSamples CSV, synthetic injections)
uv run localscrub mtsamples --fetch -n 100 --seed 42 \
    -o data/corpora/mtsamples-100.jsonl

# 4. before/after as stage 2, standard eval harness
uv run --group train python scripts/eval_lora.py \
    --template-notes 20 --mts-notes 15 \
    --mtsamples data/corpora/mtsamples-100.jsonl          # before (base) - fewer
                                                          # notes: ~30 s each
uv run --group train python scripts/eval_lora.py \
    --adapter models/stage2-qwen3-1.7b-lora \
    --mtsamples data/corpora/mtsamples-100.jsonl          # after (50 + 30 notes)

Hyperparameters, all overridable via flags: LoRA r=16, α=32, dropout 0.05,
all-linear targets; learning rate 1e-4 with cosine schedule and 3% warmup;
2 epochs; effective batch 16 (batch 2 × gradient accumulation 8); max
length 2048; bf16; completion-only loss; seed 7.

Nothing exotic - r=16 all-linear LoRA at lr 1e-4 is close to community
defaults for this model size, and that's deliberate. The recipe's leverage
is in the data (exact labels, serving distribution), not in hyperparameter
heroics. If your first instinct on a weak result is to sweep learning
rates, look at your data pipeline first.

Deployment note: to serve the adapter through Ollama, convert it with
llama.cpp's convert_lora_to_gguf.py and reference it from a Modelfile
ADAPTER line. The evaluation script talks to the model via HF
transformers directly, which skips that step for the before/after.

Reading the before/after: parseability first, accuracy second

The numbers, scored by the same harness as every other configuration in
the project. Redaction recall is the safety metric - the fraction of
gold spans whose every character is removed from the output; a half-redacted
address counts as a miss, because a half-redacted address is still a leak.

config	corpus	unparseable replies	redaction recall
base 1.7B (before)	template, 20 notes	20/20	0.66 (= stage 1 alone)
base 1.7B (before)	MTSamples, 15 notes	15/15	0.61 (= stage 1 alone)
+ LoRA (after)	template, 50 notes	0/50	1.000*
+ LoRA (after)	MTSamples, 30 notes	0/30	0.924

* optimistic by construction - train and eval share note skeletons, as
conceded above. The MTSamples row is the one to trust.

Look at the "before" rows first, because they're the finding I didn't
expect. The base model didn't merely underperform - it produced unparseable
output on every single note, 35 of 35. Rambling, format drift, never
once a reply the parser could consume. Since localscrub's merge is
fail-closed, a useless stage 2 degrades the cascade to exactly the stage-1
baseline - the 0.66 and 0.61 aren't the small model helping a little,
they're the cascade running as if stage 2 weren't there.

After 17 minutes of LoRA: zero unparseable replies across 80 notes.

The fine-tune's first product is not accuracy - it's parseability.
That transition alone converts a dead stage 2 into a working one, before
any detection improvements register. Accuracy followed: 0.61 → 0.92
redaction recall on prose the fine-tune never saw. If you're evaluating
whether a small model can replace a big one in a structured pipeline, check
format compliance before you check task skill - in my case it was the
entire difference between "useless" and "viable," and it's the cheapest
thing fine-tuning buys.

What the remaining gap is made of - and how I know

On MTSamples, the 17-minute specialist scores 0.924. The 14B generalist
scores 0.976 in the same seat on the same notes (0.967 over the full
100-note benchmark); the pretrained NER extra reaches 0.999. So
the specialist does not yet win on unseen prose - and it's worth being
precise about why, because the two candidate explanations point at
opposite fixes.

If the gap were capacity - 1.7B just isn't enough model - the fix is a
bigger model, and the 17-minute story dies. If the gap is training-data
diversity - 2,000 examples from 7 note skeletons teach the task format
perfectly but narrative prose only partly - the fix is better data, and the
whole pipeline stays consumer-hardware-sized.

The evidence leans diversity. The tuned model scores a perfect 1.000
in-distribution while dropping to 0.92 out-of-distribution - the signature
of a distribution gap, not of a model too small for the task it was
trained on. It isn't proof: a 1.7B model can ace seven memorized skeletons
and still lack the parameters for open narrative prose. But diversity is
the cheaper hypothesis, and - unlike capacity - this pipeline can test it
without buying anything.

Which makes the fix mechanical rather than speculative, and the pipeline
already contains both halves:

Mix in MTSamples-injected notes (localscrub mtsamples) from transcripts disjoint with the eval set - authentic narrative rhythm around synthetic gold spans.
Mix in --diversify paraphrases. This is the corpus generator's trick for letting a local LLM rewrite template prose without ever seeing an identifier: every gold span is masked behind a sentinel token ([[E3]]), the model paraphrases around the sentinels, values are re-substituted and offsets recomputed. A rewrite is rejected if any sentinel is dropped or duplicated, or if re-running stage 1 on the rebuilt text finds identifier-shaped strings outside the gold spans - i.e., the model invented PHI. Labels stay exact by construction; only the prose diversifies.

Same generator, same seeds, same 17-minute training loop - just a corpus
with more imagination. That's the deeper point of the whole exercise:
the synthetic corpus is the asset. Data, training, and eval all
regenerate from seeds; the model is almost a by-product. When the training
data is a function rather than a file, "collect more data" becomes a code
change.

Operational potholes (the section I wish every recipe had)

An untuned model with no token cap looks exactly like a frozen process.
The first "before" eval appeared to hang. It hadn't - the base model was
rambling toward a 4,096-token ceiling on every note, minutes per note, with
nothing printed. Two fixes, both boring, both essential: a hard token cap
on generation, and per-note progress output. Instrument your "before" run
as carefully as your "after" - it's the row of the table nobody expects to
debug, and the model most likely to misbehave is sitting in it.

Eval latency is not serving latency. The ~17-20 s/note these runs
clock is naive transformers.generate - no batching, no serving
engine. Serve the adapter through Ollama or vLLM before quoting latency
anywhere it matters.

Blackwell GPUs need cu13x torch builds, and uv wouldn't swap the
default CPU torch for the CUDA build without --reinstall and an explicit
pin: uv pip install --reinstall "torch==2.13.0+cu130" --index-url https://download.pytorch.org/whl/cu130. The training dependency group
(peft, trl, datasets, accelerate) is deliberately not part of any install
extra - trainers opt in, users never download it.

Adapters land in a gitignored models/ directory. Weights never enter
the repo - and after the last article's gitignore-trailing-comment incident
silently un-ignored a 17 MB dataset, I now test what gitignore actually
ignores rather than trusting what it appears to say.

Caveats, stated rather than buried

The template-corpus 1.000 is optimistic by construction (shared note skeletons); judge the recipe by the MTSamples 0.924.
MTSamples injections are synthetic identifiers in authentic prose, not naturally occurring PHI; i2b2-style evaluation is pending a data use agreement.
MTSamples is a well-known public corpus, so the transcripts may well sit in the base model's pretraining data - "unseen" means unseen by the fine-tune. The injected identifiers are freshly generated either way.
0.924 does not beat the 14B generalist (0.976 on the same notes) or the pretrained NER extra (0.999). The claim is not "small model wins" - it's that format discipline plus most of the accuracy is purchasable for 17 minutes of consumer GPU time and zero real records, with a documented path for the rest.
No real PHI was used anywhere in this work - training, eval, or any prompt sent to any model.

Check my math

The entire loop - corpus, training pairs, fine-tune, before/after eval -
regenerates from the seeds in the commands above on one 16 GB GPU: seconds
for the data, 17 minutes for the training, and roughly 40 more minutes of
naive generation for the two evals - call it an hour end to end. The code is on GitHub, the package is on PyPI, and the recipe lives in
docs/finetune.md.

If you try the recipe on your own extraction task - de-id or otherwise -
I'd genuinely like to hear where it breaks. The next article sends the
benchmark to the cloud de-id APIs, and that trust-boundary crossing gets a
write-up of its own.

A 125M model beat a 14B LLM at de-identifying medical text 40 faster, on CPU

vadim albarov — Sun, 02 Aug 2026 04:13:27 +0000

Your data never leaves the machine - and you can check my math

Building localscrub, a local-first PHI de-identification cascade, and
benchmarking it honestly against the standard baseline - on one consumer
laptop, with zero real patient data.

De-identifying clinical text today forces a bad trade. Cloud de-id APIs are
accurate, but you send the sensitive data out in order to scrub it - the
text crosses your trust boundary before a single character is redacted.
Local rule-based tools keep the data home, but miss exactly the PHI that
matters most: the context-dependent kind. A regex will catch an SSN every
time; it will never catch "the patient's sister works at the bakery on Elm
Street."

localscrub is my attempt to
refuse the trade. It runs a two-stage cascade entirely on your hardware: a
fast rules-and-NER pass for the well-formatted identifiers - phones, emails,
dates, account numbers - and a local LLM, served by Ollama with no network
egress, for the ambiguous remainder. (How much each stage carries is an
empirical question; the benchmark below answers it rather than assuming.)
It's on PyPI as v0.1 (pip install localscrub), and every number in this
article reproduces from seeds on a single RTX 5080 laptop.

This is the story of building it - and more importantly, of measuring it,
because a privacy tool with unverifiable accuracy claims is just a liability
with a nice README. Along the way: a test set that is a function rather than
a file, a 125-million-parameter model that beat a 14-billion-parameter one,
an eval harness that indicted its own gold standard, and one cursed note
that killed a three-hour benchmark at 99% complete.

The ground truth problem: a test set that is a function, not a file

You cannot measure a de-identifier without ground truth, and I refused to
use real PHI to get it. The gold-standard clinical de-id corpus (i2b2/n2c2
2014) sits behind a data use agreement, and scraping a third-party re-upload
would make a privacy project sloppy about data provenance on day one.

So the corpus is generated. localscrub synth renders synthetic clinical
notes from templates - seven variants across six note types - with
fabricated identifiers planted at recorded character offsets. Every phone
number is from the reserved 555-01XX block, every domain from RFC 2606,
every IP from RFC 5737, every credit card Luhn-valid on a test prefix. The
output is JSONL with exact gold spans, deterministic from a seed:

localscrub synth -n 500 --seed 42 -o eval.jsonl

That determinism buys something subtle: an eval number becomes a property
of the code, not of a dataset file. Corpora are gitignored and
regenerated at will. The test set is a function, not a file.

Template-generated text invites an obvious objection: a detector could
memorize the templates. The answer is --diversify, which lets a local LLM
paraphrase the connective prose without ever seeing an identifier: every
gold span is masked behind a sentinel token ([[E3]]), the model rewrites
around the sentinels, values are re-substituted, offsets recomputed. A
rewrite is rejected if any sentinel is dropped or duplicated - or if
re-running stage 1 on the rebuilt text finds identifier-shaped strings
outside the gold spans, i.e. the model invented PHI. The validation loop
cost about ten lines and closes the biggest ground-truth-corruption risk.
That is how you let an LLM touch your test set without trusting it.

An eval whose first job is to embarrass you precisely

The harness (localscrub eval) reports three numbers per entity type, and
keeping them separate turned out to matter more than any single one:

relaxed (type + character overlap): did you find it?
strict (type + exact boundaries): are your boundaries calibrated?
redaction recall - the fraction of gold spans whose every character is removed from the output, by any means, under any label: is the output actually safe?

Redaction recall is the safety metric, and it is deliberately unforgiving:
a detection that leaves half an address in the text does not count. Partial
redaction of an address is still a leak.

The three-metric split earned its keep on the very first run. Stage 1's URL
recognizer scored relaxed 1.00 and strict 0.00 - it was swallowing
sentence-final periods on every single URL. Overlap-only scoring would never
have surfaced it. The same first run put honest zeros on the board: NAME
0.00, GEO 0.00, because stage 1 has no name recognizer by design. Overall
redaction recall: 0.62. That 0.62 turned "stage 2 is on the roadmap" into a
quantified gap - 38% of gold spans unprotected without it.

The cascade, and the one design decision that carried the project

Stage 2 asks a local model (qwen3:14b via Ollama) to extract
context-dependent PHI. The contract is the highest-leverage decision in the
codebase: the model returns verbatim snippets plus a type - never
character offsets. LLMs cannot count characters, but they copy substrings
reliably. Every returned snippet is located in the source text by string
search, which redacts repeated mentions for free, and a hallucinated span
dies at a str.find instead of corrupting a redaction. Ask the model for
what it can do (copy), verify what it can't (locate).

The merge with stage 1 is additive and fail-closed. Stage-1 detections win
overlaps - rules are better calibrated where rules apply. Ambiguous spans
stage 1 flagged are put to the model for adjudication, but only in one
direction: an escalation the model confirms is resolved; one it stays
silent on remains escalated and gets redacted anyway. A 14B model's "no"
never unredacts anything.

First contact, 50 notes: redaction recall 0.62 → 0.89, NAME relaxed recall
0.00 → 0.98. And one open wound: the model found "Cedar Vale" but clipped
"4050 Mossbank Blvd", fully covering only 12% of address spans. Relaxed F1
made GEO look twice as healthy as it was; redaction recall told the truth.

A 125M model beat the 14B model - 40× faster, on CPU

Before reaching for fine-tuning, I tried the boring thing: an off-the-shelf
de-id-specific token classifier (obi/deid_roberta_i2b2, 125M parameters,
trained on the i2b2 2014 corpus) wrapped as an optional stage-1 recognizer
(pip install 'localscrub[ner]'). Only its name and location labels are
mapped; dates, phones, and emails stay with the regexes, whose boundaries
are already exact.

On the template corpus it was decisive: redaction recall 0.94 at 184 ms
per note on CPU - matching the 14B model at the categories it was trained
for, roughly forty times faster, no GPU.

Let me concede the framing objection before anyone raises it: a specialist
trained on exactly this task beating a prompted generalist is expected, not
shocking. The finding is the size of the trade - equal recall at
one-fortieth the latency, no GPU - and it matters because the default
recipe today is "throw an LLM at it," and for structured text the default
is measurably wrong. Nor was the 14B handicapped: it ran qwen3:14b at
temperature 0 with schema-constrained decoding and the same
escalation-hint prompt that inference uses (extraction_prompt in the repo).

The two models fail differently,
and that mattered later: the LLM copies name boundaries nearly perfectly
but clips addresses; the token classifier covers whole addresses but drags
titles and credentials into name spans. Complementary failure modes,
measurable as such.

Integrating it produced the best debugging afternoon of the project - three
real bugs and a gold-standard flaw, all surfaced by the eval:

Adding NER dropped EMAIL recall from 1.00 to 0.53. A merge bug glued model spans across paragraph breaks (name@example.org.\n\nNext Name became one NAME span).
Still 0.53 after the fix - the upstream HuggingFace pipeline aggregation produced the same cross-paragraph span. No identifier spans a line break; split on newlines first.
EMAIL 0.78 - the model labels darius.ashcombe@example.com as a PATIENT name, because emails literally contain patient names. A person name never contains @; drop such spans at the source.
GEO recall pinned at 0.52 no matter what. Not a model bug: the gold standard was wrong - the corpus planted street and city as two spans where reality has one postal address, so the model's single correct span could only ever match half the gold. The tell was 0.52 recall alongside 0.72 precision. Sometimes the eval indicts the gold, not the system.

Rule of integration, now baked into regression floors: adding a detector
must never make another detector worse.

The prose gets real: an injection benchmark

Templates were too easy, and by this point provably so. The next test
injects synthetic identifiers into ~5,000 authentic public
medical-transcription samples (mtsamples.com - downloaded with a pinned
checksum, never redistributed). Injections are unlabeled narrative
sentences woven between real sentences at deterministic positions:

localscrub mtsamples --fetch -n 100 --seed 42 -o mts.jsonl

One methodological point worth stating plainly: on an injection benchmark,
recall is exact but precision is only a lower bound. The real
transcripts contain their own name-like and date-like strings - "Dr. X"
placeholders, real dates - so a detector flagging them is penalized for
being right. Redaction recall over the injected gold is the number to
trust.

The results - including the negative one

Every configuration, both corpora, all at full size, scored by the
identical harness: 500 template notes carrying 5,021 gold spans, and 100
MTSamples notes carrying 697 injected gold spans - both from seed 42.
Redaction recall - the safety metric - plus precision on the
authentic-prose corpus, where over-flagging shows:

config	template	MTSamples	MTS precision†	latency/note	hardware
Presidio (rules-only baseline)	0.78	0.75	0.49	14–46 ms	CPU
stage 1 (rules)	0.66	0.62	0.94	µs	CPU
stage 1 + NER	0.94	0.999	0.81	184–652 ms	CPU
stage 1 + LLM	0.94	0.967	0.82	7–8 s	GPU
stage 1 + NER + LLM	0.94	0.999	0.76	7–17 s	GPU

† relaxed precision on the injection benchmark - a lower bound for every
system, per the previous section.

Because three nines invite scrutiny, here are the raw counts behind the
headline number. 0.999 is 696 of 697 injected spans fully redacted
(Wilson 95% CI 0.992–0.9997); one more miss would read 0.997, so treat the
third digit as "one miss in this sample," not a stability claim. And the
one miss deserves naming: in 2034 Harrowgate Rd, Lantern Hill, VT 93695,
the NER covered the street line ("2034 Harrowgate Rd") and the
state-plus-ZIP ("VT 93695") but dropped the city - "Lantern Hill" leaked
from between two redactions. The
address-clipping failure mode, surviving at the very tail. (The template
0.94, for comparison, is 4,711 of 5,021 - 310 misses; at that sample size
the second digit is doing honest work.)

Two findings, one of them a negative result I think the field under-reports.

On synthetic templates, the LLM buys nothing. Stage 1 + NER,
stage 1 + LLM, and the full cascade all converge at 0.94 redaction recall -
and at 0.97 relaxed F1 - at latencies spanning 184 milliseconds to 17
seconds. The residual 6% is corpus-bound, not detector-bound. If your text
is structured and identifier-dense, a good token classifier is all the
model you need, and it runs on CPU.

On authentic prose, the LLM earns its keep. Bare rules manage 0.62;
adding the LLM lifts that to 0.967; NER+LLM reaches 0.999. The two
detectors compose exactly as the merge was designed to: the LLM still
clips addresses, NER still covers them. But recall is not free - the
precision column tells the other half. The cascade over-flags on narrative
text (0.81 → 0.76 versus NER alone, both lower bounds), and over-redaction
has a real cost in clinical text: every falsely scrubbed token is signal a
downstream reader loses. That trade is why localscrub's model is
review-and-attest rather than fire-and-forget - and note the baseline pays
the same toll, with Presidio at 0.49 precision on this corpus. Recall is
what the LLM buys; know which side of the trade your application needs.

Versus Presidio, fairly

Microsoft's Presidio (rules + spaCy, the standard open-source baseline)
beats bare stage 1 on redaction recall - 0.78 vs 0.66 - because spaCy
gives it person and place names, which stage 1 intentionally defers. It is
also 13–14× faster than our recommended CPU configuration, and its NAME
boundaries are better than our NER extra's (strict F1 0.87 vs 0.73).
Every scoring ambiguity was resolved in the baseline's favor - its unmapped
types still earn redaction credit.

With the NER extra, localscrub wins where a leak hurts most. Presidio never
fully covered a single gold address on either corpus - spaCy tags "Dayton"
but drops "412 Birch Lane" - and it has no MRN or health-plan recognizer,
fully redacting 2–6% of MRNs and ≤15% of plan IDs. localscrub holds those
at 1.00 in every configuration.

The 17-minute specialist

A fair question at this point: if a pretrained 125M classifier already hits
0.999, why train anything? Because a token classifier cannot take stage 2's
seat. It tags a fixed label set - ask it about an identifier type it wasn't
trained on and it has no opinion - and it cannot adjudicate the ambiguous
spans stage 1 escalates. Stage 2's contract, verbatim snippets plus types
as JSON, is a conversation, and only an instruction-following model can
hold up its end. The 14B holds it up at seven seconds a note. The question
worth 17 minutes of GPU time is whether a small model can be taught to.

localscrub sft emits 2,000
training pairs - the exact inference-time stage-2 prompt, escalation hints
included, paired with gold JSON - and a LoRA recipe (r=16, bf16) tunes
Qwen3-1.7B-Base in 17 minutes on the laptop.

The before/after is stark, but not where I expected. The base 1.7B model
produced unparseable output on 35 of 35 notes; the tuned one failed on
0 of 80. The fine-tune's first product is not accuracy - it's
parseability. Accuracy followed: MTSamples redaction recall 0.61 → 0.92.
(Template recall hit a perfect 1.000, which is optimistic by construction -
train and eval share note skeletons; the docs say so.)

The 0.92-vs-0.999 gap against the big-model cascade is a training-data
diversity gap, not a capacity verdict - 2,000 examples from 7 templates
generalize only partway to real prose, and the recipe documents the fix
(mix in MTSamples-injected and diversified notes). The deeper point: the
synthetic corpus is the asset. Data, training, and eval all regenerate from
seeds; the specialist retrains from scratch in under half an hour on
consumer hardware, with no real PHI anywhere in the loop - including the
prompts.

Operational lessons (the part I wish someone had written for me)

One bad note must not cost you the run. Stage-2 latency is bimodal:
~5 s per note typically, ~50 s when schema-constrained decoding runs away
to the 4,096-token cap - and in each 500-note pass, exactly one template
note (deterministic at temperature 0) stalled the Ollama server for
minutes before returning HTTP 500. The first time, that single note killed
a 2-hour-50-minute benchmark at note ~495 with nothing written, because the
eval had no per-note error handling. The fix - retry the note once, then
score it without stage 2 and report an llm_failures count - turned the
second occurrence into a 4.8-second retry-and-continue. If you benchmark
local LLMs, build this in before your first long run, not after.

Cap your decoders. Uncapped schema-constrained generation once looped
past a ten-minute timeout. A num_predict cap plus a salvage parser (parse
the longest well-formed prefix of a truncated extraction list; every item
is independently verified against the source anyway) converts runaway
decoding from a crash into a bounded cost.

Assorted potholes: a gitignore trailing comment silently unignored a
17 MB dataset and it reached the git index once; Blackwell GPUs need cu13x
torch builds and uv needed --reinstall with an explicit +cu130 pin to
swap them; an untuned base model with no token cap looks exactly like a
frozen process.

Caveats, stated rather than buried

The template-corpus numbers are on easy, identifier-dense synthetic prose; MTSamples-injection is the authentic-prose test, and the numbers held there. But injected identifiers are not naturally occurring PHI: real i2b2-style evaluation remains the gap.
On data leakage: MTSamples is a well-known public corpus, so the transcripts may well sit in a model's pretraining data (obi/deid_roberta_i2b2 was trained on i2b2 2014, not MTSamples; for the LLM's pretraining the honest answer is unknown). The scored identifiers, however, are not MTSamples content: every gold span is synthetic, seeded, and injected - the identifier strings are generated, not drawn from the transcripts, so memorizing MTSamples does not hand a model the answers. What leakage could do is make the surrounding prose feel familiar, flattering the numbers relative to truly unseen clinical text - one more reason the i2b2 evaluation (behind a data use agreement, unlikely to be memorized) is the next benchmark.
Injection-benchmark precision is a lower bound for every system measured.
Stage-2 reproducibility is best-effort: fixed seed, temperature 0, but tied to model build and hardware.
No real PHI was used anywhere, at any step - including in every prompt sent to the local model.
localscrub does not claim HIPAA Safe Harbor certification. It reduces the problem to review-and-attest; residual-risk sign-off stays human. The threat model spells out the trust boundary, including what a hijacked stage-2 model can and cannot do (it can over-redact or falsely confirm; it can never unredact).

What's next

Two measurements are deliberately absent, and they are the next article.
Cloud de-id APIs: the accuracy/latency/cost legs require sending the
benchmark corpora to each provider - synthetic or not, that crossing of the
trust boundary deserves its own explicit decision and write-up, because it
is the exact trade this project exists to interrogate. And i2b2/n2c2
2014: the literature-comparable corpus of naturally occurring PHI, access
request filed and pending. When both land, the comparison table gets its
last rows.

Until then: the code is on GitHub, the package is onPyPI, and every number above regenerates from a seed. Check my math.