I have a two-window habit. My main work happens in a Claude Code session, and I don't like burning its context on side questions - so for quick lookups I alt-tab into Windows Copilot and use it as a search-flavored notepad. Ask, skim, close, back to work. It's been part of my routine for a long time, and honestly, I'd never had a problem with its answers before - side questions came back reasonable, and nothing ever sent me down a wrong path. Low stakes, or so I thought.
This week the side window taught me a HIPAA rule with total confidence. The rule was backwards. Not vague, not incomplete - inverted, in the one direction that would turn a compliance question into a breach.
I build healthcare infrastructure, so I had the reflexes to check. This is the story of what it said, what the regulation actually says, how the wrong answer survived thinking mode, search mode, and a demand for references - and how the citation trail eventually explained where the inversion came from.
The question
I was thinking about deterministic pseudonymization: replacing a patient identifier with hash(key + identifier) so the same patient always maps to the same token and records stay joinable. The obvious follow-up is what HIPAA thinks about the key. Keep it secret? Share it? Does it matter? Full disclosure: the question I actually typed was leading - I'd already run into the claim that sharing the key was the compliant option somewhere, and I asked about it as if it were established.
Copilot's answer, summarized:
- If you keep a secret key, you retain the ability to re-identify. Therefore the data is pseudonymized, still PHI, not de-identified.
- If the key is public, nobody has privileged knowledge. Nobody can reverse a one-way hash. Therefore the data can qualify as de-identified.
- HIPAA, it explained, cares about privileged access, not cryptographic strength. "You're thinking like an engineer. HIPAA is written by lawyers."
It's a seductive answer. It has a philosophy. It flatters you for being confused. It reads like someone explaining a genuinely counterintuitive corner of law.
It is also wrong on both branches, and the second branch is dangerous.
What the regulation actually says
The relevant text is 45 CFR 164.514. Safe Harbor's identifier list ends with a catch-all - "any other unique identifying number, characteristic, or code" - with exactly one exception: a re-identification code that satisfies paragraph (c). Paragraph (c) has two prongs:
- Derivation. The code must not be derived from or related to information about the individual, and must not be otherwise translatable back to the individual.
- Security. The covered entity must not use or disclose the code for other purposes and must not disclose the mechanism for re-identification.
Read those against the two branches:
Secret-key hash. HMAC(secret, MRN) fails Safe Harbor - but not for Copilot's reason. It fails because the token is mathematically derived from the identifier, and prong 1 prohibits derivation outright, regardless of how well you guard the key. Meanwhile, Copilot's actual claim - "if you can re-identify, it's not de-identified" - contradicts the regulation's text. Paragraph (c) exists precisely so a covered entity can keep a secret re-identification mechanism while the dataset remains de-identified. Retained re-identification capability is not the disqualifier; derivation and disclosure are.
Public-key hash. This one fails everything at once. Publishing the key is literally "disclosing the mechanism for re-identification" - prong 2, verbatim. And it makes the data trivially translatable - prong 1 - because health identifiers live in small, enumerable spaces. SSNs are a 10^9 space. Phone numbers, MRNs, emails, name-plus-birthdate: all enumerable. With the key public, anyone hashes every candidate and matches your entire column in seconds on a laptop. One-wayness protects high-entropy inputs; it does nothing for a nine-digit number.
HHS's own de-identification guidance closes the loop from both sides. It says a hash without a secret key counts as an identifying element exactly because recipients can reverse it over the input space. And it says keyed cryptographic hashes are acceptable under the Expert Determination pathway provided the keys are not disclosed, including to the recipients.
So the real rule is the mirror image of what my sidebar told me:
| Construction | HIPAA status |
|---|---|
| Hash with a published key | Fails everything - discloses the mechanism, trivially reversible |
| Hash with a secret key | Fails Safe Harbor (derived code); acceptable under Expert Determination with the key undisclosed |
| Random token + protected mapping table | The pattern 164.514(c) actually blesses - a random value is not "derived from" anyone |
Round two: references make it worse
Maybe I asked badly. I turned on thinking mode, then search mode, and asked it to rethink and support its answer with references.
I got the same inverted conclusion, now decorated with emoji section headers, a verdict table, and a source list - and with something worse. The centerpiece was a quoted sentence attributed to HHS, saying a code derived from PHI is an identifier "unless the re-identification key is not retained." I searched for that sentence. Not on HHS.gov, not anywhere I could find. The model composed a plausible-sounding rule that swaps the regulation's actual verb - disclose - for retain, wrapped it in quotation marks, and attached the genuine HHS URL to it.
That's the failure mode that stuck with me: a real link laundering a fake quote. Every reader's citation heuristic - "it links to hhs.gov, so it's grounded" - defeated by construction.
The funny part is that the correct answer was present in the same response, scattered in the margins. The risks section admitted that a public pepper can be brute-forced over common identifiers - which quietly destroys the headline claim that public-key hashing is irreversible. Another bullet correctly noted that keeping a secret key for linkage requires expert determination - which is the actual rule, and refutes the answer's own bottom line. The model had all the pieces and still shipped the inverted conclusion in the headline, the table, and the summary.
Round three: it's not me
One hypothesis left: maybe my framing poisoned the well, since my first question presented the counterintuitive claim as a finding. So I opened a fresh chat and asked the neutral, symmetric question - which is better for HIPAA, hash with a private key or a public key - with no premise embedded.
Verdict, verbatim in spirit: public-key hashing is "far better," private-key hashing is still PHI, and - my favorite line - the public pepper should be "long, random, and not guessable."
A public value that is not guessable. It's published. That single phrase is the whole confusion in miniature: the answer needs the key to be simultaneously known to everyone (so nobody has privileged access) and known to no one (so nobody can brute-force). For calibration, I asked ChatGPT's web version the same neutral question. It got it essentially right: public-key hashing rejected for exactly the dictionary-attack reason, keyed HMAC labeled as pseudonymization that doesn't exit PHI obligations by itself, random tokens with a secured mapping table recommended for real de-identification. Not perfect - it never cited the derivation prohibition - but directionally sound everywhere it committed.
Same question. One product inverted, reproducibly, across independent chats. The other didn't.
The citation trail explains everything
Both of the reference-backed Copilot answers - rounds two and three - leaned on the same academic source: a 2003 AMIA paper by Landi and Rao, "Secure De-identification and Re-identification". I looked it up.
The paper describes an asymmetric encryption scheme: encrypt patient identifiers with a public key so that only the holder of the matching private key - the data owner - can decrypt and re-identify. "Public key" as in public-key cryptography. One half of a keypair. A system whose entire security rests on the private key staying secret, and whose explicit purpose is to let the owner retain re-identification capability.
Now the most plausible failure chain is visible. Retrieval surfaced a paper with "public key" and "de-identification" in close proximity. Summarization collapsed public-key cryptography into publicly known key. And then - this is the part I find genuinely instructive - the model didn't just misread a term. It constructed an entire regulatory philosophy around the misreading: the "privileged knowledge" theory of HIPAA, delivered with the confidence of a law professor, appearing in no regulation, contradicted by the very paper being cited. The citation that was supposed to ground the answer was proof of the opposite.
Which model was that, even?
The chat window said "Smart." That's a mode label, not a model name. So I asked Copilot directly which model it runs on. It denied running on any outside lab's models at all - just "proprietary Windows AI technologies" - and said it cannot disclose specifics.
That answer is worth exactly nothing, and knowing why it's worth nothing is the useful part. A chatbot's claim about its own identity is generated text like everything else it says - models have no introspective access to the infrastructure serving them, and "I can't disclose" deflections are typically system-prompt policy, not knowledge. The same product that invented an HHS quote is not a reliable witness about its own internals.
The public reporting says something different and messier: modern assistant products run mixed fleets - frontier models licensed from partner labs alongside the vendor's own in-house models - with a cost-driven router deciding invisibly, per query, which one you get. The lineup shifts between announcements, the routing shifts without any announcement at all, and there's no per-response indicator. The same question may be served by a different model next week, or later today.
I don't know whether my three inverted answers came from a fast-path model, from the summarization layer garbling search results, or from the persona tuning that opens responses with "let me say this clearly and directly." That's the point: as a user, I can't know - and asking the product just adds one more unverifiable claim to the pile. When another assistant answers the same question correctly, the difference isn't necessarily raw model capability. It's everything wrapped around the model, and the wrapper is invisible. "Which LLM does this product use" turns out to be the wrong question. The right one is "which pipeline, with which retrieval, which router, and which incentives" - and no consumer product answers it. Not even when you ask it directly.
What I'm taking away
- Confidence formatting is free. Tables, verdict emoji, "let me say this clearly" - none of it correlates with correctness. The most wrong answer in this story was the best-formatted one.
- "Support this with references" does not mean "verify this." Once a model has committed to a conclusion, asking for sources can produce justification instead - up to and including an invented quote pinned to a real government URL. If a quote matters, search for the exact sentence.
- Watch for internal contradictions. The wrong answers refuted themselves in their own risk sections. A response whose caveats disagree with its headline is telling you which part was retrieved and which part was composed.
- For anything regulatory, read the primary source. 164.514(c) is two paragraphs. It cost five minutes and settled in one reading what three AI answers scrambled. Compliance-by-chatbot is how a "de-identified" dataset ships with a published key and becomes a reportable breach.
- The side window deserves the same skepticism as the main one. My mistake wasn't using Copilot - it was granting the quick-lookup window a lower evidence bar because the questions felt small. Nothing about the window makes the answers smaller.
I'm keeping the two-window setup, but the notepad seat is now vacant. Either something changed in that product - a router, a model swap, a summarization layer - or it could always do this and I simply never caught it; I have no way to tell, and that uncertainty is its own verdict. Either way, a tool I trusted for months just fabricated regulatory quotes with a straight face. So I'm dropping it until it stabilizes, and I'll know it has stabilized the same way I learned it broke: by spot-checking its answers against primary sources. A quick-lookup tool that requires verification of every answer isn't a quick-lookup tool anymore.
Try it yourself
Here is the prompt I used for the fresh-chat test, lightly tidied. Run it against your assistant of choice:
In terms of HIPAA compliance, what is better for de-identifying PHI data: HASH(private_key + PHI) or HASH(public_key + PHI)?
The correct answer rejects both as de-identification on their own, flags the public-key variant as trivially reversible by dictionary attack, and mentions that a keyed hash only works under the Expert Determination pathway with the key kept undisclosed. Anything that tells you the public key is the compliant option has inverted 45 CFR 164.514(c).
Share what you get in the comments - which product, which mode, and which way it landed. I'm genuinely curious whether this reproduces beyond my machine, and a comment thread of timestamped outputs is a better dataset.
Disclaimer: I'm an infrastructure engineer, not a lawyer; this is an engineering read of published regulations and guidance, not legal advice. The Copilot and ChatGPT responses summarized here were collected in August 2026 from consumer versions of both products; model routing and behavior change constantly, which is rather the point.
Top comments (0)