DEV Community: vadim albarov

Can a Local LLM Wash Out a Watermark Without Washing Out the Meaning? I Tested 300 Rewrites

vadim albarov — Fri, 02 Oct 2026 18:49:07 +0000

AI text watermarks hide a signal in which words a model picks. A common idea is that you can remove that signal by running the text through another model: "rewrite this in your own words."

That raises a simple question. If you rewrite a text enough to break its word patterns, do you still keep its meaning?

I tested it on a single GPU with 16 GB of VRAM. In short: yes, mostly — but the model you pick matters a lot, and there is a real trade-off between "changed a lot" and "kept everything."

The setup

Here is the pipeline:

Claude Opus 5.5 wrote 100 source texts. There were 10 genres (news, technical explainers, science, opinion, personal stories, marketing, how-to guides, business memos, history, and short fiction), with 10 topics each. Lengths were from about 190 to 900 words. Every text was packed with checkable details: names, numbers, dates, steps, and plot events.
Three local models rewrote every text with Ollama on a GPU with 16 GB VRAM:
- qwen3:4b-instruct (small)
- gemma3:12b (mid)
- qwen3:14b (upper mid, thinking mode off)
Claude Fable 5.1 judged all 300 rewrites blind. Each packet held the original and three rewrites labeled A/B/C in random order. The judge listed every lost, changed, or added fact, and gave each rewrite a score from 1 to 5.
I measured how much the wording changed by counting how many of the original's word n-grams survive in the rewrite.

All models used the same prompt:

Rewrite the text below completely in your own words.
- Change the vocabulary and the sentence structure; do not copy phrases.
- Keep ALL of the meaning: every fact, name, number, date, step, argument, and plot event.
- Do not add new information, comments, or opinions.
- Keep the same genre, tone, and roughly the same length. Keep the title line, rewritten.
Output only the rewritten text, nothing else.

Settings: temperature 0.7, num_ctx 8192, one run per text.

Why n-grams are a fair proxy for "watermark removed"

Most text watermarks (green-list schemes like Kirchenbauer et al., and Google's SynthID-Text) work at the token level. While the model writes, they nudge it toward certain tokens based on the tokens just before. A detector then counts how often these "preferred" token pairs or sequences appear.

So if most of the original word sequences are gone, most of the signal is gone too. I did not run a watermark detector, because I have no access to one for these texts. Treat the n-gram numbers as a proxy, not a proof.

Results

Meaning, by model

Model	Avg score (1–5)	"Preserved"	"Mostly preserved"	"Not preserved"	Same understanding for the reader
qwen3:14b	4.68	82%	18%	0%	100%
gemma3:12b	4.09	40%	60%	0%	98%
qwen3:4b-instruct	3.76	21%	76%	3%	81%
All 300	4.18	48%	51%	1%	93%

"Same understanding" was a yes/no question to the judge: would a reader of the rewrite come away with the same understanding as a reader of the original?

When the judge ranked the three rewrites of each text (still blind):

Model	Ranked best	Ranked worst
qwen3:14b	76	5
gemma3:12b	17	35
qwen3:4b-instruct	7	60

Average errors per rewrite, as listed by the judge:

Model	Facts lost	Facts changed	Things added
qwen3:14b	0.59	0.96	0.05
gemma3:12b	1.42	2.04	0.30
qwen3:4b-instruct	1.22	2.94	0.38

Wording change, by model

"Kept" means the share of the original's n-grams that still appear in the rewrite. Lower means a heavier rewrite.

Model	Words kept (1-gram)	2-grams kept	3-grams kept	5-grams kept	Longest copied run (words)	Length vs original
gemma3:12b	60%	34%	20%	8%	11.7	1.12×
qwen3:4b-instruct	64%	37%	23%	9%	12.4	1.05×
qwen3:14b	72%	51%	37%	21%	20.7	1.07×

The trade-off

Put the two tables side by side:

qwen3:14b kept the meaning best by far, but it also kept the most of the original wording. About 1 in 5 five-word sequences survived, and on average it copied a run of about 21 words straight from the original.
gemma3:12b changed the wording the most (only 8% of 5-grams survived) and still scored 4.09. Almost all its rewrites (98%) gave the reader the same understanding.
qwen3:4b-instruct changed the wording almost as much as Gemma, but it was the least accurate.

Across all 300 rewrites, the correlation between "meaning score" and "5-grams kept" was 0.36. So yes, rewrites that copy more wording tend to keep more meaning. But inside each model the link was weak (0.03–0.22). Most of the difference comes from which model you use, not from how hard a given model happens to rewrite a given text.

What kinds of mistakes?

The type of error mattered more than the count.

qwen3:14b mostly made tiny changes in nuance:

"two placebo pills work better than one" was hedged to "may be more effective"
"ransomware" was widened to "malware attacks"
an overall risk of "moderate" became "average"

gemma3:12b made the wording vaguer or swapped in a near-synonym:

"Most houseplants" became "Many indoor plants"
"hiring manager" became "recruiter"
Kubernetes init containers that "run to completion, one after another" became a single init container that "executes a sequence of tasks". That one is a real technical error.

Here is a typical example. Original:

SafeNest Products announced on Monday that it is recalling about 86,000 infant car seats sold in the United States because the harness buckle may fail to latch fully.

Gemma:

On Monday, SafeNest Products declared a recall of roughly 86,000 car seats intended for infants, distributed across the United States, due to a potential issue with the harness buckle's functionality.

Every fact is still there, but "may fail to latch fully" became "a potential issue with ... functionality." It reads fine, but it says less.

qwen3:4b-instruct made the dangerous mistakes: wrong names and wrong numbers.

It renamed products: "Quietwave Q7" headphones became "SilentHush S7" all through the text, and the "Elevate Pro Dual-Motor" desk became "Rise Pro Dual-Drive."
It garbled numbers: a desk height range "from 150 to 200 centimeters" became "from 150 to 20."
It changed amounts: British tea imports of "millions of pounds" a year became "hundreds of thousands of pounds."
It cut a farmer's name, "Marcos Ribeiro," to "Marcos Ribe."

A rough keyword search over the judge's notes found number-related changes in about 31 of 100 qwen3:4b rewrites, compared with about 11 for Gemma and 3 for qwen3:14b. This count is approximate, but the gap is clear.

All 3 rewrites judged "not preserved" in the whole test came from qwen3:4b. Two were the renamed products, and the third was a short story whose ending stopped making sense.

By genre and length

The genre made less difference than I expected. The average score ranged from 4.03 (marketing copy) to 4.40 (science writing). Marketing copy had the most "not preserved" verdicts. Product names and spec numbers are exactly where the small model slips, and those details are the meaning of marketing copy. Business memos and technical explainers were most often fully "preserved" (77–80%).

Length did not hurt. The 750–900 word texts scored about the same as, or a little better than, the short ones (4.33 vs 4.19). With an 8K context, all three models handled 900 words easily.

Speed

Total time for 100 rewrites on the 16 GB GPU:

Model	Total	Per text
qwen3:4b-instruct	11 min	~7 s
qwen3:14b	25 min	~15 s
gemma3:12b	25 min	~15 s

So — does the rewrite keep the meaning?

With a good 12–14B model: yes. Readers would get the same understanding from 98–100% of the rewrites, and none were judged "not preserved." The remaining issues are small shifts in nuance or precision.

With a 4B model: usually, but you can't trust the details. One in five rewrites changed what a reader would understand. Names and numbers are the weak spot, and they are often the facts that matter most.

And there is a trade-off you can't ignore:

If you want maximum fidelity, qwen3:14b is the clear winner. But it keeps more of the original wording, so if your goal is to break token-level patterns, it does the weakest job.
If you want maximum change in wording with good meaning, gemma3:12b is the sweet spot. Only 8% of 5-grams survive, and the reader gets the same understanding 98% of the time.

Caveats

No real watermark detector. N-gram overlap is a proxy. Some watermark research claims robustness to paraphrase, and semantic-level watermarks would not be hurt by word changes at all.
One judge, and it is an LLM. Fable 5.1 was strict and specific (it listed things like "price changed from X to Y"), but I did not check it against human ratings. The generator and the judge are both Claude models, which could add some bias toward Claude-like phrasing. That should affect all three local models equally, though.
One run per text at temperature 0.7. Another seed would give somewhat different rewrites.
English only, synthetic texts, one prompt. A stricter prompt ("never change any name or number") would probably help the 4B model.
The keyword counts of name and number errors are approximate.

Next time: SynthID?

This test answered only half of the question. It showed that a good local model keeps the meaning, but it only estimated watermark removal from n-gram overlap.

A real check would use SynthID-Text, Google's open-source text watermark. Would rewriting with a local model really remove a watermark that a detector can see?

Should I run that test next? Tell me in the comments.

How Much Does Your Lunch Break Cost in Claude Code Tokens?

vadim albarov — Thu, 01 Oct 2026 15:06:06 +0000

You step away for lunch with a big Claude Code session open. You come back, type one message, and continue. Nothing looks different. But that one message may have cost 20 times more than the one before it.

I wanted a real number, so I scanned a month of my own Claude Code logs. The short answer: a typical break past the one-hour mark cost me about 125k–135k tokens of cold cache writes. Over the month, these restarts added about 15% to my input usage.

This post explains why it happens, shows the script I used, and lists what I changed.

How prompt caching works in Claude Code

Every turn in a Claude Code session sends the whole conversation to the model again: system prompt, tool definitions, project files you read, tool output, everything. A long session can carry 100k–400k tokens of context.

Prompt caching makes this affordable. After the first request, the stable prefix of the prompt is stored in a cache. Later requests read it from the cache instead of processing it again. In API pricing terms, relative to the base input price:

Token type	Relative price
Uncached input	1.0x
Cache write, 1-hour TTL	2.0x
Cache read	0.1x

A warm turn reads the context at 0.1x. A cold turn writes the whole context again at 2.0x. That is where the roughly 20x comes from.

The cache has a time to live (TTL). In my Claude Code sessions it is one hour, and each request resets the timer. Tool calls count as requests, so an agent working on its own keeps the cache warm. The risk is when you are the slow part: a meeting, lunch, the end of the day.

Note: the 1-hour TTL is what my sessions used. The default API TTL is 5 minutes, and Claude Code can drop to 5 minutes in some cases, for example usage overage. The script below reports which TTL each write used, so you can check your own.

Where the data lives

Claude Code stores each session as a JSONL file under ~/.claude/projects/<project>/<session-id>.jsonl. Every assistant turn includes a usage block. Trimmed:

{
  "type": "assistant",
  "timestamp": "2026-10-01T13:28:05.993Z",
  "message": {
    "id": "msg_...",
    "usage": {
      "input_tokens": 2,
      "cache_creation_input_tokens": 32912,
      "cache_read_input_tokens": 0,
      "output_tokens": 211,
      "cache_creation": {
        "ephemeral_1h_input_tokens": 32912,
        "ephemeral_5m_input_tokens": 0
      }
    }
  }
}

Three fields matter:

cache_read_input_tokens: context served from the cache (cheap).
cache_creation_input_tokens: context written to the cache (expensive).
cache_creation.ephemeral_1h_input_tokens / ephemeral_5m_input_tokens: which TTL the write used.

On a normal warm turn, reads are large and writes are small (just the new tool output). A cache miss looks the other way round: a large write, little or no read. If the previous turn in the same file was more than an hour earlier, the idle gap is the cause.

The script

The script walks every session file, removes duplicate messages by ID, and flags turns where:

cache_creation_input_tokens is at least a threshold (default 50,000), and
the write is larger than the read.

It records the gap since the previous turn, groups the misses by gap length, and estimates a weighted cost share using the prices above.

Click to see the full script (100 lines)

import json, glob, os, sys
from datetime import datetime, timedelta
from collections import defaultdict

ROOT = os.path.expanduser("~/.claude/projects")
MIN_MISS = int(sys.argv[1]) if len(sys.argv) > 1 else 50_000

def parse_ts(s):
    return datetime.fromisoformat(s.replace("Z", "+00:00"))

misses = []
totals = defaultdict(int)
per_project = defaultdict(lambda: defaultdict(int))

for path in glob.glob(os.path.join(ROOT, "*", "*.jsonl")):
    project = os.path.basename(os.path.dirname(path))
    session = os.path.basename(path)[:8]
    prev_ts = None
    seen_ids = set()
    with open(path, encoding="utf-8", errors="replace") as f:
        for line in f:
            try:
                d = json.loads(line)
            except json.JSONDecodeError:
                continue
            if d.get("type") != "assistant":
                continue
            msg = d.get("message") or {}
            u = msg.get("usage")
            ts = d.get("timestamp")
            if not u or not ts:
                continue
            mid = msg.get("id")
            if mid in seen_ids:
                continue
            seen_ids.add(mid)
            t = parse_ts(ts)
            read = u.get("cache_read_input_tokens", 0) or 0
            created = u.get("cache_creation_input_tokens", 0) or 0
            raw = u.get("input_tokens", 0) or 0
            cc = u.get("cache_creation") or {}
            w1h = cc.get("ephemeral_1h_input_tokens", 0) or 0
            w5m = cc.get("ephemeral_5m_input_tokens", 0) or 0
            totals["turns"] += 1
            totals["read"] += read
            totals["created"] += created
            totals["raw"] += raw
            per_project[project]["read"] += read
            per_project[project]["created"] += created
            gap = (t - prev_ts) if prev_ts else None
            if created >= MIN_MISS and created > read:
                misses.append(dict(project=project, session=session, ts=t,
                                   gap=gap, created=created, read=read,
                                   w1h=w1h, w5m=w5m))
            prev_ts = t

misses.sort(key=lambda m: m["ts"])

def fmt_gap(g):
    if g is None: return "start"
    h = g.total_seconds() / 3600
    return f"{h:6.1f}h" if h >= 1 else f"{g.total_seconds()/60:5.0f}m "

print(f"Cache misses >= {MIN_MISS:,} created tokens, read < created\n")
print(f"{'date':16} {'gap':8} {'created':>9} {'read':>8} {'ttl':4} {'project':28} sess")
buckets = defaultdict(lambda: [0, 0])
for m in misses:
    ttl = "1h" if m["w1h"] >= m["w5m"] else "5m"
    print(f"{m['ts'].strftime('%Y-%m-%d %H:%M'):16} {fmt_gap(m['gap']):8} {m['created']:>9,} {m['read']:>8,} {ttl:4} {m['project'][:28]:28} {m['session']}")
    g = m["gap"]
    key = ("session start" if g is None else
           "<5m" if g < timedelta(minutes=5) else
           "5m-1h" if g < timedelta(hours=1) else
           "1h-24h" if g < timedelta(hours=24) else ">24h")
    buckets[key][0] += 1
    buckets[key][1] += m["created"]

print("\nMisses by idle gap before the request:")
for k in ["session start", "<5m", "5m-1h", "1h-24h", ">24h"]:
    n, tok = buckets[k]
    print(f"  {k:14} {n:4} misses  {tok:>12,} tokens written")

miss_tok = sum(m["created"] for m in misses)
print(f"\nAll assistant turns: {totals['turns']:,}")
print(f"  cache read tokens    : {totals['read']:>14,}")
print(f"  cache created tokens : {totals['created']:>14,}  ({miss_tok:,} in the misses listed above)")
print(f"  uncached input tokens: {totals['raw']:>14,}")
# cost weights: read 0.1, 1h write 2.0, raw 1.0
w_read = totals['read'] * 0.1
w_created = totals['created'] * 2.0
w_raw = totals['raw'] * 1.0
w_miss = miss_tok * 2.0
tot = w_read + w_created + w_raw
print(f"\nWeighted input cost share (read=0.1, write=2.0, raw=1.0):")
print(f"  cache reads : {w_read/tot*100:5.1f}%")
print(f"  cache writes: {w_created/tot*100:5.1f}%   of which listed misses: {w_miss/tot*100:5.1f}%")
print(f"  raw input   : {w_raw/tot*100:5.1f}%")

print("\nTop projects by cache-write tokens:")
for p, v in sorted(per_project.items(), key=lambda kv: -kv[1]["created"])[:8]:
    print(f"  {p[:40]:40} written {v['created']:>12,}  read {v['read']:>14,}")

Run it with an optional threshold:

python cache_misses.py          # default 50,000
python cache_misses.py 30000    # lower threshold

My results

The data covers 2026-08-31 to 2026-10-01: 2,410 assistant turns across five projects.

The misses

Restarts after an idle gap of over one hour, threshold 50k:

Idle gap	Tokens re-written
1.1h	170k
1.1h	80k
1.2h	137k
1.2h	77k
1.2h	193k
1.3h	80k
1.4h	142k
1.5h	58k
1.6h	110k
1.6h	164k
2.2h	148k
2.5h	53k
2.6h	171k
2.6h	103k
2.8h	131k
15.1h	124k
18.7h	233k
61.8h	476k

Metric	Value
Idle restarts over 1h	18
Tokens re-written cold	2.65M
Mean per restart	147k
Median per restart	134k
Median for gaps of 1.1h–1.6h (10 restarts)	123k

The mean is pulled up by one outlier: a session resumed after a 2.5-day weekend that re-wrote 476k tokens in one request. The median is the more honest "cost of a break".

The pattern

Most gaps barely passed the limit. 10 of the 18 were between 1.1 and 1.6 hours. These are lunch breaks and meetings, not abandoned sessions.
One stale session cost as much as three or four normal breaks. Resuming a huge session after days away is the worst case.
Session starts are a separate, fixed cost. At a 30k threshold the script also shows 14 session starts, each 31k–36k tokens. That is the startup context: system prompt, CLAUDE.md, memory, skills, and MCP tool definitions. Every new session pays it once, and no timing habit removes it.

The cost share

Weighted by relative price across all 2,410 turns:

Bucket	Tokens	Weighted share
Cache reads	234M	55%
Cache writes, normal incremental	6.8M	32%
Cache writes, idle misses	2.7M	13%
Uncached input	82k	0.2%

Idle misses were 13% of the total weighted input. Put another way: without them, input cost would have been 87% of what it was, so the misses added about 15% on top.

The 32% for normal writes is the ordinary cost of new tool output going into the cache each turn. Timing does not change it. Only the 13% slice can be avoided.

What I changed

1. Come back within the hour when I can. Most of my misses were 5 to 35 minutes past the limit. If I know I will be back soon, I make the next message the first thing I do.

2. Run /compact before a long break. If I will be away for more than an hour and the session is large, I compact while the cache is still warm. The summary is cheap to produce on a warm cache. The restart after the break then re-writes a small summary, not 150k tokens of history, and every turn after that is cheaper too.

3. Do not revive stale sessions. For anything older than a day, I start a fresh session with a short brief: goal, current state, key files. That costs about 35k tokens of startup context, not 400k+ of old history.

4. Do not bother with keep-alive pings. Sending a "heartbeat" message every hour does keep the cache warm, and each one costs only about 0.1x of the context. On paper it wins for gaps up to about 20 hours (20 pings at 0.1x equal one cold write at 2.0x). In practice it adds noise to the conversation, it stops when the machine sleeps, and it keeps a large context alive that you may not need. /compact is simpler.

Caveats

Plan differences. On a subscription plan, tokens count against usage limits, not a bill. The relative weights still show what eats your limits faster, but your plan may not count tokens exactly like API prices.
The weights are an estimate. Read = 0.1, 1h write = 2.0, uncached = 1.0 follows published API pricing ratios. Output tokens are left out on purpose, because they do not depend on caching.
The heuristic is simple. "Write > read and write ≥ 50k" catches idle restarts well. It also catches a few non-idle events, such as one write under 5 minutes in my data, so always look at the gap column.
One user, one month. These are numbers from my own work: several infrastructure and app repos, long agentic sessions, normal office hours.

Try it on your own logs

Save the script, run it, and look at two things: the 1h-24h bucket and the >24h bucket. If they hold a large share of your cache writes, a habit change will save you more than any prompt tweak.

Over to you: is saving ~15% of tokens worth changing how you work? Or is the extra attention not worth it? I'd like to know how you handle long sessions and breaks.

These numbers are averages from one month of my own work. Your numbers could vary.

Typos don't break LLM prompts. One missing quote mark does.

vadim albarov — Sun, 27 Sep 2026 21:02:46 +0000

Everyone who types prompts has wondered the same thing. Does the model care that I wrote "teh" or forgot a comma? Should I clean up my prompt before hitting enter?

I ran a test to find out. The short answer: stop fixing your spelling. Start checking your quotes and colons.

What I tested

I wrote a set of tasks with short, checkable answers. Arithmetic, extraction from a passage, list filtering with "except" and "not", a Python snippet's output, counting words inside quotes, and so on. Then I broke the prompts in controlled ways and sent every version to every model, several times each, in a fresh session every time.

Two prompt sets:

Easy set. 12 one-line tasks, each hand-written at five error levels, from clean to "heavy typos plus broken punctuation".
Hard set. 9 long prompts (200 to 450 words) where the question depends on a detail buried in the text. Here I separated the two kinds of damage:
- Spelling noise: a script misspells 35% or 70% of the words, but never touches the words the answer depends on.
- Structural break: the spelling is perfect, but exactly one punctuation mark that carries meaning is wrong. A missing closing quote. A colon dropped before a list. A comma moved around "except". 9.45 instead of 9:45. resign instead of re-sign.
- Two realistic profiles on top: non-native grammar (articles and verb forms off, spelling fine) and voice-to-text (no punctuation, no capitals, numbers spelled out, "resign" for "re-sign").

Models: Claude Opus 5, Opus 5.5, Sonnet 5, Haiku 4.5, Fable 5.1 (all at low and medium effort), plus seven local models through Ollama: gemma4, qwen3 14B, phi4 14B, granite 4.2, ornith-1.5 9B, llama 3.1 8B, mistral 7B.

Total: about 4,900 sessions. An answer counts as correct if the ideal answer appears in the reply, even with an explanation attached.

Result 1: spelling and grammar cost nothing

Here is the hard set. Each cell is the percent of correct answers.

model	clean	70% typos	non-native grammar	voice-to-text	one punctuation break
Opus 5	100	100	100	100	88
Opus 5.5	100	98	100	100	88
Fable 5.1	100	100	100	98	92
Haiku 4.5	100	100	100	98	88
Sonnet 5	100	100	100	89	77
granite4.2	93	67	89	56	88
phi4 14B	74	74	85	78	62
gemma4	78	78	81	44	79
qwen3 14B	67	74	85	56	62
ornith-1.5 9B	74	48	70	52	42
llama 3.1 8B	44	56	63	33	38
mistral 7B	52	26	41	56	38

Read the Claude rows first. With 70% of the words misspelled, every one of them scored the same as on clean text. Non-native grammar: 100% across the board. This is the prompt that scored 100%:

shop sellin pens 3dolar each an notbooks 5 dolar,each tom buys 4pen an 2notbook.how much he pay totaly,anser numbr only

Now read the last column. One punctuation mark, with perfect spelling everywhere else, and the same models lose 8 to 23 points.

Result 2: which punctuation mark breaks which model

Not every break matters. Most were recovered from context by every Claude model. Two were not.

break	Opus 5	Opus 5.5	Fable	Haiku	Sonnet
missing closing quote	0	33	33	0	17
colon dropped before a list	100	67	100	100	0
comma moved around "except"	100	100	100	100	100
European decimals (1.500 at 2,50)	100	100	100	100	100
9.45 instead of 9:45	100	100	100	100	100
managers' → manager's	100	100	100	100	100
resign instead of re-sign	100	100	100	100	100
comma dropped in "not closed, and assigned to Lee"	100	100	100	100	100

The missing quote. The task: count how many times "report" appears inside a quoted paragraph, not in the text after it. Remove the closing quote and every model counts the three extra "report"s after the paragraph. Answer 8 instead of 5. Opus 5.5 even wrote "the closing quote is missing, so I counted all" and still gave the wrong number. It noticed the problem and followed the broken structure anyway.

The dropped colon. "Give the plural form of each word below keeping the order mouse child foot analysis reply with the plurals only comma separated". Sonnet answered mice, children, feet, analyses, replies. Every local model did the same. The word boundary between instruction and data was gone, so "reply" became data.

Everything else was recovered, including the genuinely ambiguous one. "List the tickets that are not closed and assigned to Lee" can mean two things once the comma is gone. All five Claude models picked the intended reading every time, because the sentence started with "For Lee's standup".

Result 3: dictation is a different story

The voice-to-text column is where the small models fall apart. gemma4 goes from 78 to 44. granite from 93 to 56. Two things do the damage: spelled-out numbers ("three hundred forty pallets across twenty two trailers") and, again, the missing colon before a list. Every local model scored 0% on the dictated list task.

The frontier models mostly shrug it off, with one honest exception. "fifteen hundred units at two fifty each": Sonnet read "two fifty" as $250 in six trials out of six and answered 375,000 instead of 3,750. Fable did it once. Haiku turned "extension two zero one" into 2001 once. These are not model bugs. Spoken numbers are ambiguous, and the model has to guess.

Things that did not help

Effort level. Low versus medium made no difference for any model on any task. Structural breaks are not a reasoning problem, so more thinking does not fix them.
Warning the model. I prepended "the message below may contain spelling, grammar, and punctuation mistakes, answer according to the writer's likely intent". Zero effect on Opus 5, Haiku, and Sonnet. A few points on Opus 5.5 and Fable. Mixed or negative on the local models.

What degrades first: the format, not the answer

Sloppy prompts get slightly sloppier replies before they get wrong ones. Asked for "the number only", Fable and Opus 5.5 add a line of working in about 20% of runs on any prompt. Haiku, on the easy set, drifted into "Yes. If all bloops are razzies..." explanations as the grammar got worse, and at the heaviest level refused four times out of eight because "bloops and razzies are not real words". With clean grammar it had answered the same nonsense-word question fine.

What to actually do

Don't fix spelling or grammar. The model does not care, and neither should you.
Do check the punctuation that carries structure: closing quotes, the colon before a list, commas around "except" and "not", and anything separating your instructions from your data.
If you dictate prompts, re-read the numbers and the list boundaries before sending.
Small local models are much less forgiving. If you run a 7B to 14B model, treat every delimiter as load-bearing and avoid dictation.

Caveats

Twelve short tasks and nine long ones, three trials per cell, one random seed for the typo generator. The voice-to-text prompts are hand-simulated, not real speech engine output. Claude models were called through the Claude Code CLI with a minimal system prompt and tools disabled. Local models ran at Ollama defaults with thinking off. The harness and every prompt and reply are in the repo linked below, so you can rerun it with your own tasks.

Repo with the harness, all prompts, and every raw reply: github.com/valbarov/prompt-noise-test

12 AWS Defaults That Ship Insecure (and the One Line That Fixes Each)

vadim albarov — Sun, 27 Sep 2026 01:19:02 +0000

Every AWS default answers exactly one question: will the tutorial work?

No surprise bill. No failed API call. No "access denied" on step three. That is a great default for a tutorial. It is a terrible default for the thing you spun up "just for staging" that is now production, on the day someone asks who downloaded that bucket in March.

I run infrastructure for a healthcare company. People with clipboards read my configs. Somewhere around my third Terraform module I noticed my job had a shape, and the shape was not "design clever things". It was flipping the same switches AWS had left in the wrong position, over and over. So I wrote them down.

Here are the twelve, ranked by how likely each one is to hurt you. One rule of thumb before we start: checkov and tfsec check what you wrote. Most of this list is about what you did not write.

1. CloudTrail remembers 90 days, then forgets

Default: you get "Event history": 90 days of management events, viewable in the console. No trail to S3 unless you create one. And S3 object reads and writes (data events) are not logged at all, even when a trail exists.

How it bites: a credential gets phished in March. In September someone asks what it touched. The window scrolled off in June, and even inside the window "did they download the sensitive bucket" was never recorded. Your answer is a shrug.

The fix:

resource "aws_cloudtrail" "audit" {
  name                       = "org-audit"
  s3_bucket_name             = aws_s3_bucket.audit.id
  is_multi_region_trail      = true
  enable_log_file_validation = true

  event_selector {
    read_write_type = "All"
    data_resource {
      type   = "AWS::S3::Object"
      values = ["arn:aws:s3:::my-sensitive-bucket/"]
    }
  }
}

Data events cost real money at volume. Scope them on purpose, not by forgetting.

2. RDS storage is unencrypted, and you cannot fix that later

Default: storage_encrypted = false in Terraform and in the API. The console nudges you. Code does not. And encryption can only be enabled at creation. Later means snapshot, encrypted copy, restore, cutover.

How it bites: this is the worst one on the list because it is irreversible in place. Staging quietly becomes production, it always does, and eighteen months later a review finds your primary database in plaintext. The remediation is a maintenance window on your busiest system.

The fix:

storage_encrypted = true
kms_key_id        = aws_kms_key.data.arn

In my modules this is hardcoded, not a variable. A security invariant that callers can toggle off is not an invariant. It is a default waiting to come back.

3. Your Postgres accepts plaintext connections

Default: on RDS for PostgreSQL 14 and earlier, rds.force_ssl is 0. The server supports TLS. It also cheerfully accepts connections without it. AWS flipped this to 1 for PostgreSQL 15+, which tells you what AWS thinks of the old default.

How it bites: any client on sslmode=prefer (the libpq default) falls back to plaintext if the handshake hiccups. Nothing fails. Nothing logs. Your transmission security now depends on every developer, sidecar, and ad hoc psql from a bastion remembering a flag.

The fix: enforce it at the engine so client config stops mattering:

parameter {
  name  = "rds.force_ssl"
  value = "1"
}

4. Your load balancer still shakes hands with 2008

Default: create an HTTPS listener without a policy and you get ELBSecurityPolicy-2016-08, which accepts TLS 1.0 and 1.1. Also off by default on the same resource: access logs and deletion protection.

How it bites: nobody notices, because modern browsers pick 1.2+. Then a customer's security team runs an external scan before signing, and the deal stalls on a finding one attribute would have prevented.

The fix:

ssl_policy = "ELBSecurityPolicy-TLS13-1-2-2021-06"

And on the aws_lb itself: access_logs { enabled = true }, enable_deletion_protection = true, and drop_invalid_header_fields = true while you are there.

5. VPC Flow Logs do not exist

Default: no VPC has flow logs until you create them. No network-level record of anything.

How it bites: GuardDuty flags an instance talking to a known-bad IP. The first question is "since when, and what else did it talk to?" Without flow logs your incident response runs on vibes, and "was anything exfiltrated?" gets answered conservatively, which means expensively.

The fix:

resource "aws_flow_log" "vpc" {
  vpc_id               = aws_vpc.main.id
  traffic_type         = "ALL"
  log_destination_type = "cloud-watch-logs"
  log_destination      = aws_cloudwatch_log_group.flow.arn
  iam_role_arn         = aws_iam_role.flow.arn
}

ACCEPT only or REJECT only is half an audit trail.

6. Services create log groups you never asked for

Default: log groups default to Never Expire and an AWS-managed key. The sneaky part: RDS log exports, Container Insights, and Lambda create their own log groups on first write, with exactly those defaults, outside your Terraform state.

How it bites: you enable RDS log exports. RDS creates /aws/rds/instance/.../postgresql for you. Three years later that orphan holds three years of connection logs plus whatever your app leaked into query errors, retained forever, under a key you did not choose, and invisible to terraform destroy.

The fix: pre-create every log group a service will write to, before the service exists:

resource "aws_cloudwatch_log_group" "rds" {
  name              = "/aws/rds/instance/myapp-db/postgresql"
  retention_in_days = 2192
  kms_key_id        = aws_kms_key.logs.arn
}

7. The default security group is a party where everyone knows everyone

Default: every VPC ships with a default security group that allows all traffic between members and all outbound. Anything launched without an explicit group lands in it.

How it bites: a contractor spins up a "quick utility box" without thinking about security groups. It joins the default one, alongside everything else that drifted in over the years. That box gets popped, and lateral movement is free, because membership is the authorization.

The fix: you cannot delete the default group, but you can strip it bare. A resource with no ingress or egress blocks removes every rule, so anything landing there can talk to precisely nothing:

resource "aws_default_security_group" "this" {
  vpc_id = aws_vpc.main.id
  tags   = { Name = "default-DO-NOT-USE" }
}

Loud failure beats silent success.

8. RDS backups: one day via the API, zero via Terraform

Default: backup retention is one day via API or CLI, seven via the console, and backup_retention_period defaults to 0 in Terraform. Zero. Automated backups off. Deletion protection is also off.

How it bites: an instance defined without that line has no backups at all, and it passes plan, apply, and review, because absence does not show up in a diff. You find out during your first real restore attempt, which is the single worst moment available. Bonus: with deletion protection off, one terraform destroy in the wrong workspace takes the database and its backups together.

The fix:

backup_retention_period   = 30
deletion_protection       = true
skip_final_snapshot       = false
final_snapshot_identifier = "myapp-db-final"

Then AWS Backup with a locked vault on top, because backups attached to the instance share the instance's blast radius.

9. EBS encryption by default is off, in every region separately

Default: account-level EBS encryption by default is disabled, and it is a per-region setting. Turning it on in us-east-1 does nothing for us-west-2.

How it bites: your Terraform encrypts every volume it manages. Then someone launches a console instance for a one-off migration, copies a database dump onto it, and that volume is plaintext, because the account default governs ad hoc resources, not your module.

The fix: one resource, once per region you use, including the ones you think you do not use:

resource "aws_ebs_encryption_by_default" "this" {
  enabled = true
}

10. SNS stores your alerts in plaintext

Default: server-side encryption on SNS topics is off until you set a KMS key.

How it bites: mostly as an audit finding. Occasionally worse: a team wires appointment reminders through SNS, and now message bodies with patient data sit unencrypted in the messaging layer while every database in the stack is dutifully encrypted. One-line fix or an hours-long finding memo. Pick the line.

The fix:

resource "aws_sns_topic" "alerts" {
  name              = "security-alerts"
  kms_master_key_id = aws_kms_key.logs.arn
}

Real gotcha: once the topic is encrypted, CloudWatch and EventBridge need kms:Decrypt and kms:GenerateDataKey* in the key policy, not in IAM. Otherwise every alarm publish fails silently. Test the path end to end.

11. S3 got fixed, which is exactly the problem

Default (current): since January 2023 every new object is encrypted with SSE-S3. Since April 2023 new buckets get Block Public Access on and ACLs off. Credit where due: the two most famous S3 footguns are gone for new buckets. Still off: server access logging, versioning, and any customer-managed key.

How it bites: "S3 encrypts by default now" becomes the reason nobody configures anything further. A sensitive bucket ends up with no access log, no versioning, no policy denying plaintext transport, and a key nobody can audit.

The fix: for a bucket that matters: SSE-KMS with your own key, versioning on, aws_s3_bucket_logging to a separate log bucket, and a bucket policy denying aws:SecureTransport = false. The 2023 change is your floor, not your control.

12. ECS Exec gives you a shell, and records nothing

Default: ECS Exec logging is DEFAULT, meaning "whatever awslogs config the task has". If the container has none, session input and output are not logged at all. KMS encryption of the session channel is opt-in.

How it bites: an engineer execs into a production task to debug, runs a few queries, pastes results somewhere. Months later an access review needs that session. CloudTrail says a shell was opened at 14:32. That is the entire record.

The fix: on the cluster, execute_command_configuration with logging = "OVERRIDE", a dedicated encrypted log group, and a kms_key_id for the channel. Every session becomes a transcript.

Automate it or relive it forever

Twelve items is too many for humans to re-run reliably. That is the real lesson. What works for me:

One baseline module for the apply-once account controls: the trail, AWS Config rules like RDS_STORAGE_ENCRYPTED and VPC_FLOW_LOGS_ENABLED, EBS default encryption, the account-level S3 public access block.
Hardcoded invariants in workload modules. storage_encrypted, publicly_accessible = false, forced TLS, and log validation are not variables in mine. Exposing them as inputs just recreates the permissive default one level up, with your name on it.
Policy as code for the rest. checkov in CI catches regressions in code. AWS Config catches the console-created, script-created, "temporary" resources your code never met. You need both, because the resources that hurt most never saw a pull request.

The pattern across all twelve is the same. AWS optimizes for the first five minutes. Your auditor cares about year five, when someone asks who accessed what and the answer has to exist. Those two goals produce opposite defaults, and closing the gap is, by AWS's own shared responsibility model, your job.

Not legal or compliance advice. I am an infrastructure engineer, and defaults change, sometimes even in the right direction, so check each one against current docs.

Which default bit you? I only know the twelve that bit me.

Claude Code's Max Effort Cost Me 8x More. It Wrote the Same Code.

vadim albarov — Tue, 15 Sep 2026 21:44:18 +0000

Claude Code has an /effort command with five levels: low, medium, high, xhigh, max. The help text for low says "quick, straightforward implementation with minimal overhead." The natural assumption is that the higher levels buy you better code, and that the price is time and tokens.

I wanted to see that trade-off on a chart. So I built a small harness, gave every level the same task in a clean copy of the same folder, and measured. I did it three times with three different task shapes.

The chart never appeared. Every level got the same score on every task. Only the clock and the invoice moved.

The harness

Nothing clever. One headless Claude Code process per level, running in its own copy of the project:

claude -p "$(cat prompt.txt)" --effort low \
  --output-format stream-json --verbose \
  --permission-mode acceptEdits --allowedTools "Read,Write,Edit,Bash(python*)"

The stream-json output goes to a log file. A small viewer prints readable progress in a visible PowerShell window, so I could watch five terminals work through the same problem one after another. When all runs finish, an analyzer parses the logs for turns, tool calls, cost, and token counts, then runs a hidden test suite against each folder. The child sessions never see the hidden tests.

I ran each level once. Keep that in mind for every number below. One sample is enough to show a 9x gap. It is not enough to tell a 10 percent difference from noise.

Model was Fable 5.1 on every run. Claude Code 2.1.272.

Benchmark 1: implement from docstrings

A tiny Python order-processing library: money parsing, coupons, tax, inventory with reservations. Nine functions and one class, all stubbed out with NotImplementedError and detailed docstrings. Three public tests visible. 57 hidden tests.

The prompt:

Implement every function and method marked # TODO in the shop package exactly according to the docstrings. Do not change the docstrings, models.py, or anything in tests/. Run python -m pytest -q to check your work. When done, reply with a one-line summary.

Results:

effort	turns	wall time	cost	output tokens	hidden tests
low	12	77 s	$0.59	5,174	57/57
medium	12	69 s	$0.59	4,995	57/57
high	23	174 s	$1.22	13,009	57/57
xhigh	18	297 s	$2.04	24,919	57/57
max	31	693 s	$4.72	62,196	57/57

Every level passed everything on the first try. Max took eleven and a half minutes and $4.72 to arrive at the same 57/57 that low reached in 77 seconds for 59 cents. Output tokens grew 12x. Tool calls grew only about 3x, so most of that growth is reasoning, not doing.

Low and medium were near clones of each other. Same number of turns, same sequence: read, write three files, run pytest, one sanity check, done. The diff between their outputs is 41 lines of reordering.

The code was not identical across levels, though. Since the tests could not separate them, I diffed the implementations by hand.

Low and medium parsed money strings by stripping the sign and the dollar symbol and checking each character. They delete commas, so "1,23.45" is accepted.
High wrote one verbose regex with named groups that enforces comma groups of three.
Xhigh and max factored parsing into helpers. Max also widened Decimal precision with a localcontext for very large amounts. No test covers that.
Tax rounding: low and medium used Decimal with quantize. High, xhigh and max switched to Fraction for exact rational math and a hand-written half-up round.
Max added validation nobody asked for. The inventory constructor rejects negative stock, and reserve rejects quantities below one.

So higher effort produced more defensive, more factored code. It did not produce more correct code, because there was nothing left to be more correct about.

One detail I liked: the only actual spec deviation came from high, not low. The docstring said to raise KeyError with the coupon code as the message. High raised it with the uppercased code. My test used an all-caps code, so it passed anyway. Effort level did not predict spec fidelity.

Benchmark 2: port it to Go

The first task was too easy. A port to another language has real traps: Python has Decimal, Fraction, arbitrary ints and exceptions. Go has none of those. Errors are values, half-up rounding is manual, and the model has to decide how an OutOfStock error carries its fields.

Same harness. The base folder held the Python reference source and an empty Go module. Hidden Go tests import a fixed package path. The analyzer also runs go vet and gofmt -l for free quality signals.

I ran low, medium and high, then stopped to save budget.

effort	turns	wall time	cost	output tokens	hidden tests	gofmt	Go lines	test lines
low	18	135 s	$1.04	11,175	all pass	ok	442	148
medium	32	401 s	$2.60	31,503	all pass	test file unformatted	498	339
high	26	470 s	$3.44	38,673	all pass	ok	574	420

Again, flat on correctness. Cost roughly linear in effort. What each level did with its budget was different, and this part was genuinely interesting to watch.

Low read the sources, wrote five files, ran vet and test, and stopped.

Medium wanted to probe the Python original before writing Go, to confirm exact rounding and error behavior. Good instinct. But my tool allowlist for that run did not include Python, so it retried across PowerShell and Bash several times before giving up. That is harness noise, not effort. It also wrote a test suite more than twice the size of low's, and never ran gofmt on it.

High, with Python allowed, built its own cross-language oracle. It wrote a Python script that ran the original code over many inputs, generated a Go test file from the results, ran that against its port, and then deleted the generated files. That is exactly the grading technique I had planned to build myself. High invented it unprompted, which is the kind of behavior you would hope a higher effort setting buys.

It just did not change the score.

Benchmark 3: bug hunt with no execution

Both tasks so far had the same two properties: the spec was complete, and verification was cheap. Any level can iterate against a test suite until green. Extra thinking has nothing to buy.

So I broke both. A working Go version of the library with 8 planted bugs: a wrong rounding constant, bad comma grouping, a sign error in refund splits, case-sensitive coupon codes, percentage coupons discounting gift cards, tax applied to exempt categories, a non-atomic inventory reserve, and a double-release that inflates stock. The prompt gave 8 vague user reports, one per bug, and one hard rule:

You cannot run any commands in this session (no go build, no tests). Reason carefully from the code.

Tool allowlist: Read, Grep, Glob, Edit. Nothing else.

effort	turns	wall time	cost	output tokens	bugs fixed	diff
low	15	52 s	$0.94	3,677	7/8	+20 / -5
medium	15	55 s	$0.86	3,585	7/8	+16 / -3
high	16	69 s	$0.94	5,055	7/8	+26 / -4
xhigh	16	125 s	$1.30	10,200	7/8	+21 / -4

Finally a miss. All four levels missed the same bug, the tax one, and all four made the identical wrong fix. When I looked closely, that was my fault.

The correct rule was proportional: remove the exempt fraction of the pre-discount subtotal from the post-discount taxable amount. That rule lived only in the Python docstring, which was not in the Go project. From the Go code and the report "grocery and book orders are being taxed," subtracting the exempt amount directly is a completely reasonable fix. Four independent runs converging on it is evidence that the intended rule was not inferable from what they were given. It is not evidence that effort failed.

So the fair score is 7 of 7 for every level. Flat again.

What effort did change here was, once more, output tokens: 3.6k, 3.6k, 5k, 10k. Wall time and cost followed. And scope creep showed up at low, high and xhigh, each of which invented a fuzzy gift-card category matcher that handles "Gift Card", "gift_card" and "GIFT-CARD" when the original code used an exact string. Medium alone made the exact minimal change the prompt asked for.

What I take from this

Effort controls how much the model deliberates, not what it knows. On every task here the answer was fully determined and reachable in one careful pass. Given that, low found it. Max found it too, after thinking about it for eleven minutes.

Verification loops make effort irrelevant. If the session can run tests, any level will iterate until green. The Python and Go benchmarks measured persistence, and persistence was free at every level.

Higher effort spends its budget on self-checks and defensive code. Oracles, throwaway verification scripts, extra validation, regexes that reject malformed input the spec never mentioned. Sometimes that is exactly what you want. In a codebase with a scope rule, it is unrequested behavior you now have to review.

Medium was the quiet winner on discipline. It never won on speed or cost, but on the bug hunt it was the only level that did precisely what the prompt asked and nothing more.

The total bill for this experiment was about $20. Roughly $9 for the Python runs, $7 for the Go runs, $4 for the bug hunt. The max Python run alone was almost a quarter of it.

Where effort probably should matter

I have not proven that effort is useless. I have shown three tasks where it did not help, and I think I know why. Effort should matter when deliberation itself is the bottleneck:

Two-step inference. The symptom is in one file and the cause is in another, and the code near the symptom looks fine. A fix at the symptom passes the reported case and fails hidden siblings.
A decoy fix that breaks a hidden invariant. An inventory patch that resolves the report and introduces a race under go test -race.
A constraint hidden in a README that conflicts with the obvious fix.
A codebase too large to read. Around 2,000 lines, where effort shows in search strategy and in what gets verified before editing.
Repeat runs. Three samples per level at minimum. My single-sample wall times have variance comparable to the gaps between adjacent levels.

That is the next benchmark. Until then, my working rule: for a well-specified implementation task where the session can run tests, low or medium. Reserve high and above for tasks where the model cannot check itself and the answer is not sitting in the docstring.

And watch the clock. Nothing on max was wrong. It just took nine times longer to be right.

Your AI Knows the Rule. Your Question Decides Whether You Get the Myth

vadim albarov — Wed, 09 Sep 2026 13:30:00 +0000

TL;DR

HIPAA sets no retention period for audit logs. The six-year rule covers documentation. Six of the nine results on Google's first page say otherwise, and so did Gemini, eight times out of nine, citing only vendor pages.
Reworded, the same models were right six times out of seven. Naming the regulation section in the question worked every time, with web search on or off.
Asking the model to check its own answer made it worse. Each challenge produced a more specific regulator that does not exist.

The question

If you have never touched healthcare software: HIPAA is the US federal law on health data. Its Security Rule tells hospitals, insurers and every vendor that handles their electronic patient data, called ePHI, what safeguards to have. The HHS Office for Civil Rights enforces it, and the fines are real. The rule itself is short and technology-neutral, which is why so much of what people "know" about it comes from vendors rather than from the text.

How long does HIPAA require audit logs to be retained?

Everyone in healthcare IT knows the answer: six years. It is in every vendor blog, every compliance checklist, every SIEM sales deck. It is not in the regulation.

The audit controls standard, 45 CFR 164.312(b), is one sentence: implement mechanisms that record and examine activity in systems holding ePHI. No period. The only other mention of audit logs, 164.308(a)(1)(ii)(D), says to review them regularly. No period. Retention is a risk-analysis decision that you write into policy.

The six years is real but belongs elsewhere. 164.316(b)(2)(i) says to keep documentation, meaning policies, procedures and written records of required actions, for six years. Two more six-year clocks feed the lore: the accounting of disclosures lookback, and the six years OCR has to bring a penalty action. Many organizations pick six years for logs because of that last one. That is prudence, not a mandate.

Correct answer: HIPAA sets no log retention period, six years covers documentation, decide log retention in your risk analysis.

The scoreboard

Fresh chat, default settings, the neutral question, graded against the regulation text. LORE means the answer states the six-year log rule as a HIPAA requirement.

Assistant	Runs	Result
Gemini web app, 3.6 Flash and 3.1 Pro with reasoning	4	LORE, LORE, LORE, LORE
Gemini API, 3.6 Flash, web search off	4	LORE, LORE, LORE, partial
Gemini API, 3.8 Flash, web search off	1	LORE
ChatGPT, default model (the UI shows no model label), web search on	1	correct
Claude Opus 5, high effort	1	correct
qwen3 14B, local, no web	4	no myth, but 3 invented authorities
gemma4 4B, local, no web	2	no myth

Disclosure: Claude Code is my daily tool and got the same rubric. Its answer opened with the bold words "Six years" before correcting itself in the same sentence, and it claimed "regulators treat" logs as documentation, which regulators have never said. That sentence, it turned out, came straight from one of its sources. More on that below.

The two small local models never stated the myth. A 4B model on my laptop with no internet beat Gemini Pro with reasoning and web search. But qwen3 filled the gap with authorities that do not exist: an HHS recommendation, a NIST document, an OCR position. Right conclusion, invented support. Hold that thought.

Then I changed the question

Two more wordings of the same legal question, on the same Gemini models, the same day.

Wording	Runs	Result
"How long does HIPAA require..."	9	8 LORE, 1 partial
"Does 45 CFR 164.312(b) set a period..."	4	4 correct, search on and off
"Is it true that HIPAA requires six years..."	3	2 correct with search off, 1 LORE with search on

With search off, the "is it true" runs open with "this is a common misconception" and then explain, correctly, why everyone says six years anyway. The model knew all of it. With search on, the same sentence got "Yes, but with an important legal nuance", followed by "HHS auditors and industry standards universally map audit logs to the six-year documentation rule." They do not.

The wording that named the section worked everywhere.

Try it yourself

Fresh chat, default settings, no custom instructions. Run these in order:

How long does HIPAA require audit logs to be retained?
Is it true that HIPAA requires six years of audit log retention?
Does 45 CFR 164.312(b) set a retention period for audit logs?

Right: the audit controls standard sets no period, and six years is documentation under 164.316. Wrong: HIPAA requires six years of audit logs. If your assistant searches the web, expect the second one to still say yes. The third flipped every time for me.

Why the wording matters more than the model

Gemini's answers are grounded on Google Search. Google's own documentation says the model "automatically generates one or multiple search queries and executes them" (Grounding with Google Search). So Gemini's sources are whatever Google ranks for the queries the model writes. Here is what each assistant cited for the neutral question.

Assistant	Sources shown under the answer
Gemini web, 4 runs	Pangea (audit-log service), IS Partners (compliance consultancy), Cayosoft (Active Directory audit tool), Aptible (hosting). Vendors only. No primary source in any run that showed sources.
ChatGPT, search on	hhs.gov
Claude Opus 5, search on	USA HIPAA (training and compliance kits), HIPAA Auditors (audit services). Vendors, both accurate.
Local models	none, no web access

One of Gemini's vendor pages says, word for word, "HIPAA mandates that audit logs must be retained for at least six years, as per 45 C.F.R. § 164.316(b)(2)(i)." Gemini's headline was that sentence with the nouns rearranged. Claude did the same thing with better pages. Its closing line, "HIPAA does not set a single universal log retention period", is nearly word for word from one of its two sources, and the one flaw in its answer, the claim that "regulators treat" logs as documentation, is a sentence from the other. Three grounded assistants, three summaries of vendor pages. The verdict tracked the pages, not the model.

Then I searched Google for the exact question. Page one, 6 September 2026:

Rank	Who	What the snippet says
1	HIPAA Journal	documents must be kept six years (accurate, and about documentation)
2	Reddit r/cybersecurity	SIEM log retention thread
3	Kiteworks, vendor	"the HIPAA minimum of six years" for audit logs
4	Schellman, audit firm	"all audit logs ... at least 6 years"
5	Aptible, vendor	"HIPAA requires six years of audit log retention"
6	Columbia University policy	HIPAA documents six years (accurate)
7	SecurityMetrics, vendor	logs "retained for at least six years"
8	ChartRequest, vendor	"Audit logs must be retained for a minimum of six years, as outlined under 45 CFR § 164.316(b)(2)(i)"
9	UTMStack, vendor	"HIPAA's core retention rule is a 6-year minimum"

Six of nine state the six-year log rule as law. Five are vendors whose product benefits from it. Zero primary sources. HHS has no page that ranks here, because the true answer is "there is no number", and nobody writes a ranking page about the absence of a number except the people who would rather you kept six years of logs on their platform. The truth has no marketing budget.

Two things follow. The myth is in the training data: with search off, the neutral question still produced six years four times out of five, with the right section number attached to the wrong claim. And retrieval enforces it: with search on, the only wording that changed the answer was the one that changed what got retrieved. A question with the section number in it pulls pages about the section, and those pages, vendors included, say it sets no period. "Is it true that HIPAA requires" pulls the same pages as "how long does HIPAA require", and those pages say six years.

Even the right answers invented authority. One said "HHS has clarified" that raw logs are not documentation. HHS has clarified no such thing. The confident-specifics problem does not go away when the conclusion is right. It stops being visible.

Do not ask it to check itself

The obvious objection: push back. So I did, in one Gemini chat with search on.

First message, the neutral question: six years, plus "HHS interprets audit controls as part of required HIPAA documentation." No such interpretation exists.

Second message, "is it true": "Yes, it is true." Now "HHS and compliance auditors treat system audit logs as official documentation. Consequently, covered entities must retain these logs for the 6-year period."

Third message, "check again against the official resources": it quoted both sections verbatim and correctly, wrote that the audit controls standard "contains no timeframe", and one paragraph later: "OCR enforcement applies the 6-year rule to system access logs." No OCR action, guidance or FAQ says that.

The sources were the same vendor pages each time, so retrieval explains the "yes". It does not explain the escalation. Each challenge made the invented regulator more specific, and the third answer contradicted the text it had just quoted. Once the model has committed, a challenge is a request to justify, not to verify, and it invents the authority it needs. I saw the same pattern with a different assistant last month, when "support this with references" produced a fabricated HHS quote pinned to a real HHS URL. It is not a product quirk.

What I do now

Name the section. "Does 45 CFR 164.312(b) set a retention period" worked with search on and off, because it changes which pages the model reads.
Treat "is it true" as a partial fix. It works when the model answers from memory. With web search on, the vendor pages come back.
Never ask it to check itself. Open a new chat and ask the anchored question there. The chat with the wrong answer in it is poisoned.
Read the sources first. If every link sells the thing the answer says you need, ask again.
Read the section. One sentence. Faster than any of the chats.

Post your product, the model label the UI shows, the wording number, and which way it landed. I want to know whether the flip reproduces beyond my machine.

I am an infrastructure engineer, not a lawyer. This is an engineering read of the published regulation, not legal advice. Answers were collected on 6 and 7 September 2026 from consumer web apps, the Gemini API and local models; models and retrieval change constantly, which is part of the point.

Appendix: the pages the assistants cited, with the exact sentence

The pages that state the six-year log rule are named but not linked. A link is a ranking signal, and ranking is the problem. The four that got it right are linked.

Cited under the six-year answers

Page	What they sell	Date on page	Exact sentence
pangea.cloud, "HIPAA audit log requirements"	managed audit-log service	Aug 2024	"HIPAA mandates that audit logs must be retained for at least six years, as per 45 C.F.R. § 164.316(b)(2)(i)."
cayosoft.com, "HIPAA audit log requirements"	Active Directory audit software	Sep 2025	"HIPAA requires six years of audit log retention." No section cited.
ispartnersllc.com, "HIPAA audit log retention six years"	HIPAA audits and consulting	Aug 2022	Admits "the HHS does not list a set-in-stone rule about the time frame for retaining audit logs", then concludes "you should maintain your audit logs for six years."
aptible.com, "HIPAA audit log retention"	hosting	updated Mar 2026	Headline: "HIPAA requires six years of audit log retention." The body then quotes 164.316 as a documentation rule. Gemini used the headline for a wrong answer and the body for a right one.

Cited under the correct answers

Page	What they sell	Date on page	Exact sentence
LakeRidge Technologies	compliance services	Jul 2026	"HIPAA § 164.312(b) does not state a specific number of days or years that security logs must be retained."
Accountable	HIPAA compliance software	Feb 2026	"The regulation does not prescribe a specific audit log retention period."
USA HIPAA (cited by Claude)	training, certifications, compliance kits	Jul 2026	"HIPAA does not print a retention number inside its audit-controls rule, so teams guess, and most guess too short." Also the source of Claude's "regulators treat that evidence as documentation" sentence.
HIPAA Auditors (cited by Claude)	audit and certification services	Sep 2026	"HIPAA does not set a single universal audit-log day count; define periods risk-based and retain documentation per 164.316."

Four vendors got it right. Their pages rank for the precise question and not for the popular one, which is the whole story in one row.

How Many Lines of Code Do You Ship to Prod a Week? This Week I Shipped One.

vadim albarov — Wed, 26 Aug 2026 00:18:27 +0000

Yes, one. But I am as exhausted as if I had shipped a full feature.

'BackupRetentionPeriod': 0,

That is the whole fix. It took a week, more Claude Code sessions than I want to count, a /code-review pass before every pull request, one restart from scratch, and a diff that grew to a couple hundred lines before it shrank back to this. The sentence that made the one-liner possible was fetched in the very first turn. It lost - to a file on my laptop I had not updated in months, and to the model's memory, which was just as old.

The setup

Every weekday morning a Step Functions workflow restores an analytics replica from the main database's snapshot, runs some SQL, flips a DNS record, and deletes yesterday's copy. The replica lives one day.

Except a restored RDS instance inherits its backup retention from the snapshot. So every disposable replica came up with the source's week-long automated backups and started snapshotting itself within minutes. Wasted money, and another copy of PHI nobody asked for.

I know RDS. Backups are one setting. On a normal instance I would have flipped it without thinking. But this was a restore from a snapshot, and a snapshot carries the source's settings with it - so I had a real doubt: does that one setting still apply at restore time, or does the snapshot win and you have to modify afterwards? Guess: probably one parameter. Doubt: maybe not in this path.

And the workflow was agent-written. It worked, I had reviewed it, but I had not typed it and did not know every corner of it. Finding the right dict while also settling my doubt felt like a job for the model. So I asked for the change: reliable, no race conditions, check the edge cases, and - I actually wrote this - trivial and bulletproof.

I know vague prompts do not work well. But "trivial and bulletproof" did not feel vague to me. In my head it had a strict meaning: the smallest change that works, plus proof that nothing around it breaks. The model read it differently - as a license to extend the scope. At least that is what I think now; the model did not share its thoughts.

"Verified"

The planning turn was impressive. It read the module, queried both production accounts, confirmed the inherited retention, found the first automated backup landing about a minute after each restore. It fetched a stack of AWS docs. Then it wrote:

And you cannot fix it at restore time. I checked the RDS API model shipped with the local CLI:
RestoreDBInstanceFromDBSnapshot -> BackupRetentionPeriod: False

That did not fight my belief. It confirmed my doubt, with a checkmark. The thing I was unsure about had been settled by someone who looked.

The plan that followed was several times the change I had hoped for: modify the instance after restore, absorb the reboot that causes, add IAM, add a retry, widen the timeouts. I asked why. It answered every line - confidently, consistently, with a perfect track record from the values it had just pulled out of production. I had a guess, a doubt, and no working knowledge of the code. It had reasons. I was persuaded, which is not the same as convinced. I said "implement."

Here is what I missed. In the same response, to show that retention is inherited, it had quoted the RDS User Guide:

"When you don't explicitly set this value, the restored database inherits the backup retention period from the source snapshot or instance."

When you don't explicitly set this value. The doc was saying the value can be set. The model quoted it for the second half and moved on. Asked about it later, it said it had noticed the conflict and "trusted the stale artifact over the doc."

The loop

Every session that week ran the same cycle. /code-review finds real defects. The model fixes all of them, correctly. At least one fix lands outside the original ask - a cleanup path that was already weak, a timeout that "should" go up while we are here. The next review has new code to examine. Repeat.

The first review said a mutation did not belong inside a polling Lambda; the fix moved it into its own Step Functions state, with a wait, a three-way Choice, and IAM on a different role. The second review found several bugs in that new surface, and a pre-existing orphan-cleanup problem while it was in there. The third review found bugs in the cleanup rewrite. Each pass was right. Each fix was good. The ask had touched one Lambda; the diff now covered two Lambdas, the state machine, IAM, and the variables file, and had rewritten instance cleanup and database connection handling along the way. None of it was the task.

Midweek I threw it all away and started from scratch in a fresh session, hoping a clean context would find a better direction. It ran the same check against the same CLI and produced the same design with fresh confidence. Starting over resets the conversation. It does not reset the tools.

/code-review is good at its job, and that is the problem. It reviews the diff in front of it. It has no way to say the diff should not exist. So every pass certified the direction by omission and handed back a list of things to fix inside it.

The lottery

After the third pass I stopped asking for fixes and asked which findings were real, which were theoretical, and which were irrelevant - and said "none" was a fine answer. That got the most honest reply of the week: a handful real, a few theoretical, one right for the wrong reason, one pure scope creep that had spawned half the last round. "I'd stop here." It even noted the design still lost a race with RDS's first automated snapshot. I asked: commit or rewrite? Commit. I was about to.

Then I ran /code-review one last time, as I always did. This one went deep for reasons I still do not know: twenty parallel agents, the maximum, each reading the repo on its own. They used up my almost‑fresh five‑hour session token limit in just 20 minutes. The run died when the limit hit. Three agents came back. One of them said, high severity: the restore API already accepts BackupRetentionPeriod.

One of twenty. One of the three that finished. A minute later on the limit and I would have shipped a couple hundred lines and no budget left to look again.

The model re-fetched the API reference, found the announcement, confirmed my CLI predated it, and said plainly that its verification had been wrong. Everything came out except the one line.

One line.

Where "verified" came from

The CLI on my laptop was many months old. AWS added the parameter after that build, and the CLI never updates its own API model - it is a JSON file frozen into the installer. Every time the agent checked, in every session, it got the same honest wrong answer.

Half of that is mine. An agent trusts the output of commands on my machine the way I trust my eyes. Stale tools mean stale observations, and the agent cannot tell. Keeping the CLI current used to be housekeeping. With an agent doing the checking, it is ground truth - and when it rots, the agent does not fail. It succeeds at verifying the wrong world.

The other half: the parameter also postdates the model's training. So the model's memory said no, the local file said no, and I had walked in doubting. Three against one fresh sentence from a live doc. Nothing in the loop weighs a source by its date, so the newest one counted least.

What I took from it

A "verified" that lands on your own doubt is the one to check. A claim that contradicts you gets scrutiny for free. A claim that confirms your uncertainty gets none - it feels like closure. It was the one question I had come with, and I let the model close it by assertion.
"Why?" buys a justification, not a check. Every answer was about how to do the thing safely. None was about whether it was needed, because I never asked that. The premise question is: what fact, if wrong, makes this whole plan unnecessary - and where did we verify it?
"Verified" has a date. So does the model. Ask when the tool was current. Then remember the tools on your machine are yours to keep current; an agent inherits every stale binary you leave it and turns it into confident output.
Diff review certifies direction by omission. Use /code-review for what it is - a defect pass. Ask the premise question separately, in plain words, before the second review, not after the fourth.
A fresh session is not a second opinion. Same model, same tools, same stale CLI, same answer. Independence needs a changed input: a different model, a different source, or a human who knows the system. I was that human, and I had already deferred.
Parallel agents under a hard budget are a lottery. Twenty agents share one window. When it hits, the unfinished ones are gone - not returned, not cached. I bought twenty tickets and got to read three.

The uncomfortable version

I did not lose a week because the model was wrong. Models are wrong sometimes; that is priced in. I lost it because the wrong answer matched my own doubt, and a wrong answer that agrees with you does not feel wrong. The model was not lying. It was thorough about the wrong question, standing on a file I had let go stale, and I let its thoroughness stand in for the check I should have made. What finally broke the loop was one lucky agent and one human question - and I would rather not rely on the luck again.

The one line is committed. One last thing I did check this time, against the botocore changelog rather than a file on my laptop: the Lambda runtime's bundled SDK is newer than the release that added the parameter, so the call will not be rejected client-side. The fix that took a week to find took a few minutes to verify. That ratio is the whole article.

Sources

RestoreDBInstanceFromDBSnapshot - Amazon RDS API Reference - lists BackupRetentionPeriod as a request parameter ("Default: Uses existing setting").
Backup retention period - Amazon RDS User Guide - "During restore operations, you have the option to specify a backup retention period..."
Amazon RDS now supports backup configuration when restoring snapshots - AWS What's New - "Previously, restored database instances inherited backup parameter values from snapshot metadata and could only be modified after restore was complete."
restore-db-instance-from-db-snapshot - AWS CLI Reference - the current CLI accepts --backup-retention-period; an older installed CLI does not, because its API model is frozen at build time.

Windows Copilot Inverted a HIPAA Rule Three Times - and Cited a Real Paper That Proves the Opposite

vadim albarov — Thu, 20 Aug 2026 03:49:01 +0000

I have a two-window habit. My main work happens in a Claude Code session, and I don't like burning its context on side questions - so for quick lookups I alt-tab into Windows Copilot and use it as a search-flavored notepad. Ask, skim, close, back to work. It's been part of my routine for a long time, and honestly, I'd never had a problem with its answers before - side questions came back reasonable, and nothing ever sent me down a wrong path. Low stakes, or so I thought.

This week the side window taught me a HIPAA rule with total confidence. The rule was backwards. Not vague, not incomplete - inverted, in the one direction that would turn a compliance question into a breach.

I build healthcare infrastructure, so I had the reflexes to check. This is the story of what it said, what the regulation actually says, how the wrong answer survived thinking mode, search mode, and a demand for references - and how the citation trail eventually explained where the inversion came from.

The question

I was thinking about deterministic pseudonymization: replacing a patient identifier with hash(key + identifier) so the same patient always maps to the same token and records stay joinable. The obvious follow-up is what HIPAA thinks about the key. Keep it secret? Share it? Does it matter? Full disclosure: the question I actually typed was leading - I'd already run into the claim that sharing the key was the compliant option somewhere, and I asked about it as if it were established.

Copilot's answer, summarized:

If you keep a secret key, you retain the ability to re-identify. Therefore the data is pseudonymized, still PHI, not de-identified.
If the key is public, nobody has privileged knowledge. Nobody can reverse a one-way hash. Therefore the data can qualify as de-identified.
HIPAA, it explained, cares about privileged access, not cryptographic strength. "You're thinking like an engineer. HIPAA is written by lawyers."

It's a seductive answer. It has a philosophy. It flatters you for being confused. It reads like someone explaining a genuinely counterintuitive corner of law.

It is also wrong on both branches, and the second branch is dangerous.

What the regulation actually says

The relevant text is 45 CFR 164.514. Safe Harbor's identifier list ends with a catch-all - "any other unique identifying number, characteristic, or code" - with exactly one exception: a re-identification code that satisfies paragraph (c). Paragraph (c) has two prongs:

Derivation. The code must not be derived from or related to information about the individual, and must not be otherwise translatable back to the individual.
Security. The covered entity must not use or disclose the code for other purposes and must not disclose the mechanism for re-identification.

Read those against the two branches:

Secret-key hash. HMAC(secret, MRN) fails Safe Harbor - but not for Copilot's reason. It fails because the token is mathematically derived from the identifier, and prong 1 prohibits derivation outright, regardless of how well you guard the key. Meanwhile, Copilot's actual claim - "if you can re-identify, it's not de-identified" - contradicts the regulation's text. Paragraph (c) exists precisely so a covered entity can keep a secret re-identification mechanism while the dataset remains de-identified. Retained re-identification capability is not the disqualifier; derivation and disclosure are.

Public-key hash. This one fails everything at once. Publishing the key is literally "disclosing the mechanism for re-identification" - prong 2, verbatim. And it makes the data trivially translatable - prong 1 - because health identifiers live in small, enumerable spaces. SSNs are a 10^9 space. Phone numbers, MRNs, emails, name-plus-birthdate: all enumerable. With the key public, anyone hashes every candidate and matches your entire column in seconds on a laptop. One-wayness protects high-entropy inputs; it does nothing for a nine-digit number.

HHS's own de-identification guidance closes the loop from both sides. It says a hash without a secret key counts as an identifying element exactly because recipients can reverse it over the input space. And it says keyed cryptographic hashes are acceptable under the Expert Determination pathway provided the keys are not disclosed, including to the recipients.

So the real rule is the mirror image of what my sidebar told me:

Construction	HIPAA status
Hash with a published key	Fails everything - discloses the mechanism, trivially reversible
Hash with a secret key	Fails Safe Harbor (derived code); acceptable under Expert Determination with the key undisclosed
Random token + protected mapping table	The pattern 164.514(c) actually blesses - a random value is not "derived from" anyone

Round two: references make it worse

Maybe I asked badly. I turned on thinking mode, then search mode, and asked it to rethink and support its answer with references.

I got the same inverted conclusion, now decorated with emoji section headers, a verdict table, and a source list - and with something worse. The centerpiece was a quoted sentence attributed to HHS, saying a code derived from PHI is an identifier "unless the re-identification key is not retained." I searched for that sentence. Not on HHS.gov, not anywhere I could find. The model composed a plausible-sounding rule that swaps the regulation's actual verb - disclose - for retain, wrapped it in quotation marks, and attached the genuine HHS URL to it.

That's the failure mode that stuck with me: a real link laundering a fake quote. Every reader's citation heuristic - "it links to hhs.gov, so it's grounded" - defeated by construction.

The funny part is that the correct answer was present in the same response, scattered in the margins. The risks section admitted that a public pepper can be brute-forced over common identifiers - which quietly destroys the headline claim that public-key hashing is irreversible. Another bullet correctly noted that keeping a secret key for linkage requires expert determination - which is the actual rule, and refutes the answer's own bottom line. The model had all the pieces and still shipped the inverted conclusion in the headline, the table, and the summary.

Round three: it's not me

One hypothesis left: maybe my framing poisoned the well, since my first question presented the counterintuitive claim as a finding. So I opened a fresh chat and asked the neutral, symmetric question - which is better for HIPAA, hash with a private key or a public key - with no premise embedded.

Verdict, verbatim in spirit: public-key hashing is "far better," private-key hashing is still PHI, and - my favorite line - the public pepper should be "long, random, and not guessable."

A public value that is not guessable. It's published. That single phrase is the whole confusion in miniature: the answer needs the key to be simultaneously known to everyone (so nobody has privileged access) and known to no one (so nobody can brute-force). For calibration, I asked ChatGPT's web version the same neutral question. It got it essentially right: public-key hashing rejected for exactly the dictionary-attack reason, keyed HMAC labeled as pseudonymization that doesn't exit PHI obligations by itself, random tokens with a secured mapping table recommended for real de-identification. Not perfect - it never cited the derivation prohibition - but directionally sound everywhere it committed.

Same question. One product inverted, reproducibly, across independent chats. The other didn't.

The citation trail explains everything

Both of the reference-backed Copilot answers - rounds two and three - leaned on the same academic source: a 2003 AMIA paper by Landi and Rao, "Secure De-identification and Re-identification". I looked it up.

The paper describes an asymmetric encryption scheme: encrypt patient identifiers with a public key so that only the holder of the matching private key - the data owner - can decrypt and re-identify. "Public key" as in public-key cryptography. One half of a keypair. A system whose entire security rests on the private key staying secret, and whose explicit purpose is to let the owner retain re-identification capability.

Now the most plausible failure chain is visible. Retrieval surfaced a paper with "public key" and "de-identification" in close proximity. Summarization collapsed public-key cryptography into publicly known key. And then - this is the part I find genuinely instructive - the model didn't just misread a term. It constructed an entire regulatory philosophy around the misreading: the "privileged knowledge" theory of HIPAA, delivered with the confidence of a law professor, appearing in no regulation, contradicted by the very paper being cited. The citation that was supposed to ground the answer was proof of the opposite.

Which model was that, even?

The chat window said "Smart." That's a mode label, not a model name. So I asked Copilot directly which model it runs on. It denied running on any outside lab's models at all - just "proprietary Windows AI technologies" - and said it cannot disclose specifics.

That answer is worth exactly nothing, and knowing why it's worth nothing is the useful part. A chatbot's claim about its own identity is generated text like everything else it says - models have no introspective access to the infrastructure serving them, and "I can't disclose" deflections are typically system-prompt policy, not knowledge. The same product that invented an HHS quote is not a reliable witness about its own internals.

The public reporting says something different and messier: modern assistant products run mixed fleets - frontier models licensed from partner labs alongside the vendor's own in-house models - with a cost-driven router deciding invisibly, per query, which one you get. The lineup shifts between announcements, the routing shifts without any announcement at all, and there's no per-response indicator. The same question may be served by a different model next week, or later today.

I don't know whether my three inverted answers came from a fast-path model, from the summarization layer garbling search results, or from the persona tuning that opens responses with "let me say this clearly and directly." That's the point: as a user, I can't know - and asking the product just adds one more unverifiable claim to the pile. When another assistant answers the same question correctly, the difference isn't necessarily raw model capability. It's everything wrapped around the model, and the wrapper is invisible. "Which LLM does this product use" turns out to be the wrong question. The right one is "which pipeline, with which retrieval, which router, and which incentives" - and no consumer product answers it. Not even when you ask it directly.

What I'm taking away

Confidence formatting is free. Tables, verdict emoji, "let me say this clearly" - none of it correlates with correctness. The most wrong answer in this story was the best-formatted one.
"Support this with references" does not mean "verify this." Once a model has committed to a conclusion, asking for sources can produce justification instead - up to and including an invented quote pinned to a real government URL. If a quote matters, search for the exact sentence.
Watch for internal contradictions. The wrong answers refuted themselves in their own risk sections. A response whose caveats disagree with its headline is telling you which part was retrieved and which part was composed.
For anything regulatory, read the primary source. 164.514(c) is two paragraphs. It cost five minutes and settled in one reading what three AI answers scrambled. Compliance-by-chatbot is how a "de-identified" dataset ships with a published key and becomes a reportable breach.
The side window deserves the same skepticism as the main one. My mistake wasn't using Copilot - it was granting the quick-lookup window a lower evidence bar because the questions felt small. Nothing about the window makes the answers smaller.

I'm keeping the two-window setup, but the notepad seat is now vacant. Either something changed in that product - a router, a model swap, a summarization layer - or it could always do this and I simply never caught it; I have no way to tell, and that uncertainty is its own verdict. Either way, a tool I trusted for months just fabricated regulatory quotes with a straight face. So I'm dropping it until it stabilizes, and I'll know it has stabilized the same way I learned it broke: by spot-checking its answers against primary sources. A quick-lookup tool that requires verification of every answer isn't a quick-lookup tool anymore.

Try it yourself

Here is the prompt I used for the fresh-chat test, lightly tidied. Run it against your assistant of choice:

In terms of HIPAA compliance, what is better for de-identifying PHI data: HASH(private_key + PHI) or HASH(public_key + PHI)?

The correct answer rejects both as de-identification on their own, flags the public-key variant as trivially reversible by dictionary attack, and mentions that a keyed hash only works under the Expert Determination pathway with the key kept undisclosed. Anything that tells you the public key is the compliant option has inverted 45 CFR 164.514(c).

Share what you get in the comments - which product, which mode, and which way it landed. I'm genuinely curious whether this reproduces beyond my machine, and a comment thread of timestamped outputs is a better dataset.

Disclaimer: I'm an infrastructure engineer, not a lawyer; this is an engineering read of published regulations and guidance, not legal advice. The Copilot and ChatGPT responses summarized here were collected in August 2026 from consumer versions of both products; model routing and behavior change constantly, which is rather the point.

The De-identification Datasets Problem: i2b2/n2c2 Access Is Broken, and What to Do Instead

vadim albarov — Wed, 19 Aug 2026 03:27:41 +0000

TL;DR: I'm building a PHI de-identification tool, and step one of doing that honestly is benchmarking it against the standard corpora: the i2b2 2006 and 2014 de-identification challenge datasets, now distributed as "n2c2" through Harvard DBMI's data portal. Step one failed. The portal's n2c2 page has said "Temporarily Unavailable" since at least 2026-07-28, registration is closed, and the old i2b2.org dataset page returns HTTP 500. Even when the door is open, access means per-user registration, a data use agreement, and an approval wait - which is why published de-id numbers are so hard to reproduce and why the field's canonical scores are 12-20 years old. I built a synthetic corpus generator instead. Here's the whole investigation, dated, plus what synthetic data honestly can and cannot replace.

Step one: you can't get the data

Benchmarking a de-identification tool sounds like the easy part of the project. The hard part is supposed to be the NLP: finding every patient name, date, medical record number, and phone number buried in messy clinical prose, so the notes can be used for research without exposing anyone. Detection is the science. Evaluation is just downloading the test set, right?

Here is how evaluation actually went for me.

Every de-identification paper of the last two decades benchmarks against the same corpora: the i2b2 de-identification challenge datasets from 2006 and 2014. If you want your numbers to mean anything to anyone, you report them on i2b2 2014. So on 2026-07-28 I went to get the data from the Harvard DBMI Data Portal, where the corpora live under their post-2018 name, n2c2 (National NLP Clinical Challenges). What I found was a notice: the n2c2 datasets are temporarily unavailable.

Fine, I thought. Maintenance happens. I built other things and came back.

As of today, 2026-08-18, nearly three weeks later, here is the exact state of the front door, checked from a fresh session:

The portal itself is up and loads fine.
The portal's data sets listing shows two entries. Neither is an n2c2 NLP corpus (both are 4CE COVID-related sets).
The n2c2 NLP Research Data Sets project page - the page for the actual challenge corpora, 2006 through 2018 - carries this banner, verbatim: "Temporarily Unavailable. The n2c2 datasets are temporarily unavailable. If you are trying to access data from the 2019 Challenge, tracks 1 (Clinical Semantic Textual Similarity) and 2 (Family History Extraction) are available directly through Mayo Clinic." Below the dataset descriptions: "Registration is not open ... at this time."
The legacy home of these corpora, i2b2.org/NLP/DataSets, returns HTTP 500.
The companion n2c2 informational site refused my scripted requests outright with an Akamai "Access Denied" page. That one I'll hedge: it may just be bot filtering rather than downtime. But it means I cannot even verify the documentation programmatically.

No ETA. No status page. No mirror. The canonical benchmark for an entire subfield of clinical NLP is a "temporarily unavailable" banner, and "temporarily" has meant at least July 28 through August 18 so far. It may come back tomorrow; the point of this article survives either way, because the outage is only the loudest symptom of a structural problem.

What these datasets are, and why everyone needs them

Some history, because the names are confusing. i2b2 (Informatics for Integrating Biology and the Bedside) was an NIH-funded center based at Partners HealthCare that, starting in 2006, ran annual clinical NLP shared tasks on real (de-identified) hospital notes. In 2018 the challenge series was renamed n2c2, and stewardship of the datasets moved to the Department of Biomedical Informatics at Harvard Medical School, distributed via their portal. Same corpora, three names, one door.

Two of those challenges define de-identification evaluation to this day:

The 2006 challenge (Uzuner, Luo, and Szolovits, JAMIA 2007) used hospital discharge summaries in which the authentic PHI had been replaced with synthesized surrogates - including deliberately out-of-vocabulary, made-up names to punish systems that just memorized name lists. Seven teams, sixteen system runs, and the best systems scored above 98% F-measure across PHI categories.

The 2014 i2b2/UTHealth challenge (Stubbs and Uzuner, JBI 2015) raised the bar: 1,304 longitudinal records covering 296 patients, over 28,000 annotated PHI instances, annotated under a broad interpretation of HIPAA. Human annotators managed a token-level F1 of 0.927 against the gold standard; the best automated system hit a strict micro-averaged F1 of 0.936.

Notice something about both: even the "real" gold standards contain synthetic PHI. The notes are genuine clinical text, but the identifiers in them are surrogates, inserted so the data could be released at all. Keep that in mind for later - the field's ground truth has always been real prose plus fake identifiers.

That 0.936 from 2014 is, functionally, still the number. When you publish a de-id tool in 2026, reviewers ask how you compare on i2b2 2014. The corpus is twelve years old, drawn from one hospital system, and pre-dates most of what modern EHRs do to note formatting. It is also, at the moment, undownloadable.

The access process, even on a good day

Suppose the portal comes back tomorrow. What does access look like then? Per the archived instructions on i2b2.org, the datasets are "freely available" to researchers - subject to a data use agreement, and "each individual user must access the data independently" through the portal. In practice:

Register an account on the DBMI portal.
Sign the Rules of Conduct and the Data Use Agreement (the n2c2 DUAs are real legal documents, not click-through checkboxes).
Wait for a human to approve you. There is no published turnaround time.
Repeat for every individual on your team, because the DUA is per-person, not per-lab.
Never redistribute the data - which also means never shipping it as a test fixture, never putting it in CI, never publishing your evaluation harness with the inputs included.

This is not villainy. The data is real patient prose; a DUA is the legally and ethically appropriate wrapper, and the people who built and maintain these corpora did the field an enormous service. But note the architecture: the standard benchmark for a global research area is administered by one team at one institution, through one portal, with per-user paperwork and no fallback. When that single point of distribution goes down - for maintenance, for a compliance review, for a staffing gap, for whatever is happening right now - the benchmark simply ceases to exist for anyone who doesn't already have a copy.

Why this hurts more than my project

Play the incentives forward and the damage compounds:

Published results become unreproducible in practice. A paper says "0.94 F1 on i2b2 2014." You cannot check that claim, cannot run the same test set through your own tool, cannot even eyeball the annotation decisions the score depends on. Reproducibility in de-id research is gated on a portal login, and today on a portal banner.

New tools cannot compare against prior art. The literature has twenty years of numbers on these corpora. A new open-source tool that cannot access them either skips comparison (and gets dismissed) or quotes other papers' numbers against its own results on different data (and misleads).

The canonical numbers fossilize. Because making a new shareable gold standard from real notes is brutally expensive - the 2014 corpus took double annotation, arbitration, and multiple proofreading rounds to reach that 0.927 human F1 - nobody replaces the old benchmarks. The field's reference points are frozen in 2006 and 2014 while clinical documentation, and the models reading it, changed completely.

Insiders and outsiders diverge. Groups with long-standing access or local hospital data keep publishing; independent developers and open-source maintainers evaluate on whatever they can scrape together. The people most likely to ship a de-id tool you can actually download are the least able to prove it works.

The alternatives tour

Before building anything, I did the diligence on every other door:

PhysioNet. The MIMIC family of ICU databases includes clinical notes, and PhysioNet also hosts a gold standard corpus of 2,434 de-identified nursing notes with realistic surrogate PHI (Neamatullah et al., 2008). Access requires becoming a credentialed user: identity verification, the CITI "Data or Specimens Only Research" training course, and then a separate DUA per dataset. To PhysioNet's credit, this process is documented, predictable, and actually functioning. But it is still weeks of process, still per-person, still non-redistributable. And MIMIC's notes are already de-identified with placeholders, so to evaluate a de-id tool you must first re-inject surrogate PHI - which is exactly what prior work on synthetically-identified MIMIC notes does. The "real data" path quietly becomes a synthetic-PHI path anyway.

MTSamples. MTSamples.com hosts thousands of publicly available transcribed sample medical reports across dozens of specialties. No registration, no DUA, real clinical language structure. The catch: they are samples, so they contain no PHI to find. To make a de-id benchmark out of them you inject fake PHI into the text - an approach with an established lineage in the literature. This is the honest public option, and it is the spirit my workaround follows.

Synthea. Synthea generates fully synthetic patients with medically plausible histories - fantastic for structured FHIR data. But its narrative output is template-driven fill-in-the-blank SOAP text linked to the structured record. For de-id evaluation, where the entire game is the messiness of real prose - copy-paste artifacts, headers, inconsistent formatting, abbreviations - templated narrative is the wrong distribution by design.

Newer synthetic clinical text efforts. This space is heating up: Synthetic4Health generates annotated synthetic clinical letters; ASQ-PHI proposes an adversarial synthetic benchmark specifically for de-identification; and there is active work on whether LLM-generated notes actually match real note distributions (early answer: imperfectly, and you should measure the gap rather than assume it away). None of these is yet a community-standard replacement for i2b2 2014. All of them are bets on the same thesis I ended up betting on.

The workaround: a synthetic corpus generator

So I built my own synthetic corpus generator for the PHI de-identification tool I'm building. The design is deliberately boring:

Templates styled after real clinical note genres - discharge summaries, progress notes, radiology reports - with the section structure, boilerplate, and formatting quirks those genres actually have.
Controlled PHI injection. Fake names, MRNs, dates, phone numbers, addresses, providers, and facilities generated and inserted at known offsets. Every injected entity is recorded with its exact span and category at generation time.
Category balancing. Real corpora are dominated by dates and names; rare categories (fax numbers, device IDs, URLs) barely appear. A generator can oversample the rare stuff so your recall numbers on those categories are backed by hundreds of instances instead of four.

The advantages are structural, not incidental:

Labeled by construction. No annotation budget, no inter-annotator disagreement, no 0.927 ceiling on ground-truth quality. The generator knows where the PHI is because it put it there.
Shareable. No DUA, because there is no patient. The corpus - and more importantly the generator - can live in a public repo, run in CI, and ship as test fixtures. Anyone can regenerate the exact evaluation set from the code and a seed. That is a property the i2b2 corpora can never have.
Scalable and steerable. Need 10,000 more notes with dates in weird formats? That's a parameter, not a grant application.

The honest limits

Here is where I'm obligated to argue against myself, because synthetic evaluation has failure modes that will flatter your tool if you let them:

Distribution shift. My templates are styled after clinical notes; they are not clinical notes. Real notes contain dictation artifacts, OCR junk, mid-sentence copy-paste, and formatting chaos that no template library fully reproduces. A recall number earned on synthetic text is an upper bound, not an estimate, of real-world recall. There is emerging work on quantifying exactly this synthetic-to-real gap for PHI taggers, and the gap is real.

Overfitting to your own generator. This is the insidious one. If the same mental model writes both the fake-name generator and the name-detection logic, the benchmark and the tool share assumptions, and you are grading your own homework. Mitigations exist - independent sources for injection values, formats the detector wasn't designed around, adversarial edge cases - but the risk never reaches zero.

It complements real-data validation; it does not replace it. My position after all this: synthetic corpora are for development, regression testing, category-level diagnostics, and public reproducibility. Before anyone trusts a de-id tool with actual patient data, it needs validation on real notes under proper governance - PhysioNet credentialing, an institutional dataset, or the n2c2 corpora if the door ever reopens. Remember, though, that even those gold standards are real prose with surrogate identifiers. The line between "real benchmark" and "synthetic benchmark" was always a spectrum, not a wall.

What better infrastructure would look like

None of this requires new science. It requires plumbing:

A status page and a mirror. If a dataset is the reference benchmark for a field, "temporarily unavailable" with no ETA on a single portal should be impossible. PhysioNet already demonstrates the model: documented process, predictable credentialing, many datasets under one durable roof.
Standing distribution instead of single-lab stewardship. Move canonical corpora to infrastructure whose job is distribution, with credentialing handled once per user, not per corpus per portal.
Publish generators, not just corpora. A community-maintained synthetic benchmark - generator code plus seeds, calibrated against real data - would give the field something no DUA can: an evaluation anyone can run, extend, and verify. The recent synthetic-benchmark papers are steps in this direction; what's missing is convergence on one that leaderboards accept.
Two-track evaluation as the norm. Report on the gated real corpus for comparability, and on an open synthetic corpus for reproducibility. Either number alone is half a result.

Takeaways

Check the data before you plan the benchmark. The canonical de-id corpora are behind a per-user DUA on one portal, and that portal's n2c2 datasets have been "temporarily unavailable" from at least 2026-07-28 through 2026-08-18, with the legacy i2b2.org page throwing HTTP 500. Verify the current state yourself; date what you find.
Single-institution gatekeeping of a field's benchmark is a reliability bug, independent of any outage. Per-user DUAs also mean no CI, no fixtures, no redistribution - reproducibility is structurally capped.
The field's reference numbers are from 2006 and 2014. Best strict F1 of 0.936 on 1,304 notes from one hospital system is still the bar new tools are measured against.
PhysioNet is the functioning real-data path - credentialing, CITI training, per-dataset DUA - but its notes need surrogate PHI re-injection for de-id evaluation anyway.
Synthetic corpora buy you labels by construction, shareability, and category balance. They cost you distribution realism, and they tempt you into grading your own homework. Use them for development and public reproducibility; validate on real data before production.
The gold standards were already part synthetic. Real notes, surrogate PHI. Synthetic evaluation isn't a betrayal of rigor - unexamined synthetic evaluation is.

If you work in clinical NLP: how did you get your benchmark data, and how long did access take? And if anyone has current information on when the n2c2 datasets are coming back, the comments are open.

Is 'HIPAA-Eligible' the Same as 'HIPAA-Compliant'? I Audited AWS's List Against the Security Rule

vadim albarov — Tue, 18 Aug 2026 03:53:27 +0000

Here is a sentence I have heard, in some form, from three different engineers and one vendor sales deck: "We're fine, we only use HIPAA-eligible AWS services."

I build healthcare infrastructure for a living, and every time I hear it I do the same mental translation: "We're fine, our data center signed a contract." It's true, it's necessary, and it answers roughly none of the questions a HIPAA auditor will ask you.

So I decided to make the gap measurable. I took six AWS services that show up in practically every PHI-handling stack - S3, RDS for PostgreSQL, ECS on Fargate, KMS, the CloudTrail/CloudWatch audit pair, and the ALB - and audited each against the Security Rule's technical safeguards (45 CFR 164.312): what the regulation demands, what the service does by default, and what you must configure yourself.

Spoiler: of the 35 concrete controls I mapped, the defaults covered 6.

What "HIPAA-eligible" actually means

Start with what AWS itself says. The HIPAA-eligible services reference defines eligible services as those that may "create, receive, process, maintain, or transmit" electronic protected health information (ePHI) - provided you have signed AWS's Business Associate Addendum (BAA) first. And then, in plain sight, the sentence everyone skips: "Customers still must configure these services consistent with HIPAA requirements."

The fast primer:

A covered entity (provider, health plan, clearinghouse) or a business associate (anyone handling PHI for them - your SaaS included, if hospitals are your customers) must comply with HIPAA. AWS becomes your business associate when you sign the BAA - today a self-service click in AWS Artifact.
The BAA covers AWS's side of the shared responsibility model: data centers, hypervisors, service internals - security of the cloud. Everything you configure - security in the cloud - stays yours.
The Security Rule is the part of HIPAA that talks to engineers. Its technical safeguards (164.312) name five standards: access control, audit controls, integrity, person or entity authentication, and transmission security.
Implementation specifications are required or addressable. Addressable does not mean optional - it means implement it, implement a documented equivalent, or document a defensible reason you did neither. Encryption at rest and in transit are both addressable, and every healthcare shop I know treats them as required: "we decided not to encrypt the PHI" is not a paragraph anyone wants to defend in a breach investigation.
And the kicker, straight from AWS's own HIPAA compliance page: "There is no HIPAA certification for a cloud service provider (CSP) such as AWS." There is no certification for you either. Only your configuration, your documentation, and eventually someone's audit.

So "HIPAA-eligible" means exactly one thing: the service is covered by AWS's BAA, so AWS's side of the split is contractually handled. Whether your side is handled is the rest of this article.

Last year I distilled a few years of building this into a set of internal Terraform modules for my team - one hardened block per service, each control traced to its Security Rule citation. The most useful artifact of that project was the diff between "what the module enforces" and "what AWS gave us out of the box." That diff is essentially this article.

The audit

Per service: what 164.312 asks, what you get by default, what you must add. Every default claim is sourced; where a default changed recently, I say when.

S3 - the poster child of improved-but-insufficient

Credit where due: S3's defaults have genuinely improved. Since January 2023, every new object is encrypted with SSE-S3, and you cannot turn that off. Since April 2023, new buckets get all four Block Public Access settings enabled and ACLs disabled. Two real 164.312(a) wins, for free. Now the rest:

Transmission security (164.312(e), required standard): S3 happily serves plaintext HTTP unless you attach a bucket policy denying aws:SecureTransport = false. Nothing does this for you.
Integrity (164.312(c)): versioning - your recovery path when an object is improperly altered or deleted - is off by default.
Audit controls (164.312(b)): server access logging is off by default, and object-level API logging (who read which object) doesn't come from S3 at all - it needs CloudTrail data events, also off by default (see CloudTrail below).
Encryption, but accountably: SSE-S3 encrypts with a key you can't scope, rotate on your terms, or audit the use of. For PHI you want SSE-KMS with a customer-managed key, plus a policy denying uploads that request the wrong key - so a misconfigured client fails loudly instead of writing PHI under the wrong crypto.

One non-obvious lesson from building the S3 module: keep patient identifiers out of object keys. Keys leak into access logs, CloudTrail events, error messages, and URLs - surfaces your PHI inventory forgot about.

RDS for PostgreSQL - encryption is opt-in, still, in 2026

Encryption at rest (164.312(a)(2)(iv)): storage encryption is not enabled by default at the API level - StorageEncrypted defaults to false - and can only be set at creation time. Forget it in your Terraform, and the fix is a snapshot-copy-restore migration, not a flag flip.
Transmission security (164.312(e)): a moving target. For RDS PostgreSQL 15 and later, rds.force_ssl defaults to 1; for 14 and older it defaults to 0. It also lives in a parameter group, one console edit away from silently becoming 0 again. In my modules it's pinned at the engine, so a client that forgets sslmode=require gets an error instead of a plaintext session.
Access control (164.312(a)(1)): nothing stops you from making the instance publicly accessible or dropping it in a public subnet. Private placement, security-group-to-security-group ingress (no CIDR allowlists - identities beat address ranges), and publicly_accessible = false are all decisions you must make and enforce.
Integrity (164.312(c)): deletion protection is disabled by default in the API and IaC (the console pre-checks it for production templates - your safety net depends on which tool created the database).
Authentication (164.312(d)) and audit (164.312(b)): RDS can generate and rotate the master password in Secrets Manager so it never touches Terraform state; it can export connection logs to CloudWatch. Both are opt-in. By default the master password is wherever you put it, and connection history is a shrug.

ECS on Fargate - the good defaults are the invisible ones

Fargate's best HIPAA property doesn't appear in any console checkbox: there are no instances. No AMIs to patch, no SSH, no node agents - a whole slab of the malicious-software surface moves to AWS's side of the BAA. That and IAM's deny-by-default (a fresh task role can do nothing) are real defaults in your favor. Everything else is on you:

Access control (164.312(a)(1)): nothing prevents tasks in public subnets with public IPs. Private placement is a choice you codify.
Authentication (164.312(d)): plain environment variables appear in DescribeTaskDefinition output and the console. Database credentials must go through the task definition's secrets integration with Secrets Manager - injected at container start, never written down.
Unique identity (164.312(a)(2)(i)): the execution role (ECS pulling images, injecting secrets) and the task role (your app) are separate identities - if you bother creating them separately instead of reusing one fat role.
Audit controls (164.312(b)): my favorite gap in this audit. ECS Exec - an interactive shell inside a container that may hold PHI in memory - records, by default, only the ExecuteCommand API call in CloudTrail. What happened inside the session goes unrecorded unless you configure session I/O logging to an encrypted log group. A shell in a PHI container is an access event; out of the box, it's one with no transcript.

KMS - the service that only helps if you invite it

KMS is HIPAA-eligible, excellent, and entirely opt-in. Every encryption integration on this page uses KMS if you wire it in; the eligible list has no opinion about whether you did. Even once you create customer-managed keys:

Automatic key rotation is disabled by default for customer-managed keys (AWS-managed keys rotate yearly; the keys you control don't, until you say so).
The default key policy hands the account root full access and calls it a day. Separating key administration (policy, rotation, deletion - no decrypt) from key use (decrypt, no administration) is a policy you write yourself. Worth writing: no single identity can then both reconfigure the crypto and read the data - exactly what 164.312(a)(1) wants from an access control story.
One default that genuinely protects you: key deletion requires a 7-30 day waiting period. Deleting a key that ever encrypted PHI destroys the data with it, so in my modules the window is pinned to the 30-day maximum and ScheduleKeyDeletion events page a human.

One design decision I'd defend anywhere: separate keys per purpose - data, logs, backups. Revoking access to PHI data should never break log delivery, and CloudTrail can hold the logs key without gaining any path to the data key.

CloudTrail and CloudWatch - the audit trail you think you have

164.312(b) is a required standard with no addressable escape hatch: "implement... mechanisms that record and examine activity in information systems that contain or use ePHI." What you have by default is 90 days of management-event history in the console. That's it. Specifically:

No durable trail. For records past 90 days you must create a trail delivering to S3. HIPAA's documentation retention requirement (164.316(b)(2)(i)) is six years; most programs apply the same horizon to audit artifacts. 90 days is not in the same universe.
Data events are not logged by default. Management events tell you someone changed a bucket policy; they do not tell you someone downloaded 10,000 patient records via GetObject. Object-level audit is a separate, per-event-billed opt-in - in my modules it's a flag with a cost warning, but a flag someone consciously declines, not a silence nobody noticed.
Log file validation (the SHA-256 digest chain that makes trail tampering detectable - the 164.312(c) integrity story for the audit trail itself) is a setting, not a given.
On the CloudWatch side: log groups never expire by default - retention is a policy you set deliberately - and while log data is SSE-encrypted at rest, customer-managed KMS encryption is optional, attached per log group.
And the quiet half of 164.312(b): "record and examine." Metric filters, alarms, anything that turns the trail into a signal a human reviews - none of it exists until you build it. An unread audit trail satisfies nothing.

ALB - TLS quality depends on which tool you clicked

The load balancer is your transmission security (164.312(e)) chokepoint, with the most path-dependent defaults of the lot:

Create an HTTPS listener in the console today and you get a modern TLS 1.3/1.2 policy. Create it through the CLI or CloudFormation and the default is ELBSecurityPolicy-2016-08, which still accepts TLS 1.0 and 1.1. Your cipher floor depends on which tool provisioned the listener. Pin the policy explicitly and this stops being interesting.
Nothing forces HTTPS to exist at all. A port-80 listener forwarding plaintext is perfectly deployable; the 80-to-443 redirect is a choice.
Access logs are disabled by default - there goes 164.312(b) at the edge until you wire up the log bucket.
Deletion protection: off by default. Deleting the front door of a PHI service should take two steps.
And one my team debated for a week: by default TLS terminates at the ALB, and the hop to your tasks is HTTP inside the private network. A common, defensible posture - but a documented risk decision your analysis has to own, not something the eligible list settled for you.

The scoreboard

Counting controls involves judgment calls, so here's mine - each row counts the specific 164.312-relevant controls discussed above:

Service	Controls mapped	Covered by default	You configure
S3	7	2 (SSE-S3 since 2023, Block Public Access since 2023)	5
RDS Postgres	8	1 (forced TLS, PG15+ only)	7
ECS Fargate	6	2 (no node surface, IAM deny-by-default)	4
KMS	3	1 (deletion waiting period)	2
CloudTrail / CloudWatch	6	0 (90-day history is partial credit at best)	6
ALB	5	0	5
Total	35	6	29

Roughly one control in six arrives configured. The other 29 are your job, on services that are all on the HIPAA-eligible list. And notice which six: the freebies cluster in encryption at rest, where AWS has spent a decade raising defaults. Audit controls and transmission security - both required standards - score near zero.

The floor is genuinely rising: S3 in 2023, PostgreSQL 15's force_ssl, console TLS policies. But that's also the trap - your actual defaults depend on the year the feature shipped, the engine version, and whether the resource was born in the console or in CloudFormation. A compliance posture made of remembered defaults is a compliance posture made of trivia.

What to do about it: treat the list as a floor

The eligible list answers one question: may PHI touch this service under our BAA? Everything after that yes is configuration, and configuration you rely on should be code:

Codify the invariants, and don't make them variables. The rule that survived every review of our modules: if the Security Rule requires it, there is no toggle. Storage encryption, TLS enforcement, log validation, private placement - hardcoded. Variables exist only for things a compliant deployment may legitimately vary, like retention above the floor. A toggle someone can forget is a finding someone will write.
Let policy-as-code catch the drift. checkov and tflint in CI flag most of the gaps in this article - unencrypted RDS, missing bucket policies, permissive TLS policies - before they exist. Native terraform test can pin the invariants so a refactor can't quietly reintroduce a toggle. Where you do deviate, suppress with a written justification inline; that comment is your addressable-specification documentation.
Map controls to citations, in the repo. Every module of ours carries a table: citation, what the code enforces, and - the column that earns its keep - what remains yours. No amount of Terraform covers the administrative safeguards: risk analysis, a log review procedure someone actually follows, restore testing, access reviews, BAAs with every other PHI-touching vendor. The code can only make the technical floor solid enough for your people to stand on.

"HIPAA-eligible" was never a lie. It's a contract term that got promoted, somewhere between the sales deck and the standup, into a security property. The list tells you where PHI may go. The Security Rule tells you what must be true when it gets there. The distance between the two is not covered by anyone's BAA - it's covered by you, ideally in version control.

Disclaimer: I'm an infrastructure engineer, not a lawyer; this is engineering analysis of published regulations and AWS documentation, not legal advice. Defaults cited were verified in August 2026 and do change (sometimes for the better). Run the details past your compliance officer, who will find at least one thing here that your specific situation makes wrong.

What's the widest eligible-vs-compliant gap you've hit in the wild? I collect these.

"Oops, I Forgot to Tell You That's Dangerous": Claude Code Watched Me Wipe Production Redis - Then Helped Carve It Back Off the Disk"

vadim albarov — Wed, 12 Aug 2026 03:47:47 +0000

TL;DR: I enabled Redis AOF persistence by editing redis.conf and restarting the service. On the test environment it worked fine. On production, all three nodes came back with zero keys, and the empty dataset overwrote dump.rdb. The missed step: you must enable AOF on the running instance with CONFIG SET appendonly yes before putting it in the config file. We got the data back by carving the unlinked RDB file out of raw disk blocks with dd. Oh, and the AI assistant that helped me plan the change knew about this trap the whole time - it just didn't mention it until after the wipe. Here's the full story.

Some context first

The stage for this story is a legacy healthcare project that's been running in production for years. Redis sits at the center of it, wearing two hats: it's the cache in front of SQL Server, and it's also the feature flag storage - the backend seeds flags into Redis at startup, and both backend and client read them from there at runtime. Over the years the flag storage format evolved (more on that in a minute), the cache quietly accumulated datasets that exist only in Redis, and the Redis setup itself - a bare tarball install with an untouched config - predates everyone's memory of who set it up. In other words: exactly the kind of system where nobody looks at the persistence settings until something forces them to.

Something did.

It started with a feature flag that "flipped itself"

After we published a new release, one of our feature flags apparently turned itself on. Nobody had touched it. The flag had existed for months with a default of false, and suddenly the client started behaving as if it were true.

I diffed the two release tags first: zero changes to flag definitions or defaults. So I went to where the flags actually live in Redis - and found the flag stored in two places, disagreeing with each other:

redis-cli GET  FeatureFlag:CrossWindowCommands   → "true"    # legacy string key
redis-cli HGET FeatureFlag CrossWindowCommands   → "false"   # current hash field

The current implementation stores flags as fields in a FeatureFlag hash, but there was also a stale legacy string key left over from the old storage format. The false in the hash, it turned out, wasn't the original state at all - a teammate had dug into the same problem before me and manually set it back to false. Which made the mismatch the real hint: the legacy key still held true, untouched, and the values in Redis always win - the startup migration copies legacy keys into the hash with HSETNX, and the config default only applies when the field doesn't exist yet.

So the explanation was less dramatic than a flag flipping itself: the flag had been true in Redis all along. It just didn't matter, because for months no code path ever read it. Then the new client release shipped code that actually did something with the flag, and a value that had been harmlessly wrong the whole time suddenly had teeth. An unused flag quietly became a used one, and it looked like a spontaneous flip.

The real finding: our persistence was a time bomb

While investigating, I checked how the Redis instance itself was configured. It's a redis-stack tarball install running natively on RHEL under a custom systemd unit - no Docker, no operator, one master and two replicas with Sentinel. The persistence settings were all defaults - nothing configured at all:

appendonly no                    # default
save 3600 1 300 100 60 10000     # default snapshot thresholds

At our write volume, that works out to roughly one snapshot an hour. So every restart could silently roll the dataset back by up to an hour - and worse, a rollback could resurrect old values (like stale legacy flag keys) that had been deleted since the last snapshot. That's exactly the kind of environment where flags "change themselves" and nobody can explain why.

So we made a decision: park the whodunit, fix the root cause. Enable AOF (append-only file). With aof-timestamp-enabled yes you even get a timestamped log of every write command - a free audit trail for exactly this class of mystery:

appendonly yes
appendfsync everysec
aof-timestamp-enabled yes
auto-aof-rewrite-min-size 64mb
auto-aof-rewrite-percentage 100

(Honest caveat: it's a what-and-when audit, not a who - the AOF doesn't record which client or user issued a command. Since this instance effectively has a single user anyway, that was good enough for a start.)

Testing: everything green

I tested the change end to end. First in a throwaway Docker container: enabled AOF, wrote some flags, and confirmed the AOF captured every command with #TS: timestamps. I even benchmarked it: 100,000-op redis-benchmark runs against two identical containers came back at ~181k SET/s with AOF versus ~142k without. Yes, the AOF run scored higher - which tells you the difference is pure run-to-run noise. With appendfsync everysec, the write-throughput cost is unmeasurable.

Then on the test environment cluster. I enabled it live on the running instance:

redis-cli CONFIG SET appendonly yes
redis-cli CONFIG SET aof-timestamp-enabled yes

I did that live purely to check the concept: right after flipping it I changed a couple of values and watched them show up in the AOF file. A quick sanity check, nothing more - or so I thought. I had no idea this throwaway "test" step was the one doing the heavy lifting.

I then added the same settings to redis.conf so they'd survive a restart, rebooted the nodes one by one, and verified. Everything held: aof_enabled:1, data intact, dummy writes visible in appendonlydir/*.incr.aof, timestamps and all.

The test environment was perfect.

Production: 0 keys

On production I did what felt like the same change: edited redis.conf on all three nodes, added appendonly yes, stopped the sentinels (deliberately, so no surprise failovers mid-maintenance), and rebooted the nodes one by one.

Why full reboots instead of just restarting the Redis service? Two reasons that felt responsible at the time. All three servers had quietly accumulated 500+ pending OS updates, including some severe security patches - so since I was already in a maintenance window, it seemed silly not to apply them and reboot in one go. And a real restart was part of the plan anyway: the whole point of putting appendonly yes into the config file was to prove the setting survives a node going down, so I wanted to see it hold through a genuine reboot.

Then:

redis-cli DBSIZE
(integer) 0

Empty. Master and both replicas. And dump.rdb on disk? Also empty - about 100 bytes of RDB header and nothing else. Roughly 290,000 production keys, gone: cache, feature flags, and one Redis-only ID-mapping dataset that has no SQL fallback at all.

The missed step

Here's the trap, and if you take one thing from this article, take this:

When Redis 7 starts with appendonly yes, it loads the dataset from the AOF - and ignores dump.rdb.

On the test environment I had run CONFIG SET appendonly yes on the live instance first. That triggers an AOF rewrite that builds the AOF base file from the data currently in memory. By the time the test environment restarted, a fully populated AOF existed on disk, and Redis restored from it.

On production I skipped straight to the config file. So the first startup with appendonly yes found no AOF containing the dataset, initialized an empty one, and came up blank. From there the damage compounds automatically:

Redis starts empty and creates an empty AOF base.
The next background save writes the empty dataset over dump.rdb - and Redis replaces the file via write-temp-then-rename(), so the original file's inode is unlinked.
Replicas reconnect and do a full resync from the empty master, wiping themselves too.

Three nodes of redundancy, all faithfully replicating nothing. (Replication protects you from machine failure. It does not protect you from yourself - it distributes your mistake.)

Full disclosure: I wasn't doing this alone. I had an AI assistant (Claude) in the loop for the whole change - it designed the test rehearsal, wrote the config, and later drove the recovery. It knew about this trap. It just never said it out loud before the prod run. Here's the moment it understood what had happened, opening with its own mea culpa:

This is the classic - and brutal - AOF-enablement trap, and I should have flagged it explicitly for prod: when Redis starts with appendonly yes in the conf but no AOF files exist yet, it loads from the (nonexistent) AOF and ignores dump.rdb entirely - it starts empty and creates an empty AOF. On the test environment this didn't bite because you ran CONFIG SET appendonly yes before rebooting - that built the AOF from the live dataset, so the reboot loaded it. On prod the conf was edited and rebooted directly - first boot came up empty, the next background save overwrote dump.rdb with the empty dataset, and each replica that reconnected did a full resync from the empty master and wiped itself too. That's how all three ended up at 0 keys.

A correct diagnosis in one message - six and a half hours too late to be a warning. (To be fair: I never asked "what could go wrong with this rollout?" either. Neither of us rehearsed the failure mode; we only rehearsed success.)

One honest footnote from the later forensics: block-level evidence showed that at one point during the rollout an AOF base with the full dataset existed on disk for a while, and the actual wipe most likely happened on a subsequent restart in the sequence. The exact fatal moment is unrecoverable; the end state was unambiguous - empty AOF, empty RDB, empty replicas.

And of course: no backup copy of dump.rdb taken before the change, and no off-box backups. (I know. I know.)

The recovery: your data is probably still on the disk

Here's the insight that saved us: because Redis replaces dump.rdb via rename(), the old file wasn't overwritten in place. Its inode was unlinked, but the data blocks were still sitting in the free space of the filesystem, waiting to be reclaimed.

So the plan became: stop all writes, and go dig through the raw block device.

Step 1 - freeze everything. Stop Redis, don't reboot, minimize writes to the filesystem. Every write is a chance for the filesystem to reclaim the very blocks you need.

Step 2 - try an LVM snapshot. The volume group had zero free extents, so no snapshot headroom. Plan B.

Step 3 - scan the block device for RDB signatures. Every RDB file starts with the magic bytes REDIS00. The obvious approach dies immediately on a 2 GB RAM box:

$ sudo grep -abo 'REDIS00' /dev/mapper/rhel-root
grep: memory exhausted

(Block devices have no newlines; grep tries to buffer one infinite "line".) So: a tiny Python scanner that reads the device in 32 MB chunks with an overlap of the pattern length, and prints the absolute byte offset of every match:

DEV   = "/dev/mapper/rhel-root"
PAT   = b"REDIS00"
CHUNK = 32 * 1024 * 1024

import sys
with open(DEV, "rb", buffering=0) as f:
    prev, base = b"", 0
    while True:
        buf = f.read(CHUNK)
        if not buf:
            break
        data  = prev + buf
        start = base - len(prev)
        i = data.find(PAT)
        while i != -1:
            print(start + i, flush=True)
            i = data.find(PAT, i + 1)
        prev  = data[-(len(PAT) - 1):]   # overlap so boundary-spanning hits aren't missed
        base += len(buf)
        if base % (1024**3) < CHUNK:
            print(f"... scanned {base // 1024**3} GB", file=sys.stderr)

Run it with sudo and nohup, go make coffee.

Step 4 - triage the hits. The scan found 10 candidates. Each got a 128-byte peek, read-only:

sudo dd if=/dev/mapper/rhel-root iflag=skip_bytes skip=$OFFSET bs=128 count=1 | od -c

Two hits were, hilariously, the scanner's own output files (the data directory shared the LV we were scanning - which is also why carve output had to be shipped off-box). Four were post-incident debris: the empty AOF bases and near-empty dumps left behind by the wipe. That left four candidates of 35-38 MB with real data, distinguishable by the timestamps embedded in their headers: the last pre-incident dump.rdb, and AOF base files from during the incident window.

Step 5 - carve the candidates to another machine. Never write recovery output to the disk you're recovering from:

sudo dd if=/dev/mapper/rhel-root iflag=skip_bytes skip=59609546752 bs=1M count=64 \
  | ssh user@rescue-box 'cat > /home/redis/cand_D.rdb'

Step 6 - validate. redis-check-rdb on the first candidate reported a CRC error - a block near the tail of the file had already been partially reclaimed - but it still parsed all 290,826 keys. Usable in an emergency, so we set it aside and kept going. The next candidate, the freshest one, came back clean:

$ redis-check-rdb cand_D.rdb
[offset 39730397] Checksum OK
[offset 39730397] \o/ RDB looks OK! \o/
[info] 290834 keys read

Step 7 - verify the contents in a sandbox. Never point production at an unverified file. A throwaway instance on the rescue box, isolated port, no config inheritance:

redis-server --port 6391 --dir /home/redis/rescue-test --dbfilename dump.rdb \
             --appendonly no --daemonize yes
redis-cli -p 6391 DBSIZE
redis-cli -p 6391 HGETALL FeatureFlag
redis-cli -p 6391 shutdown nosave

Real keys, real flag values. We had our database back - carved out of unallocated disk blocks.

Putting it back - carefully this time

Restoring into a Sentinel topology has its own trap: if a sentinel promotes a replica that still holds the empty dataset while you're restoring the master, replication will happily sync the emptiness right back over your restored data. So the order was strict:

Confirm sentinels are stopped on all nodes, then stop replicas, then the master.
On the master: move the poisoned artifacts aside - never delete evidence mid-incident:

   mv appendonlydir appendonlydir.bad
   mv dump.rdb dump.rdb.empty
   cp cand_D.rdb dump.rdb

Set appendonly no in redis.conf - the whole point is to force this boot to load dump.rdb.
Start the master and hold your breath:

   redis-cli DBSIZE
   (integer) 290834

Start the replicas (they full-resync from the restored master - you do not restore the RDB onto replicas), then the sentinels, last.
And only now, enable AOF the right way, on the live instance:

   redis-cli CONFIG SET appendonly yes
   # wait for: INFO persistence → aof_rewrite_in_progress:0
   # verify:   appendonlydir/ base file is megabytes, not ~100 bytes

and then persist appendonly yes into redis.conf. Plus, finally, an immediate scp of the recovered dump to another machine.

290,834 keys - eight more than the older candidate, because the winning file was written a couple of minutes later in the timeline. Full recovery.

One detail I still enjoy: the winning file wasn't the old dump.rdb at all. It was an AOF base file written during the incident - the populated base from that brief window when everything still existed. In Redis 7 the AOF base is itself RDB-format, which is why we could rename it to dump.rdb and boot straight from it.

Lessons

Enable AOF on the running instance first. CONFIG SET appendonly yes, wait for the rewrite to finish, verify the AOF base has real size - then edit the config file. Config-file-then-restart is the data-loss path, because Redis with AOF enabled ignores your RDB at boot.
"It worked in test" only counts if test rehearsed the same sequence. My test-environment run succeeded because I happened to run CONFIG SET first there. Same change, different order of operations, opposite outcome. Rehearse the runbook, not the end state.
Copy the data files before touching persistence settings. A 30-second cp dump.rdb dump.rdb.$(date +%F) (and ideally an scp off-box) would have turned a five-hour incident into a five-minute one.
Verify between steps of a rolling change. I rebooted three nodes back to back and only checked DBSIZE at the end. One check after the first node would have contained the blast radius.
Replication is not backup. The replicas didn't save the data - they synchronized its destruction.
If you do lose a file: stop writes immediately. Deleted ≠ gone. rename()-replaced files leave their blocks in free space, and dd + a signature scan can get them back - but only until something reuses those blocks.
Loose RDB-only persistence (save 3600 1, appendonly no) is its own slow-motion incident. Ours had been quietly able to roll back up to an hour of writes on every restart - that's what made a feature flag look haunted in the first place, which is the only reason we went looking.

The irony of the whole story: the change that destroyed the database was the one meant to make it durable. The fix was correct; the order was fatal. In operations, sequence is part of the change.

Have you ever had a "the fix caused the outage" incident? I'd love to hear about it in the comments.

A Design Flaw in Claude Code's Documentation Skill: One Question, 265k–355k Tokens

vadim albarov — Sun, 09 Aug 2026 16:34:33 +0000

I typed a one-line question into Claude Code - does fable use api billing? - and then ran /context out of habit.

30% of a 1,000,000-token context window was gone. One question, one answer, 295k tokens used.

Then I ran the same prompt on a second laptop: 40%. Same question, same answer, ~100k tokens more.

Same CLI version on both machines. I'll call them laptop 1 (30%) and laptop 2 (40%). This is the story of finding those 100k tokens. Spoiler: every theory I had was wrong, and the root cause turned out to be a design flaw in a single bundled skill - one you can partially work around by adding files to a folder.

The baseline

Here's what /context showed on laptop 1 after that single exchange:

Category	Tokens
System prompt	5.3k
System tools	23.7k
MCP tools (59 tools, deferred)	0
Memory files	318
Skills (16 skill descriptions)	2.2k
Messages	264.6k
Free space	703.9k

Two things jumped out.

First, the fixed overhead everyone worries about - MCP servers, skills, memory - is nearly free. 59 MCP tools sat at 0 tokens because Claude Code defers their schemas until they're actually used. All 16 skill descriptions together cost 2.2k tokens.

Second, the Messages category held 264.6k tokens after a one-line question. The conversation itself was maybe 2k tokens. The rest arrived because my question mentioned a Claude model name, which triggered the built-in claude-api skill - and a skill trigger doesn't just load instructions. This one injected its entire documentation payload into the conversation as a single message.

Worth pausing on that: my question was a billing lookup that a single Google search answers in five seconds. And before you conclude "well, agent sessions are just expensive" - they aren't. As a control, I asked two other lookup questions in fresh sessions: "what's the latest Node LTS version?" and "MIT vs Apache 2.0?". Both together cost 7.5k tokens. Ordinary questions are cheap.

The 265k burn has one specific trigger: mentioning a Claude model name. That summons the built-in claude-api skill, which has no notion of question weight - a casual pricing question gets the exact same multi-hundred-KB documentation payload as "implement a streaming tool-use loop." I paid a quarter of a million tokens for what one web search would have told me.

On laptop 2, that same category showed 354k tokens. The 90k-token mystery lived entirely inside one message.

Wrong theory #1: local CLAUDE.md files

My first guess: laptop 2 has more project instruction files - CLAUDE.md, memory, rules - quietly injected into context.

Dead on arrival. Memory files accounted for 318 tokens on laptop 1, and the gap was ~90k tokens ≈ 360 KB of text. A CLAUDE.md would have to be a small book. More importantly, the gap persisted when I ran the prompt from an empty folder on laptop 2 - no project files at all, still ~355k.

Wrong theory #2: skill version

Bundled skills live in a content-addressed cache:

%LOCALAPPDATA%\Temp\claude\bundled-skills\<cli-version>\<hash>\claude-api\

Both machines ran CLI 2.1.226 with skill version 2.1.226. Same version... but different hashes. Promising! Except when I compared the actual files, every doc folder was byte-identical - 847 KB of assets on both machines. The hash difference was real but, as it turned out later, a symptom rather than a cause.

Wrong theory #3: the tokenizer

This one almost fooled me, because the arithmetic was beautiful. I had tried Sonnet 5 at some point, and it ships a new tokenizer that produces roughly 30% more tokens for the same text.

264.6k × 1.3 ≈ 344k. Almost exactly laptop 2's number. Same bytes, different ruler!

Then I ran the control: same model, same effort, both machines. The gap survived. Tokenizer eliminated. (Keep this failure mode in mind though - token-budget intuitions genuinely don't transfer across model families.)

Transcript forensics

Same CLI, same skill assets, same model, same prompt - and different token counts. At this point the only honest move was to stop theorizing and diff the actual bytes.

Claude Code writes every session to a JSONL transcript. Finding the heavy message takes one loop:

$proj = Get-ChildItem "$env:USERPROFILE\.claude\projects" -Directory |
  Sort-Object LastWriteTime -Descending | Select-Object -First 1
$t = Get-ChildItem $proj.FullName -Filter *.jsonl |
  Sort-Object LastWriteTime -Descending | Select-Object -First 1
$i = 0
Get-Content $t.FullName | ForEach-Object { $i++
  if ($_.Length -gt 50KB) { "Line $i : $([math]::Round($_.Length/1KB)) KB" } }

Laptop 1 transcript: one message of 719 KB. Laptop 2 transcript: one message of 957 KB. There's the gap - 238 KB of text, ~90k tokens at ~2.6 characters per token.

The payload is the skill's documentation, embedded as <doc path="..."> blocks. Extracting the doc lists:

[regex]::Matches($line, '<doc path=\\"([^\\"]+)\\"') |
  ForEach-Object { $_.Groups[1].Value }

Laptop 1: 32 docs - shared API docs + the python/ folder.
Laptop 2: 65 docs - shared API docs + all eight language folders: Python, TypeScript, Go, Java, C#, PHP, Ruby, and cURL.

All 32 docs the payloads had in common were byte-identical. The laptop 2 payload simply contained 33 extra language docs totaling 237 KB.

The reveal

Diffing the instruction text at the top of the two payloads (60 KB each, otherwise identical) surfaced exactly one difference:

Laptop 1 payload:

→ Refer to python/claude-api/README.md

Laptop 2 payload:

→ Refer to unknown/claude-api/README.md
No project language was auto-detected. Ask the user which language they are using, then refer to the matching docs below.

That's the whole mechanism. When the claude-api skill triggers, Claude Code detects your project's language from the working directory and injects only that language's documentation. When it detects nothing - an empty folder, a docs-only repo - the fallback is to inject documentation for every supported language, because the model might need any of them.

And why did laptop 1 detect Python? The folder I was in had a subdirectory containing a Python project with a .venv - thousands of .py files. A stray virtualenv saved me 90k tokens.

The different cache hashes made sense now too: the skill bundle appears to be cached per rendered variant - the assets are identical, but the instruction text differs by detection outcome, so each outcome gets its own hash directory.

The irony, and the side effect worth knowing

Here's my favorite part. When I got serious about controlling variables, I ran the "clean" experiment: empty folder, fresh session, same prompt. That methodologically pure setup is precisely what maximizes the payload. The controlled experiment created the condition it was measuring.

The flip side is a genuinely useful, if odd, side effect: having language context in your working folder reduces token usage. Any file that lets Claude Code detect a language - a .py file, a package.json, a pyproject.toml, even a leftover .venv - pins the skill payload to one language's docs and cuts ~90k tokens (about 25% of the payload) off every claude-api skill trigger. The final scoreboard, reproduced on both machines:

Working directory	Docs injected	Payload	Messages after one question
Any folder with language markers	32 (shared + one language)	719 KB	~265k tokens
Empty / language-less folder	65 (shared + all 8 languages)	957 KB	~355-360k tokens

So if you're about to ask Claude Code API questions from some scratch directory: don't. Run it from a real project - or drop a single pyproject.toml (or the equivalent for your language) into the scratch folder first. It reads as a joke, but it's a measurable 90k-token difference per session, and on smaller context windows it's not funny at all: the all-languages payload alone wouldn't fit in a 200k-token context window. This entire question is only answerable on a 1M-window model.

Why this is a flaw, not a feature

Skills in Claude Code are designed around progressive disclosure: a one-line description sits in context (all 16 bundled skills together cost 2.2k tokens), and the full instructions load only when triggered. The claude-api skill follows that pattern for its trigger - and then abandons it entirely for its content: instead of letting the model read the docs it needs on demand, it eagerly injects the whole documentation set as a single message. It's the only bundled skill big enough to need a disk cache at all (847 KB; the other 15 are trivially small).

The no-language fallback makes it worse in a way that's almost comic. The injected instruction text literally says:

No project language was auto-detected. Ask the user which language they are using, then refer to the matching docs below.

...but by the time the model can ask, all eight languages' documentation has already been paid for. The question comes after the purchase. Any of the obvious designs - ask first and inject one language; inject the shared docs and let the model read language files on demand; scale the payload to the question - would cap the cost at the detected-language level or below.

To be fair about scope: this is one skill, in one CLI version (2.1.226), and the payload is genuinely useful when you're writing code against the Claude API in a detected-language project. The flaw is the eager all-languages fallback and the trigger's insensitivity to question weight - both fixable upstream without losing what the skill is for.

Takeaways

Skill payloads dominate context cost. Everything people usually blame - MCP servers, memory files, system prompts - added up to ~31k tokens for me. One skill trigger added 265-360k. Audit accordingly.
The flaw is specific, not general agent overhead. Control questions in fresh sessions ("latest Node LTS?", "MIT vs Apache 2.0?") cost 7.5k tokens combined. The expensive trigger is mentioning a Claude model or API name. For quick Claude pricing/docs lookups, use a web search - inside Claude Code, the skill will fire.
Deferred loading works. 59 MCP tools at 0 tokens until used. If your setup loads MCP schemas eagerly, that's worth fixing, but it wasn't my problem.
/context tells you the category; the transcript tells you the culprit. The JSONL line-length trick above takes 30 seconds and points at the exact message.
Same CLI version ≠ same context cost. The cost depends on runtime conditions - in this case, what's sitting in your working directory.
Language markers in the folder are a token optimization. Unintuitive, but reproducible: give the language detector something to find, and the skill injects one language's docs instead of eight.
Token intuitions don't transfer across model families. My tokenizer theory was wrong this time, but the ~30% Sonnet 5 difference is real - re-baseline when you switch models.

Everything above was measured on Claude Code 2.1.226 (bundled claude-api skill 2.1.226) with Fable 5 and Sonnet 5, on two Windows machines. The behavior may well change in future releases - arguably the fallback should ask before injecting 237 KB of polyglot documentation - but the audit method will keep working regardless.

Reproduce it yourself: ask Claude Code any Claude-API question from an empty folder, run /context, then do the same from inside a Python or TypeScript repo and compare the Messages category.

UPD: The 2.1.234 update fixed the issue (at least for me). Now the claude‑api skill is no longer bloated and does not exhaust your limits anymore