DEV Community: vadim albarov

The 49-Byte Field That Doubles Our SQS Bill

vadim albarov — Sun, 25 Oct 2026 15:55:49 +0000

TL;DR: Amazon SQS does not bill per message. It bills per request, and every 64 KiB chunk of payload counts as one request. A message of 65,536 bytes is one request. A message of 65,537 bytes is two. If your producer packs messages tightly under that line (a lot of chunkers do exactly this, on purpose), then adding a harmless traceId to the envelope doubles the billed requests for every send and every receive, and the total SQS bill goes up 67% to 95% depending on how you delete. Every price here comes from the AWS pricing page; the scenario is invented to show the mechanism. Here is how the cliff works, how to see whether you are standing on it, and how to make it a slope instead.

The rule nobody re-reads

The SQS pricing page has one sentence that matters here:

Each 64 KB chunk of a payload is billed as 1 request.

The example right below it says a 1 MiB payload is billed as 16 requests. 16 × 65,536 = 1,048,576, so "64 KB" means 64 KiB and the chunk is 65,536 bytes. AWS does not publish the exact boundary or which bytes it counts, so treat 65,536 as the implied line, not a documented one.

Two more facts complete the picture:

Message attributes count. The developer guide says all components of a message attribute (name, type, value) are included in the message size. Moving a field from the body to an attribute does not hide it.
The rule applies to every API action's payload. SendMessage is chunked on the way in. ReceiveMessage is chunked on the way out, and its response also carries receipt handles and metadata, so a message near the line may already cost two on receive. DeleteMessage carries only a receipt handle, so it stays at one request.
One exception. The X-Ray system attribute AWSTraceHeader does not count toward message size. If your tracing is X-Ray, use it. The scenario below assumes it is not.

The prices themselves are small, which is exactly why nobody looks:

Queue type	Per million requests (first 100 billion, us-east-1)
Standard	$0.40
FIFO	$0.50

A scenario that is boring enough to be real

Imagine a service that exports large reports through SQS. Years ago, someone read the pricing page and wrote a chunker: split the report into pieces, each sized to fit in 64 KiB, send each piece as one message. The constant is called MAX_CHUNK_BYTES and it is set to 65_536 minus the size of a small JSON envelope. There is no comment explaining the number.

The chunker works. The event records are small, so every full chunk lands within a few dozen bytes of the line, between about 65,490 and 65,535 bytes. One request each. Nobody thinks about it again.

Then a debugging ticket comes in. Cross-service tracing is painful, so the team adds a trace ID to the shared message envelope library. The change is one line in the serializer:

,"traceId":"4f1c2b7e-9a3d-4e58-b2f0-6c8d1a9e3b47"

That is 49 bytes. The PR is reviewed in four minutes. Tests pass. It ships.

The chunker never sees this field. It sized the payload before the envelope library serialized it. So every full chunk is now 65,539 to 65,584 bytes. All of them cross 65,536.

What changes:

NumberOfMessagesSent in CloudWatch: unchanged.
Throughput, latency, error rate: unchanged.
Billed requests per message: send goes from 1 to 2, receive goes from 1 to 2, delete stays at 1.

Nothing in the usual dashboards moves. The bill does.

The math

Take a Standard queue, no batching, 1 billion chunk messages per month (about 390 per second, sustained). Consumers are EC2 or Lambda in the same region, so there is no data transfer charge. Ignore the free tier and empty receives.

	Requests per message	Requests per month	Cost per month
Before (send 1 + receive 1 + delete 1)	3	3 billion	$1,200
After (send 2 + receive 2 + delete 1)	5	5 billion	$2,000

That is +$800 per month, $9,600 per year, for 49 bytes nobody needed in production. On a high-throughput FIFO queue it is $1,500 to $2,500 (a default FIFO queue caps at 300 unbatched requests per second per partition, so this workload would not fit without batching).

Notice the total is +67%, not +100%. The send and the receive did double. The delete did not, because it carries only a receipt handle, and it dilutes the ratio. You cannot skip it: an undeleted message comes back after the visibility timeout and gets received, and billed, again.

Most consumers delete in batches, though. With DeleteMessageBatch at 10 per call the delete almost disappears from the math, and the total goes from 2.1 to 4.1 requests per message, +95%. So the honest range is "the bill goes up 67% to 95%, depending on how you delete", and the honest headline is "the cost of every message you send and receive doubles". If you say "doubled" in a meeting, be ready to say which one you mean.

Also notice the shape of the function. The cost per message is:

requests = ceil(size_in_bytes / 65536)

It is a staircase. Between steps it is free to grow. At each step it jumps. The 49-byte field was expensive only because the messages were already standing on the edge of a step. The same 49 bytes added to 20 KiB messages cost nothing at all.

How to know if you are on the edge

You cannot see this from message counts. You need sizes.

CloudWatch, per queue. The SentMessageSize metric has Average, Maximum, and SampleCount. Plot Maximum and Average against a horizontal line at 65,536. If the Average sits within a few hundred bytes under the line, you have a packer, and you are on the edge.

Cost Explorer. Filter to SQS, group by usage type. The line is Requests-Tier1 or Requests-RBP (FIFO has its own variants, and regions outside us-east-1 add a prefix). Divide monthly billed requests by NumberOfMessagesSent summed over your queues, after subtracting NumberOfEmptyReceives. For an unbatched workload with small messages the ratio is about 3. If it is about 5, your messages are between 64 KiB and 128 KiB and nobody told you.

One-off check. Pull 1,000 messages with ReceiveMessage and a short visibility timeout, dedupe on MessageId, measure the UTF-8 byte length of body plus attributes, and histogram it. A spike just under 65,536 is the signature.

Making the cliff a slope

Batch. This is the one people underestimate. AWS's wording applies the chunk rule to the payload of the API action, and a SendMessageBatch call holds up to 10 messages and up to 1 MiB in total. Read literally, the batch is summed and then rounded. Ten messages of 65,500 bytes are 655,000 bytes, which is 10 chunks. Ten messages of 65,549 bytes are 655,490 bytes, which is 11 chunks. AWS does not state the sum-then-round behaviour outright, so check it against your own bill before you depend on it. The same change that doubled the unbatched send cost adds 10% to the batched one. Batching does not just reduce request count. It averages out the rounding.

Same scenario, batched 10 at a time for send, receive, and delete:

	Requests per month	Cost per month
Before	2.1 billion	$840
After	2.3 billion	$920

+$80 instead of +$800.

Compress. SQS does not compress for you. It stores the bytes you send and bills on them. But JSON report chunks typically gzip 3x to 10x in application code. One catch: the message body must be valid text, so raw gzip output has to be base64-encoded, which adds about a third to the compressed size. A 65 KiB chunk still ends up around 15 to 25 KiB. A Binary message attribute with a one-character body avoids base64 in your code, but AWS does not document whether it bills the raw or the encoded bytes, so do not count on it saving money. Either way, mark compressed messages with an attribute like contentEncoding so consumers can handle both formats during rollout.

What does it cost in CPU? I measured a 65 KB JSON chunk with CPython's zlib on a desktop CPU:

gzip level	Size after gzip + base64	Compress	Decompress
1	18 KB (3.6x smaller)	0.6 ms	0.1 ms
6	14 KB (4.6x smaller)	1.4 ms	0.1 ms
9	13.5 KB (4.8x smaller)	6.1 ms	0.1 ms

Level 9 is not worth it. At level 1 and 1 billion messages a month, the producer needs about a quarter of a core on average and under one core at a daytime peak, and the consumer a few percent of a core. Call it 1 to 2 vCPUs on cloud hardware, about $60 a month, against an $800 a month increase. Latency per message goes up by about a millisecond, which is noise next to a 10 to 30 ms HTTPS round trip.

And once chunks are 14 KB, batching works again: ten compressed chunks are 140 KB, which is 3 billed chunks instead of 10. Compressed and batched, the same workload is about 0.7 billion requests a month, or $280. That is below where you started, not just back to it. You are now nowhere near a step, and you can add all the debug fields you like for years.

Size the envelope, not the body. If you keep a chunker, make it measure the final serialized message, after every library has had its turn. Better: make it a test.

def test_chunk_fits_in_one_billing_unit():
    msg = build_largest_possible_chunk()
    wire = serialize(msg).encode("utf-8")  # body + attributes, as sent
    assert len(wire) <= 65_536 - 512, (
        "SQS bills per 64 KiB chunk; see docs/sqs-billing.md"
    )

The margin and the assertion message are the point. The original author of MAX_CHUNK_BYTES knew the rule. The test is how the next person finds out.

Claim check. If chunks are large because the payload is large, put the payload in S3 and send a pointer. The SQS Extended Client Library (Java and Python) does this for you. It was the standard answer before the limit went to 1 MiB in August 2025, and it is still the answer above 1 MiB.

What I would actually take from this

The uncomfortable part of the scenario is not the rule. It is that the system had the knowledge once, encoded it as a number, and lost the reason. A field that costs nothing to add, nothing to run, and nothing to test can still cost $9,600 a year, and the only place it shows up is a line in Cost Explorer that someone checks quarterly.

If you run SQS at volume, spend ten minutes this week: open SentMessageSize for your busiest queues and look at where the Average sits relative to 65,536. If it is close, you have found either a careful engineer from years ago or a bill waiting to happen. Usually both.

When Does Claude Code Check the Web? I Logged My Sessions to Find Out

vadim albarov — Thu, 15 Oct 2026 13:30:00 +0000

On October 12, 2026, AWS launched Lambda Web Functions. You can now run a normal Express or Next.js app on Lambda, with up to 128 concurrent requests per execution environment.

Two days later I asked three Claude models in Claude Code: "Our Express app is in app.js. Add the files needed to deploy it to AWS Lambda with the fewest moving parts."

All six runs wrote a setup that was the best answer until October 11: serverless-http, serverless-express, or the Lambda Web Adapter. Five of them looked nothing up. The sixth only checked the latest version of serverless-http on npm.

Then I asked "What's the newest way to run an Express app on AWS Lambda, as of this month?" Every run searched the web. Five of six named Web Functions.

The topic was the same. Only the wording changed. I wanted to know how far that goes. When does Claude Code look things up, and when does it answer from its training data? And when it answers from training data, how often is the answer out of date?

The setup

Seven topics that changed after the models' training cutoff (June 2026). I checked each one against the official source on the day of the test:

Topic	Current fact	Since
Lambda Web Functions	new Lambda form factor for HTTP apps	2026-10-12
Lambda Python runtimes	`python3.14` GA, `python3.15` in preview	2025-11 / 2026-08
`actions/setup-node`	v7	2026-07-14
openai Python SDK	3.x, built on HTTPX2, no longer installs `httpx`	2026-08-12
Stripe API version	`2026-09-30.endive`	2026-09-30
Django	6.1	2026-08-05
Python	3.15.0	2026-10-09

Three ways to ask about each topic:

Neutral question: "Which major version of actions/setup-node should a GitHub workflow use?"
Time cue: "What's the latest major version of actions/setup-node today?"
Coding task: "Add a GitHub Actions workflow that runs npm test on every push."

Plus three stable controls, like "What does HTTP status 409 mean?"

That makes 24 prompts. I ran them on Haiku 5.5, Sonnet 5.5, and Opus 5.5, two trials each: 144 sessions. Each one was a fresh headless session in a small repo (an Express app, a Python Lambda handler, a README):

claude -p "<prompt>" --model <model> --output-format stream-json --verbose \
  --max-turns 10 --permission-mode acceptEdits \
  --allowedTools "Read,Glob,Grep,Edit,Write,WebFetch,WebSearch,Bash,PowerShell,ToolSearch"

Before each run, the repo is reset with git and there is no project memory.

What counts as a web call: WebFetch, WebSearch, and shell commands that reach the network. That last group matters. Claude often skips WebFetch and calls gh api, curl against the PyPI JSON API, pip index versions, or git ls-remote --tags.

Grading. A Sonnet subagent graded each of the 126 non-control answers against the ground truth table, as one of three grades:

current: uses the current fact as a fact.
hedged: old or unsure, but tells the user it may be out of date.
stale: presents old information as current.

For coding tasks it also graded the file the model wrote. I checked the grades by hand and changed four of them (see Limits).

Finding 1: the trigger is the words, not the topic

Share of runs with at least one web call:

	Stable control	Neutral question	Coding task	Time cue
Haiku 5.5	0/6	8/14	5/14	14/14
Sonnet 5.5	0/6	7/14	4/14	14/14
Opus 5.5	0/6	12/14	7/14	14/14

"Latest", "today", "right now", "as of this month": 42 out of 42 runs looked it up. Stable questions: 0 out of 18. That part works as you would hope.

The middle columns are where it gets interesting. Within them, the pattern depends on what kind of question it is:

"Which version should I use?" usually triggers a lookup. For setup-node, Lambda Python runtimes, and Stripe API versions, 18 of 18 neutral runs checked.
"How do I use this API?" never does. "How do I pass a custom httpx client with a proxy to the OpenAI Python SDK?" got 0 lookups in 6 runs. None mentioned that version 3 exists. Five of six used httpx.Client, which fails on a fresh install of the current SDK (details below). One Opus run used the SDK's DefaultHttpxClient, which still works.
"Can this service do X?" depends on the model. "Can AWS Lambda serve many concurrent HTTP requests from one execution environment?" Haiku and Sonnet answered from memory every time ("no, one request per environment"). Opus searched both times. More on what it found below.

Finding 2: no lookup means a stale answer, and usually no warning

Here is every graded answer, split by whether the session made a web call:

	Runs	Current	Hedged	Stale
With a web call	85	69 (81%)	12	4
Without a web call	41	3 (7%)	7	31 (76%)

The three "current" answers without a web call were all Opus on the OpenAI topic. Each used the SDK's own DefaultHttpxClient helper, which version 3 still supports. In the two coding runs, Opus also ran the code against the installed package, which happened to be the current version. None of the three knew that version 3 exists. The code worked because the SDK kept an old name. Apart from that one case, no answer from training data alone was current.

That is expected for facts that changed after the cutoff. What I didn't expect was how rarely the models said so. Of the 35 stale answers, 28 gave no warning at all. The other 7 had some caveat, but often a narrow or generic one. For example, one answer warned about checking the Lambda Web Adapter layer version, but it didn't know Web Functions existed.

When Sonnet did hedge, the hedges were good ones: "Django 6.1 if it is out, I'm not sure it has shipped", "I answered from memory and did not check". When it hedged, it meant it.

Finding 3: coding tasks are the blind spot

Here is what ended up in the files for the coding tasks that did not mention "newest":

Task	Web lookups	What was written (6 runs)	Current
GitHub workflow running `npm test`	0/6	`setup-node@v4` (Haiku, Sonnet), `@v5` (Opus)	v7
SAM template for a Python Lambda	0/6	`python3.13` (5 runs), `python3.12` (1 run)	`python3.14`
OpenAI client through a proxy	0/6	`import httpx` + `httpx.Client(proxy=...)` (Haiku, Sonnet), `DefaultHttpxClient` (Opus)	`DefaultHttpx2Client`
Express app on Lambda	1/6	`serverless-http`, `serverless-express`, Lambda Web Adapter	Web Functions
Stripe client with a pinned API version	3/6	`2026-09-30.endive` (3), `2025-09-30.clover`, `2025-03-31.basil`, `2024-06-20`	`2026-09-30.endive`

The Stripe row shows it best. Opus searched in both runs and pinned the current version. Haiku searched in one of two runs and got it right in that one. Sonnet never searched and pinned versions from 2025.

Two coding tasks did trigger lookups every time: "pin Django to the newest stable version" and "a Dockerfile on the newest stable Python". 12 out of 12. The word "newest" in the coding task was enough.

The OpenAI row needs a note, because "outdated" can mean "still works". I checked it in a clean virtual environment with only openai==3.31.0 installed. The Haiku and Sonnet code fails on its first line:

ModuleNotFoundError: No module named 'httpx'

openai 3.x depends on httpx2 and no longer pulls in httpx. In an older environment that still has httpx, the same code runs. The Opus version uses the SDK's own DefaultHttpxClient, which 3.x keeps as an alias built on httpx2. That one works on a fresh install too.

Finding 4: the training data is older than the cutoff

These models have a June 2026 training cutoff. But look at what the coding tasks wrote:

Lambda added python3.14 on November 18, 2025, seven months before the cutoff. All six SAM templates used 3.13 or 3.12.
actions/setup-node v6 shipped in October 2025. Haiku and Sonnet wrote v4 (2023). Opus wrote v5 (September 2025).
Haiku pinned Stripe API version 2024-06-20.

So "the model knows things up to its cutoff" is too generous for coding defaults. My guess at the reason: when a model writes a version from memory, it writes the version it saw most in training. That is the version most existing code used, often a year or two behind the cutoff. The newest versions are thin in the training data, because little code used them yet.

The practical effect: checking whether something changed after the cutoff is not enough. A version pinned from memory can be stale even for things released well before it.

Finding 5: a lookup is not the same as a current answer

Four stale answers came after a web call. They show two different ways a search can fail.

The search asked about what the model already knew. On "Can AWS Lambda serve many concurrent HTTP requests from one execution environment?", Opus searched both times. Both times the query was:

AWS Lambda Managed Instances multi-concurrency execution environment

Lambda Managed Instances is a real feature from late 2025. Opus knew it from training, searched to confirm it, and got a good, sourced answer about Managed Instances. It never found Web Functions, launched two days earlier, which is the direct answer to the question. The search was built from the model's memory, so it could only confirm that memory.

The search index was behind. Python 3.15.0 came out on October 9. On October 14, Sonnet searched "latest stable Python release python.org downloads" and got cached snapshots of python.org listing 3.14.8 as the newest release and 3.15 as a pre-release. It wrote python:3.14-slim into the Dockerfile. In a different run, the query "latest stable Python release version October 2026" found 3.15. The search with the month in it worked. The one without it got old snapshots.

To be fair, Sonnet said it got the version from a web search. And when asked the plain question ("What is the newest stable Python release today?"), it got the same stale snapshots. Both times it said the search results disagreed, and once it added that 3.15 "may have shipped since". It was honest about the gap. In the Dockerfile run, it still wrote the old version.

Finding 6: one line in CLAUDE.md fixes most of it

If the gap is that coding tasks don't trigger lookups, the obvious fix is to tell the model to look things up. I added this CLAUDE.md to the repo and reran the seven coding tasks, 3 models x 2 trials:

Your training data may be out of date. Before you write any version number,
runtime identifier, GitHub Action version, API version, or SDK usage pattern
into a file, look up the current value (web, package registry, or official
docs). Prefer the newest stable option unless the repo pins something else.

Coding tasks with a web lookup went from 16 of 42 runs to 42 of 42. Answers graded current went from 15 of 42 to 33 of 42.

Per task, files with the current value (6 runs each):

Task	Without CLAUDE.md	With CLAUDE.md
GitHub workflow (`setup-node`)	0/6	6/6 (`@v7`)
SAM template (Python runtime)	0/6	6/6 (`python3.14`)
Stripe client (API version)	3/6	6/6 (`2026-09-30.endive`)
OpenAI client through a proxy	2/6 work	4/6 (`DefaultHttpx2Client`), Haiku still wrote `import httpx`
Django requirements ("newest")	6/6	6/6
Dockerfile ("newest" Python)	5/6	5/6
Express app on Lambda	0/6	0/6 (Opus found Web Functions, then ran out of turns)

Version pins are fixed. The two tasks that already said "newest" didn't change, because they were already looking things up.

Two things the rule did not fully fix:

Haiku looked up the version, not the API. On the OpenAI task, Haiku checked the latest version on PyPI both times. It then wrote the same import httpx code as before. It treated "look up" as "find the version number", not "find out how the SDK is used now". Sonnet and Opus read the SDK README and wrote DefaultHttpx2Client.

The approach changed only for the models that read past the version number. Haiku and Sonnet used the rule to look up versions for the approach they had already chosen from memory: the latest serverless-http on npm, the latest Lambda Web Adapter layer, the latest Node runtime. One Sonnet run also opened the AWS page on Web Functions runtimes. It then named Web Functions as an alternative and explained why it didn't use them: app.js would need a small change, and it wasn't sure SAM supports them yet. That's a reasonable call. The other Haiku and Sonnet runs never mentioned Web Functions.

Opus went further. In both runs, the Lambda runtimes page led it to Web Functions. It then read the Web Functions runtimes page, the samples, the "create" guide, and the aws lambda-web deploy CLI reference. It was clearly heading for the new approach. Then both runs hit my 10-turn limit before writing a file. So I can't say what it would have written, only that the rule made it find the feature that launched two days earlier.

The cost of the rule: across the 42 coding runs, the total went from $2.51 to $4.47 (+78%), and Sonnet's average time went from 10 to 18 seconds. For setup work that pins versions, that's cheap. For every small edit, you may not want it.

The models compared

	Avg web calls per run	Coding tasks graded current	Cost for 48 runs
Haiku 5.5	1.08	4/14	$0.57
Sonnet 5.5	0.81	3/14	$2.02
Opus 5.5	1.33	8/14	$5.68

Opus looks things up the most, mostly on neutral questions (12 of 14 against 7 or 8 for the others), and it is the only one that searched on the Stripe coding task both times. Haiku searched more than Sonnet. Sonnet was the most likely to answer from memory, and also the most likely to say so.

Costs are the total_cost_usd Claude Code reports per session. The whole experiment cost about $8.30 for the 144 main runs, plus $4.50 for the 42 CLAUDE.md runs.

Limits

Two trials per model. The patterns were very stable across trials, but rare behavior won't show up.
Seven topics. All are developer tooling with a version number or a launch date. I didn't test topics where "what changed" is fuzzier.
Headless sessions with acceptEdits. An interactive session may behave a bit differently.
One grader, a Sonnet subagent, with my spot checks. I changed four grades in two cases. One rewarded a generic "AWS may have changed" footer. In the other three, old-style OpenAI code turned out to still work on the new SDK.
A 10-turn cap. Both Opus runs of the Express task with the CLAUDE.md rule hit it while reading the Web Functions docs, so they wrote no files. With a higher cap, that row would probably look better for Opus.
The test date was close to the launches on purpose. Lambda Web Functions were two days old. That is the hardest case, and the most realistic one for "what changed last week".

What I take from this

Time words are the trigger. "Latest", "newest", "current", "today": with them, all three models looked things up in every run. Without them, the model mostly answers from memory.

Coding tasks don't trigger lookups. "Write a workflow", "write a SAM template", "write a Stripe client" all pin a version, and the model picks it from memory. Most of the stale values in this test were written into files, not stated in answers.

"How do I use this API?" is the hardest case. It doesn't sound time-sensitive, so no model checked. But SDKs change their APIs, and the answer is the old API with no warning.

Read the version pins in every diff. Actions versions, runtime identifiers, API version strings, base image tags. These are the lines most likely to come from old training data, and the easiest to check.

Put the rule in CLAUDE.md. One paragraph took version pins in coding tasks from 0 of 6 to 6 of 6 for setup-node and the Lambda runtime. It costs extra lookups on every coding task. It fixes versions reliably. It changes the approach only when the model reads beyond the version number, which Opus did and Haiku did not.

When you need the newest thing, say "newest", and name the month if it's recent. "Newest stable Python" triggered a lookup every time. "October 2026" in the query got past stale search snapshots. And if you know a feature launched last week, name it. A search built from the model's memory can only confirm that memory.

A Reader Asked Whether a Hotter Rewriter Washes Out SynthID Faster. I Ran the Sweep.

vadim albarov — Wed, 14 Oct 2026 16:10:47 +0000

In my SynthID post I rewrote 100 watermarked texts with three local models, and the detector still saw a trace in most of them. Every rewriter ran at one sampling setting: temperature 0.7, with Ollama's defaults of top-p 0.9 and top-k 40. I never varied it.

A reader asked the obvious question. SynthID lives in exact token sequences: the watermark nudges each token based on the 4 tokens before it, so a rewrite carries watermark only where it reproduces a 5-token window from the original. Would a hotter rewriter shatter those windows faster? Or would it just shatter the meaning faster?

I did not know. So I ran it.

Short answer: both, in that order. Turning the temperature up to 1.3 while keeping the usual nucleus cut nearly halved the number of rewrites the detector flagged, and a blind judge could not tell the difference in meaning or readability. Removing the nucleus cut as well removed only a little more watermark, and that is where the text broke: a quarter of those rewrites lost or distorted facts. Greedy decoding went the other way and kept more watermark. And nothing got to zero.

Why the sampling knob should matter

Two things decide how much of the original's wording a rewriter reuses.

Temperature sets how strongly the rewriter prefers its most probable next token. Low temperature means the most natural phrasing, and the most natural phrasing is often what the original model wrote too. That was the lesson of the second-pass result in the first post, where paraphrasing a paraphrase landed back on the original's phrases. A hotter rewriter should land on them less often.

Top-p, the nucleus cut, decides which tokens can be sampled at all. At top-p 0.9 the rewriter only ever picks from the plausible set, however hot the temperature. At top-p 1.0 the whole tail is open, including tokens the model thinks are unlikely.

In the first post, the share of the original's 5-grams that survived a rewrite tracked the watermark z-score with a correlation of 0.77. So the prediction is simple: anything that lowers 5-gram overlap should lower the signal. The question is what it costs.

The setup

Rewriter: gemma3:12b, the heaviest washer from the first post. Same prompt, same seed, same context length. Only the sampling options changed.
Five settings: temperature 0 (greedy), 0.7 (the original run, used as the baseline), 1.0, 1.3, and 1.3 with top-p raised to 1.0.
Texts: the same 100 watermarked originals, 190 to 900 words across 10 genres. Each setting also rewrote the 100 clean originals, so I can check that the detector's null does not move when the rewriter gets hotter. That was a lesson from the readers of the first post.
Detector and cutoffs: unchanged from the October update. The z cutoff of 2.42 was set on a calibration half of 850 clean texts for a 1% false-positive rate. Nothing was recalibrated for this test.
Comparisons: every setting rewrote the same 100 originals, so differences from the baseline are paired by source text, with bootstrap intervals over the 100 sources.
Meaning: 500 original-and-rewrite pairs, shuffled into 25 batches of 20 with the settings mixed, graded by a Claude judge on meaning (1 to 5: facts, names, numbers, steps and plot events preserved, nothing invented) and fluency (1 to 5). The judge never saw which setting produced a rewrite.

What happened

Rewriter setting	5-grams kept	Mean z	Flagged (mean test)	95% CI	AUC vs clean	AUC vs rewritten null
temp 0 (greedy)	12.7%	1.78	31%	23 to 41	0.80	0.79
temp 0.7 (first post)	10.4%	1.56	25%	18 to 34	0.79	0.80
temp 1.0	9.3%	1.38	21%	14 to 30	0.77	0.79
temp 1.3	8.2%	1.08	14%	9 to 22	0.71	0.71
temp 1.3, top-p 1.0	5.9%	0.88	11%	6 to 19	0.70	0.67

Paired against the 0.7 baseline over the same 100 texts:

Rewriter setting	Change in z	95% CI	Change in 5-grams kept	Meaning (1 to 5)	Change in meaning	Fluency (1 to 5)	Change in fluency
temp 0	+0.22	-0.01 to +0.46	+2.3 pts	4.26	+0.07 (-0.06 to +0.19)	4.92	-0.03 (-0.09 to +0.03)
temp 0.7	baseline			4.19		4.95
temp 1.0	-0.18	-0.44 to +0.08	-1.1 pts	4.22	+0.03 (-0.09 to +0.15)	4.90	-0.05 (-0.11 to +0.01)
temp 1.3	-0.48	-0.72 to -0.23	-2.1 pts	4.11	-0.08 (-0.19 to +0.03)	4.93	-0.02 (-0.09 to +0.05)
temp 1.3, top-p 1.0	-0.67	-0.99 to -0.36	-4.4 pts	3.82	-0.37 (-0.50 to -0.24)	4.32	-0.63 (-0.77 to -0.49)

Greedy decoding keeps more watermark. At temperature 0 the rewriter kept 12.7% of the original's 5-grams instead of 10.4%, and the flagged rate went up from 25% to 31%. The z change is +0.22 with an interval that just touches zero, so call it a lean rather than a proof. But the direction is the second-pass effect again: the most probable phrasing is the original's phrasing, and the watermark comes with it.

Up to the nucleus cut, temperature is a free knob. At 1.3 with the default top-p 0.9, the flagged rate dropped from 25% to 14%, and the paired z change of -0.48 has an interval well clear of zero. The judge saw nothing: meaning 4.11 against 4.19, fluency 4.93 against 4.95, both differences inside noise. Temperature 1.0 was halfway and not clearly different from 0.7 on anything.

Open the tail, and the text breaks before the watermark does. Raising top-p to 1.0 at the same temperature took the flagged rate from 14% to 11%, which the intervals cannot separate. What it clearly did was damage the rewrites. Mean meaning fell by 0.37 points and fluency by 0.63, and a quarter of texts scored 3 or lower on meaning against 7% for the baseline. The judge's notes are specific: a science explainer dropped the inverse fourth-power law and invented "two to ten times"; a story dropped a mother's death date and reversed who said what in a dialogue; an incident report's "voltage drop below threshold" became "voltage reading below permissible level," which flips the sense. One in ten of these rewrites also read awkwardly, where none of the others did.

The null does not move. Clean rewrites scored a mean z of 0.07, -0.01, -0.04, 0.08 and 0.20 across the five settings, and they lost 5-grams at the same rate as the watermarked ones. The drop in signal is signal loss, not a shift in what clean text looks like to the detector.

Nothing gets to zero. Even the hottest setting left 11% of rewrites flagged at a 1% false-positive cutoff, and the AUC against the rewritten null was still 0.67. The point from the first post stands: too faint to accuse any one short text, strong enough to add up across a body of work.

One more number. Across all 500 judged rewrites, the correlation between 5-grams kept and the meaning score was 0.15. Keeping the original's wording buys almost no faithfulness. Faithfulness comes from the model understanding the text, and copying is just a habit that happens to carry the watermark.

What it means

For someone trying to hide AI use: the rewriter's temperature is a real lever, and it is free up to the nucleus cut. A hot rewrite with top-p left at its default removed about half of what the default setting left behind, at no measurable cost. Past the nucleus cut there is almost nothing more to gain, and the text starts losing facts. Deterministic rewriting is the worst choice.

For someone trying to detect AI use: the attacker has a knob you cannot see, and the first half of its range costs them nothing. The trace that survives the best attacker setting here is about one rewrite in ten at a 1% false-positive rate. Your pooled statistics still work; your per-document verdicts get weaker.

For the 5-gram proxy from the first two posts: it held again. Every setting that lowered overlap lowered the signal, in order. If you cannot run a detector, counting surviving 5-grams still tells you most of the story.

Caveats

One rewriter, one prompt, one seed per setting. Temperature 0 is deterministic by construction. A different rewriter could have a different nucleus cut where the text breaks.
One judge per item. No second rater, no agreement check. The 1-to-5 scale is coarse, so read the paired intervals, not the raw means. The baseline is not spotless either; the judge caught the 0.7 run turning a tea price of "a crown per pound" into "one pound per crown," and two other settings turning a 17-year-old bistro into a 17-year-old owner.
100 texts per setting. The flagged-rate intervals are wide. The AUC columns and the paired z differences are the stronger evidence.
I did not vary top-k, or the generator's own temperature. The watermark strength at generation time was fixed at the first post's setting.
The Ollama model is 4-bit quantized, as before.

The pipeline

Four short scripts on top of the first post's code: one runs the rewrites per setting through Ollama's HTTP API, one scores them with the same detector and counts surviving 5-grams, one does the paired analysis against the stored cutoffs, and one builds the blinded judge batches and collects the scores. About three hours of GPU time, nearly all of it rewriting.

Thanks to the reader who asked. It was the right question, and the answer was not the one I would have guessed: the knob works, but the cliff is in top-p, not temperature.

Fancy prompts don't confuse small LLMs. One format line does

vadim albarov — Tue, 13 Oct 2026 14:00:00 +0000

Ask an LLM to "improve" your prompt and you often get something like this:

Kindly undertake a meticulous examination of the order correspondence reproduced hereunder and distill therefrom the following salient particulars...

Does a small local model still understand that? And the other way round: if you write in very simple words, short sentences, like for a child, does that help a 1B model?

In my last test I broke prompts with typos and bad punctuation. This time the spelling is perfect. Only the register changes: plain, normal, or ornate. I ran every version on eight local models from 0.6B to 14B, the biggest that fit in my 16 GB of VRAM.

The short answer: the register itself barely matters. What matters is one line, the one that tells the model how to format its answer.

What I tested

50 tasks, each with an answer a script can check. Five kinds, ten tasks each:

Math: multi-step word problems, answer is one number.
Extract: an order note, return JSON with five fixed keys.
Classify: a support ticket, return one of four labels.
Filter: a list of records, return the ones that match a condition, sorted. Each list has a trap at the edge: an employee aged exactly 30 for "older than 30", a flight at 14:05 for "between 10:00 and 14:00".
Format: writing tasks with strict rules. "Exactly 3 bullets, no word health." "Write yes 7 times."

Each task exists in three versions:

	plain	normal	ornate
example	Give only the number.	Answer with the number only.	Furnish solely the numerical figure, without accompanying exposition.
reading grade (Flesch-Kincaid)	4.1	8.3	17.9
words per sentence	9.8	12.3	22.6
words with 9+ letters	1%	4%	26%

The facts, numbers, and conditions are the same in all three versions. A Claude model checked every triple for differences in meaning before the run (it missed one, more on that below). Data you would paste, like the order note or the record list, is identical across versions. Only the words around it change.

Models, all through Ollama with thinking off: Qwen3 0.6B, 1.7B, 4B-instruct, 8B, 14B and Gemma3 1B, 4B, 12B. Three trials per prompt with fixed seeds. 3,600 calls in the main run, plus 1,440 for a follow-up on math.

Result 1: outside math, register makes no difference you can trust

Here are the 40 non-math tasks. Percent correct, and the difference from normal with a 95% interval.

model	plain	normal	ornate	plain − normal	ornate − normal
qwen3 0.6B	31	39	39	−8 [−21, +3]	+0 [−9, +9]
gemma3 1B	52	61	53	−9 [−19, +1]	−8 [−19, +4]
qwen3 1.7B	68	68	60	+1 [−8, +9]	−8 [−17, +2]
qwen3 4B-instruct	82	89	79	−8 [−18, +2]	−10 [−22, +1]
gemma3 4B	69	70	68	−1 [−8, +7]	−2 [−12, +8]
qwen3 8B	73	75	74	−2 [−9, +6]	−1 [−9, +7]
gemma3 12B	78	74	78	+4 [−4, +13]	+3 [+0, +9]
qwen3 14B	78	76	77	+3 [−5, +10]	+1 [−8, +9]

From 8B up, the three versions land within 4 points of each other. Ornate English with 23-word sentences did not hurt them at all.

At 4B and below there is a pattern: normal scores highest, and both plain and ornate lose about 8 points. Every one of those intervals touches zero, so I can't call it proven. But it points one way. For a small model, very simple language is not a safe choice either. My plain prompts were slightly longer than the normal ones (more, shorter sentences), and the facts were spread over more sentences.

Result 2: math swings 30 points, and it is not about reading

Math looked like the big story. Pooled over all models: plain 28%, normal 34%, ornate 18%. qwen3 4B-instruct went from 93% on normal to 43% on ornate.

Then I looked at the replies. When a model wrote out its working, it was right 79% of the time. When it gave a bare number, it was right 11% of the time. These models have thinking off, so writing the steps is the thinking.

And the format line decided how often they wrote the steps:

format line	replies with working shown
"Answer with the number only." (normal)	37%
"Give only the number." (plain)	27%
"Furnish solely the numerical figure, without accompanying exposition." (ornate)	8%

The ornate prompts didn't confuse the models. The models obeyed them more strictly, and that stopped them from reasoning.

To separate the two effects, I kept each problem in its register but gave all three versions the same last line:

setup	plain	normal	ornate
each register's own format line	28	34	18
A: all end with "Answer with the number only."	37	35	40
B: all end with "Work it out step by step, then write the final number on the last line."	84	83	79

With the same last line, the register of the problem text moves the score by 5 points at most. Changing the last line moves it by 45.

The strangest part: two plain phrasings with the same meaning. qwen3 1.7B showed its working in 17% of replies to "Give only the number." and in 80% to "Answer with the number only." Same model, same problems, almost the same words.

Result 3: fancy words can change what you asked

One extraction task had this note: "Hi, this is Laura Becker from Becker Design. We'd like 12 desk lamps..."

Plain ("the name of the customer"): 88% answered Laura Becker.
Normal ("customer name"): 96%.
Ornate ("the appellation of the commissioning customer"): 25%. Most models answered Becker Design.

That is not really a wrong answer. A "commissioning customer" can well be the company. The ornate wording made the question vaguer, and the models picked the other reading. The Claude reviewer I used to check the triples for equal meaning did not flag it either. This is exactly the risk with prompts an LLM has rewritten: each fancy word is a small chance that the meaning moved.

Is it just what the model was trained on?

The theory. Instruction-tuning data is mostly ordinary, direct English. So "normal" prompts should look the most familiar to a model, and both very simple and very ornate English should look less familiar. Small models cope worse with unfamiliar input. That would explain why small models did best on normal, and why big models did not care.

The test. Perplexity measures how surprised a model is by a piece of text. Lower means more familiar. I scored the instruction wording of all 150 prompts with the same Qwen3 models (0.6B, 1.7B, 4B-instruct), leaving out the pasted data that is identical in all three versions.

model	plain	normal	ornate
qwen3 0.6B	15.6	16.7	37.2
qwen3 1.7B	15.9	16.3	31.2
qwen3 4B-instruct	17.4	17.4	31.9

Then I checked, task by task, whether the less familiar version was also the one the model got wrong more often. Outside math, the link was weak (r between −0.15 and +0.08) and not distinguishable from zero for any model. Inside math it was clear for the 4B (r = −0.41). But that is the format line again: the ornate "furnish solely the numerical figure" is both the most unfamiliar text and the line that stopped the model from reasoning.

The explanation. The theory is half right.

Ornate English really does look foreign to these models: about twice the perplexity. But they understand it anyway. Unfamiliar words alone barely cost accuracy.
Plain English looks exactly as familiar as normal. Yet small models still did a bit worse on it. So familiarity can't be the reason. The likely reason is structure: my plain versions split each task into more, shorter sentences, and a small model has to join the facts back together.

So it is less about which words you use and more about how the task is laid out, and how strictly the model reads the format line.

Final thoughts

Don't bother rewriting a prompt into fancier English. It won't help, and it can shift the meaning.
Don't dumb it down for small models either. Normal, direct English did as well as anything, and slightly better at 4B and below.
Spend your effort on the format line. "Number only" on a model without thinking tells it to guess. If you want a correct number, let it show its work, and read the last line.
If an LLM rewrote your prompt, read the rewritten version for words that could mean two things.

Caveats

50 tasks, three trials per prompt, fixed seeds, Ollama default temperatures. The registers were written by one person (with help from an LLM), so another writer's "ornate" could be harder or easier. The ornate prompts are longer than normal ones, so length and register are partly mixed. Only English. Only local models up to 14B with thinking off. Frontier models were not tested this time; my typos test suggests they would shrug all of this off. The perplexity check covers only the three smaller Qwen models, scored with full-precision weights, while the runs used Ollama's 4-bit builds. The "structure" explanation for plain is my best reading, not something this test isolated.

Repo with the prompts, the harness, and every raw reply: github.com/valbarov/prompt-register-test

Your Laptop Speakers Pop When You Pause? It's Not a Bug, It's Power Saving

vadim albarov — Sat, 10 Oct 2026 13:00:00 +0000

My ASUS Zephyrus has great speakers and one annoying habit: a pop every time sound starts or stops. Pause a video: pop. Play again: pop. Click "skip 10 seconds" five times in a row on YouTube: pop-pop-pop-pop-pop.

This is not an ASUS problem, and not a problem of one model. Many modern laptops have it. It seems most common on laptops with good speakers, where the speakers are driven by power-saving "smart amps". I have only tested one laptop, so take that part as a guess.

This post shows how I found the cause and fixed it with a small PowerShell script. No admin rights, no driver hacks.

The hardware

The speakers are not driven by a classic onboard codec. Device Manager shows a SoundWire setup:

Cirrus Logic CS35L56 smart amps (left and right speakers)
Cirrus Logic CS42L43 codec (headphone jack, mics)
Intel Smart Sound Technology over SoundWire

Smart amps like the CS35L56 save power. When the audio line goes silent, they shut down. When sound comes back, they power up again. On this laptop, that power change is loud enough to hear as a pop.

Step 1: Find out what causes the pop

There are two common causes:

The stream – the app opens or closes an audio stream, and the DSP mutes/unmutes.
The power-down – the amp goes to sleep on silence and wakes up on sound.

To tell them apart, I wrote a test script (Test-SilentKeeper.ps1) that plays a looping file of pure digital silence while I use the laptop as normal.

The pop was still there. That was a useful clue: pure zeros keep the Windows audio stream open, but the amp still detects silence and goes to sleep. So the amp itself watches the signal level.

Step 2: What did not work

Most forum posts say "disable power saving for the audio device". I tried that in the registry for the three SoundWire amp devices:

HKLM\SYSTEM\CurrentControlSet\Control\Class\{4d36e96c-e325-11ce-bfc1-08002be10318}\<instance>\PowerSettings
  ConservationIdleTime = FF FF FF FF
  PerformanceIdleTime  = FF FF FF FF
  IdlePowerState       = 00 00 00 00

After a reboot: no change. The amp's silence gate runs inside the amp and its firmware, not through Windows device power management. I reverted it.

Step 3: Keep the sound device busy... all the time

The next idea: keep the sound device busy with a signal so quiet that you can't hear it, but loud enough that the amp doesn't see "silence".

There are small tools that do exactly this. I used SoundKeeper (open source, MIT). Any similar tool that plays a steady, inaudible signal to the output device should work the same way. The important part is that it plays a real signal, not pure zeros, because the amp ignores zeros.

This is not an advertisement. I have no ties to SoundKeeper or its author. It is simply the tool I found first, and it worked.

With SoundKeeper, the command is:

SoundKeeper64.exe sine f10 a5

That is a 10 Hz sine wave at 5% volume. Humans can't hear 10 Hz, and laptop speakers can't really play it anyway. But to the amp, it is "not silence", so it stays awake.

Result: no more pops.

The downside: the amps never sleep. That costs battery, and it feels wrong to keep the amps on all day just to avoid a pop when I pause a video.

Step 4: Keep the amp awake only when it matters

What I want:

When I'm watching or listening to something, keep the amp awake, so pause, seek and skip don't pop.
When nothing has played for a while, let the amp sleep.

So I wrote AmpPopGuard: a watchdog that

reads the peak level of the default output device every 200 ms,
starts the keeper tool (SoundKeeper in my case) when it hears real audio,
stops it after 10 seconds with no real audio.

Rapid seeking now happens inside the 10-second window, so the amp never sleeps between clicks. After real silence, you get one pop on the next cold start, and that's it.

Reading the peak meter from PowerShell

Windows exposes the output level through the Core Audio API (IAudioMeterInformation). It's the same meter you see in the Volume Mixer. PowerShell can't call it directly, but Add-Type with a few lines of C# COM interop works fine:

[Guid("C02216F6-8C67-4B5B-9D00-D008E73E0064"),
 InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
interface IAudioMeterInformation
{
    int GetPeakValue(out float pfPeak);
}

Get the default render device from IMMDeviceEnumerator, Activate the meter, and call GetPeakValue – you get a float from 0.0 to 1.0.

The trap: the script hears itself

The first version had a bug. The keeper's own 10 Hz sine shows up on the same peak meter at about 0.01–0.02. With a single threshold of 0.01, the watchdog started the keeper, then "heard audio" (its own sine), and never stopped it.

The fix is two thresholds:

State	Counts as real audio when peak >
Keeper off	0.01
Keeper on	0.05

While the keeper runs, only sound clearly louder than its own sine resets the 10-second timer.

The full script

# AmpPopGuard.ps1
# Prevents the Cirrus amp pop on rapid pause/seek WITHOUT keeping the amps
# awake all the time.
# Only one instance at a time
$script:mutex = New-Object System.Threading.Mutex($false, 'Global\AmpPopGuard')
if (-not $script:mutex.WaitOne(0)) { exit }

$KeeperExe     = Join-Path $PSScriptRoot 'SoundKeeper64.exe'   # script lives next to the exe
$KeeperArgs    = 'sine', 'f10', 'a5'   # 10 Hz sine @ 5% - inaudible, defeats the amp's silence gate
$LingerSeconds = 10      # keep amps awake this long after last real audio
# Two thresholds: the keeper's own 10 Hz sine reads ~0.01-0.02 on the meter,
# so while it runs, only levels above ThresholdOn count as real audio.
$ThresholdOff  = 0.01    # start trigger while keeper is off
$ThresholdOn   = 0.05    # sustain trigger while keeper is running
$PollMs        = 200

if (-not (Test-Path $KeeperExe)) {
    Write-Error "SoundKeeper not found at $KeeperExe"
    exit 1
}

# --- Core Audio peak meter interop -----------------------------------------
Add-Type -TypeDefinition @'
using System;
using System.Runtime.InteropServices;

[ComImport, Guid("BCDE0395-E52F-467C-8E3D-C4579291692E")]
class MMDeviceEnumeratorComObject { }

[Guid("A95664D2-9614-4F35-A746-DE8DB63617E6"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
interface IMMDeviceEnumerator
{
    int NotImpl1();
    int GetDefaultAudioEndpoint(int dataFlow, int role, out IMMDevice ppDevice);
}

[Guid("D666063F-1587-4E43-81F1-B948E807363F"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
interface IMMDevice
{
    int Activate(ref Guid iid, int dwClsCtx, IntPtr pActivationParams,
        [MarshalAs(UnmanagedType.IUnknown)] out object ppInterface);
}

[Guid("C02216F6-8C67-4B5B-9D00-D008E73E0064"), InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
interface IAudioMeterInformation
{
    int GetPeakValue(out float pfPeak);
}

public static class AudioMeter
{
    static IAudioMeterInformation meter;

    public static float GetPeak()
    {
        if (meter == null)
        {
            var en = (IMMDeviceEnumerator)new MMDeviceEnumeratorComObject();
            IMMDevice dev;
            en.GetDefaultAudioEndpoint(0, 1, out dev);   // eRender, eMultimedia
            var iid = typeof(IAudioMeterInformation).GUID;
            object o;
            dev.Activate(ref iid, 23, IntPtr.Zero, out o); // CLSCTX_ALL
            meter = (IAudioMeterInformation)o;
        }
        float peak;
        meter.GetPeakValue(out peak);
        return peak;
    }

    public static void Reset() { meter = null; }  // call after device change errors
}
'@

function Stop-Keeper {
    Get-Process -Name 'SoundKeeper64' -ErrorAction SilentlyContinue | Stop-Process -Force -ErrorAction SilentlyContinue
}

# Clean slate: no stray keeper from a previous run
Stop-Keeper

# --- Watchdog loop ----------------------------------------------------------
$keeperOn = $false
$lastLoud = [DateTime]::MinValue

Write-Host "AmpPopGuard running (linger ${LingerSeconds}s, thresholds $ThresholdOff/$ThresholdOn). Keeper: $KeeperExe"

try {
    while ($true) {
        try {
            $peak = [AudioMeter]::GetPeak()
        } catch {
            # default device changed / audio engine restarted - reconnect
            [AudioMeter]::Reset()
            Start-Sleep -Milliseconds 1000
            continue
        }

        $now = [DateTime]::UtcNow
        $threshold = if ($keeperOn) { $ThresholdOn } else { $ThresholdOff }
        if ($peak -gt $threshold) {
            $lastLoud = $now
            if (-not $keeperOn) {
                Start-Process -FilePath $KeeperExe -ArgumentList $KeeperArgs -WindowStyle Hidden | Out-Null
                $keeperOn = $true
            }
        } elseif ($keeperOn -and ($now - $lastLoud).TotalSeconds -gt $LingerSeconds) {
            Stop-Keeper
            $keeperOn = $false
        }

        Start-Sleep -Milliseconds $PollMs
    }
} finally {
    Stop-Keeper
}

Put SoundKeeper64.exe in the same folder as the script. Get it from the GitHub releases or the project page. I use v1.3.6. To use a different keeper tool, change $KeeperExe, $KeeperArgs and the process name in Stop-Keeper.

Start it at login, hidden

PowerShell's -WindowStyle Hidden still flashes a window for a moment. A small VBS launcher in the Startup folder avoids that. Save as
%APPDATA%\Microsoft\Windows\Start Menu\Programs\Startup\AmpPopGuard.vbs:

Set sh = CreateObject("WScript.Shell")
sh.Run """C:\path\to\pwsh.exe"" -NoProfile -ExecutionPolicy Bypass -WindowStyle Hidden -File ""C:\path\to\AmpPopGuard.ps1""", 0, False

To stop it:

Get-CimInstance Win32_Process -Filter "Name='pwsh.exe'" |
  ? CommandLine -match AmpPopGuard.ps1 | % { Stop-Process -Id $_.ProcessId }

Trade-offs

One pop is left: the first sound after more than 10 s of silence still pops. That pop is part of how the amp works.
Cost: one COM call every 200 ms. CPU use is near zero.
Tuning: $LingerSeconds sets how long the amp stays awake. I use 10. The thresholds may need small changes on other hardware.
Device changes: if you switch outputs (headphones, Bluetooth), the meter call fails, and the script reconnects to the new default device.

Does this apply to your laptop?

Probably, if you have:

a pop or click when audio starts or stops,
that gets worse when you seek or skip quickly,
and "disable power saving" tweaks don't help.

Many modern laptops from different brands use smart amps (Cirrus Logic CS35Lxx, TI TAS, Realtek and others). These amps often have the same silence gate. Higher-end laptops with better speakers seem to use them more, but I am not sure how wide it goes. The script does not depend on the brand or the model. It only watches the Windows peak meter.

Have you seen this on other laptops? Tell me in the comments.

Your AWS Backup vault is locked. Who holds the key?

vadim albarov — Thu, 08 Oct 2026 16:59:12 +0000

AWS Backup Vault Lock in compliance mode is a strong guarantee. Once the cooling-off period ends, nobody can delete a recovery point early. Not an admin, not root, not AWS Support. For regulated workloads it is the closest thing to an offline tape you can get in the cloud.

We recently moved our RDS backups to this setup: an AWS Backup plan, a local vault, and a copy of every recovery point to a locked vault in a second region. Then a reviewer asked one question that undid the whole feeling of safety.

What protects the KMS key that encrypts the locked vault?

Nothing did. The vault key was a normal customer-managed key with the default policy. Any administrator in the account could disable it or schedule it for deletion. The recovery points would survive, locked and immutable, but nobody could decrypt them. A backup you cannot read is not a backup.

This post is about what we tried, what failed in a way the docs would have told us, and what we ended up with.

Step 1: Deny the destructive actions in the key policy

Three KMS actions can make a key unusable: kms:ScheduleKeyDeletion, kms:DisableKey, and kms:PutKeyPolicy. The third one matters because anyone who can rewrite the policy can remove the other two denies.

A key policy can deny these to everyone except a single break-glass principal:

statement {
  sid       = "DenyKeyDestruction"
  effect    = "Deny"
  actions   = ["kms:ScheduleKeyDeletion", "kms:DisableKey", "kms:PutKeyPolicy"]
  resources = ["*"]
  principals {
    type        = "AWS"
    identifiers = ["*"]
  }
  condition {
    test     = "StringNotEquals"
    variable = "aws:PrincipalArn"
    values   = [var.break_glass_role_arn]
  }
}

Two details are easy to miss here.

First, this deny applies to the account root user too. The key policy is always evaluated, and an explicit deny in it beats the default root allow.

Second, KMS will refuse to create this key. The lockout safety check rejects any policy that denies PutKeyPolicy to the principal creating the key. You have to pass bypass_policy_lockout_safety_check = true. That flag exists for exactly this reason: if the exempt principal turns out to be unusable, the key becomes permanently unmanageable. Remember that sentence.

Step 2: Who is the exempt principal?

Our first exempt principal was the existing key-owners role in the same account. The reviewer rejected it in one line: any DevOps admin can assume that role. The deny adds one extra API call to an attack, not a wall.

Inside a single account, this is a hard problem. An admin with AdministratorAccess can edit any trust policy, attach any policy, and enrol a new MFA device on a stolen user. Whatever principal you exempt, the account can grant itself access to it.

So we wanted the exempt principal outside the account. A break-glass role in a separate, tightly controlled account, assumable only with MFA by two named people. Prod credentials could never reach it.

Step 3: The test that saved us

We did not apply this to the real key. We created a throwaway key with the same policy, plus an Allow statement granting the external break-glass role the key-management actions, and tried the role against it.

aws kms put-key-policy --key-id <test-key> --policy-name default --policy file://policy.json

An error occurred (AccessDeniedException) when calling the PutKeyPolicy operation:
This operation cannot be called cross account

Same error for DisableKey, EnableKey, and GetKeyPolicy.

This is in the KMS documentation and has been for years. A key policy can grant principals in other accounts permission to use a key: encrypt, decrypt, generate data keys, describe it, create grants. It cannot let them administer it. Key-management operations are refused cross-account, whatever the key policy says.

Our throwaway key is now permanently unmanageable. Every principal in the account is denied by the key policy. The only exempt principal is in another account and cannot call the API. AWS Support is the documented way to recover such a key. It costs a dollar a month and encrypts nothing, so we left it as a monument.

Picture that on the real vault key, holding every copy of a production database.

Step 4: The design that works

The exempt principal must live in the same account as the key. The question becomes: can we make a same-account role that no same-account identity can assume?

Yes. Trust policies decide who can assume a role, and IAM permissions cannot override a trust policy. A role whose trust policy names only a principal from another account is unreachable from inside, no matter how many admin policies the caller holds.

resource "aws_iam_role" "dr_key_break_glass" {
  name = "${var.env}-dr-key-break-glass"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Action    = "sts:AssumeRole"
      Principal = { AWS = var.external_break_glass_role_arn }
    }]
  })
}

resource "aws_iam_role_policy" "dr_key_break_glass" {
  role = aws_iam_role.dr_key_break_glass.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Action = [
        "kms:DescribeKey", "kms:GetKeyPolicy", "kms:PutKeyPolicy",
        "kms:DisableKey", "kms:EnableKey",
        "kms:ScheduleKeyDeletion", "kms:CancelKeyDeletion"
      ]
      Resource = aws_kms_key.dr_vault.arn
    }]
  })
}

The key policy deny exempts this role's ARN. The creating principal is still denied PutKeyPolicy, so the bypass flag stays. The role is not listed as a principal anywhere in the key policy; the default root statement lets IAM policies take effect, and the inline policy above does the rest. That also avoids a race on first apply, where KMS rejects a key policy naming an IAM principal that was created seconds ago.

The legitimate path is two hops. A named user assumes the external break-glass role with MFA. That session assumes the in-account role. That session administers the key. Role chaining drops the MFA context, so the MFA condition lives on the first hop only.

We tested again on a second throwaway key:

Caller	Action	Result
Account admin	AssumeRole into the in-account role	AccessDenied
Account admin	DisableKey, ScheduleKeyDeletion, PutKeyPolicy	Explicit deny
Two-hop chain	GetKeyPolicy, PutKeyPolicy	OK
Two-hop chain	DisableKey, then EnableKey	OK
Two-hop chain	ScheduleKeyDeletion	OK, used for cleanup

What an attacker has left

One path. An admin in the account can edit the in-account role's trust policy to name themselves. That is an IAM write, iam:UpdateAssumeRolePolicy on a specific role name. IAM management events land in CloudTrail by default, so an EventBridge rule can page someone the moment it happens. Compare that with the original design, where the attack was a plain AssumeRole on a role the team uses every day.

To close that last path you need a control the account cannot undo. A Service Control Policy on the organizational unit, denying those IAM writes on the role and the three KMS actions on the key to every principal except the break-glass role, does it. An SCP mistake is reversible from the management account. A key policy mistake is not. A dedicated backup account that owns the vault and the key is the stronger end state. Both need the org management account, which for us meant a ticket to a partner. The in-account role shipped in the meantime.

Operational costs, stated plainly

Any Terraform apply that changes the key policy must run through the two-hop chain. Adding a new consumer does not, because the local admin role can still issue grants.
Destroying the module needs the chain too.
If the external break-glass role is deleted and recreated, the trust policy keeps the old principal ID. The next apply fixes it, but the chain is dead until then.
The deny matches the role ARN as a string. Recreating the in-account role with the same name keeps working. Renaming it freezes the policy.

Three habits

Read the current docs before designing an IAM control. The design came from Claude Code, which answered from memory. The cross-account rule was one paragraph away, on a page neither of us read.

Test anything that can lock a resource on something disposable. The lockout safety check is a warning, not an obstacle. If you bypass it on purpose, prove that the principal you trust can actually act, before the real key exists.

Ask what protects the thing that protects the thing. A locked vault, a versioned bucket, an immutable snapshot. Each one depends on a key, a policy, or a role that someone can still change. Follow the chain until you reach something the account itself cannot undo, and if you cannot reach it, know exactly which step is left open.

I Watermarked 100 Texts With SynthID and Asked Local LLMs to Wash It Out. The Detector Still Saw Some of It.

vadim albarov — Tue, 06 Oct 2026 15:00:25 +0000

In my last post I asked whether a local LLM can rewrite an AI text enough to break its word patterns while keeping the meaning. The answer was yes, with a good 12–14B model. But I only estimated watermark removal by counting how many word sequences survived. I had no real watermark and no real detector.

This time I had both.

The question: if a text carries Google's SynthID-Text watermark, and you run it through a local model with "rewrite this in your own words," can a detector still tell?

Short answer: mostly no, but not entirely. The standard detector flagged 3% to 15% of the rewrites, depending on the rewriting model. A simpler, more sensitive test flagged 23% to 41%. For long texts rewritten by the most faithful model, it flagged 89%. The model that kept the meaning best in the last test also kept the most watermark. And rewriting a second time did not help at all.

What SynthID-Text does

SynthID-Text is Google DeepMind's text watermark. It ships in Hugging Face transformers. It changes nothing about the model weights. While the model writes, a logits processor looks at the last 4 tokens, uses them plus a secret key to split the vocabulary into "preferred" and "not preferred" tokens, and nudges the sampling toward the preferred ones. The text still reads normally.

A detector needs only the key and the tokenizer, not the model. For each token it asks: given the 4 tokens before it, was this a preferred token? In normal text that is true about half the time. In watermarked text it is true more often. SynthID calls this score the g-value. Human text has a mean g-value of 0.500. Watermarked text sits above it.

Two ways to turn g-values into a verdict:

The Bayesian detector. This is the one in the Hugging Face example. It is a tiny model (about 900 parameters) that you train on watermarked and clean text. It outputs a probability that the text is watermarked.
The mean test. Just check whether the mean g-value is above 0.5 by more than chance allows. I report it as a z-score. Under no watermark the z-score is about 0; a z of 2.3 or more happens in about 1% of clean texts.

The setup

Everything ran on one laptop GPU with 16 GB of VRAM.

1. A watermarked writer. Gemma is gated on Hugging Face and I was not logged in, so I used Qwen/Qwen3-4B-Instruct-2507 in bf16 with the SynthID config from the Hugging Face example: 30 keys, n-gram length 5, temperature 1.0, top-k 40. Everything you need is watermarking_config=SynthIDTextWatermarkingConfig(...) passed to model.generate().

2. A detector. I followed the training recipe from transformers/examples/research_projects/synthid_text, minus TensorFlow:

Watermarked examples: 1,200 answers to "explain like I'm five" questions from the ELI5 dataset, about 290 tokens each.
Clean examples: 3,000 English Wikipedia articles and 480 answers to other ELI5 questions written by the same model without the watermark.
70/30 train/validation split, 100 epochs, 6 minutes of training.

On the validation set, the detector caught 99.7% of watermarked 200-token snippets at a 1% false-positive rate, and 98.1% at 0.1%.

3. The test texts. The same 100 topics as last time: 10 genres (news, technical explainers, science, opinion, personal stories, marketing, how-to, business memos, history, short fiction), 190 to 900 words each. The watermarked model wrote all 100. It also wrote the same 100 without the watermark, as a control.

4. The rewrites. The same three Ollama models and the same prompt as last time:

qwen3:4b-instruct (14 minutes for 100 texts)
gemma3:12b (27 minutes)
qwen3:14b, thinking off (50 minutes, including a slow patch; the median was 15 seconds per text)

After the first round I also ran a second pass: Gemma rewrote its own 100 rewrites once more, same prompt (37 minutes).

Rewrite the text below completely in your own words.
- Change the vocabulary and the sentence structure; do not copy phrases.
- Keep ALL of the meaning: every fact, name, number, date, step, argument, and plot event.
- Do not add new information, comments, or opinions.
- Keep the same genre, tone, and roughly the same length. Keep the title line, rewritten.
Output only the rewritten text, nothing else.

5. The scoring. I tokenized every text with the Qwen tokenizer and ran both detectors. The "flagged" cutoffs were set so that 1% of 500 clean control texts get flagged. The controls were the 100 unwatermarked Qwen texts, 300 held-out Wikipedia articles, and the 100 Claude-written texts from my last post.

Does the watermark work at all?

Yes, very well. This is the baseline the rewrites have to beat.

Text	n	Flagged (Bayesian)	Flagged (mean test)	Mean g-value
Watermarked originals	100	100%	100%	0.546
Same prompts, no watermark	100	1%	2%	0.500
Wikipedia (human)	300	1.3%	1.0%	0.500
Claude-written texts (other AI)	100	0%	0%	0.500

Every watermarked text scored a probability of 1.0 or close to it. The weakest original, a 330-token piece, still came in at 0.99994. The watermark also shows up fast: using only the first 100 tokens of each original, 94% were flagged; at 200 tokens, 100%.

The control numbers are the point of reference below. "1%" means "no better than chance."

Does a rewrite remove it?

Mostly.

Rewriter	Flagged (Bayesian)	Flagged (mean test)	Mean g-value	Mean z	AUC (mean test)	5-grams kept from original
gemma3:12b	3%	26%	0.5054	1.56	0.79	10%
qwen3:4b-instruct	10%	23%	0.5056	1.50	0.79	11%
qwen3:14b	15%	41%	0.5087	2.52	0.87	18%
gemma3:12b, second pass	3%	38%	0.5063	1.79	0.83	14%

Four things to read from this table.

The standard detector lost most of the trail. The Bayesian detector, which is the one you would deploy, flagged 3% of Gemma's rewrites. That is close to the 1% you would get on clean text. Across all 300 rewrites, only 18 of the 100 originals had even one rewrite flagged by it.

But the watermark did not fully disappear. Every rewriter's mean g-value sits above the clean baseline of 0.500. The difference is small for any single text, which is why the Bayesian detector rarely fires. Pooled over 100 texts it is unmistakable: a z of 17 for Gemma, 16 for the 4B model, and 28 for the 14B model. Clean controls pooled the same way give 0.2 to 1.2. The simple mean test, which is more sensitive to weak signals than the trained detector in this setup, flagged 23% to 41% of individual rewrites, and 53 of the 100 originals had at least one rewrite it caught.

The best meaning-keeper is the worst watermark-washer. Last time, qwen3:14b kept the meaning best (4.68 out of 5) but also kept the most original wording. Here it also kept the most watermark: 41% of its rewrites were flagged by the mean test, 15% by the Bayesian detector, and 8% of them scored above 0.95, which the Hugging Face example treats as "watermarked." Gemma, the heaviest rewriter, washed out the most.

A second rewrite did not remove more. It removed less. I expected that rewriting the rewrite would halve the trace again. It went the other way. Gemma's second pass scored a little higher than its first: mean z 1.79 versus 1.56, flagged by the mean test 38% versus 26%, AUC 0.83 versus 0.79. The second pass was a real rewrite; it shared only 24% of its 5-grams with the first pass. But in 86 of 100 texts it shared more 5-grams with the watermarked original than the first pass did (14% versus 10%). When a model paraphrases a paraphrase, it tends to fall back on the most natural phrasing, and the most natural phrasing is often what the original model picked too. The watermark lives in exact token sequences, so landing back on the original's phrases means landing back on its watermark. The per-text increase is small (paired t of 1.8, so not clearly significant), but there is no sign of a decrease.

Longer texts are easier to catch

A watermark detector adds up evidence token by token. A rewrite leaves a little evidence per token, so the longer the text, the more it adds up.

Original length	Rewrites	Flagged (mean test)	Flagged (Bayesian)	Mean z
150–250 words	99	8%	7%	0.9
300–450 words	99	29%	8%	1.7
500–700 words	75	45%	12%	2.6
750–900 words	27	70%	15%	3.9

By model, for the 750–900 word texts: qwen3:14b rewrites were flagged 89% of the time, gemma3:12b 78%, qwen3:4b-instruct 44%. For 150–250 word texts, all three were near the noise floor (3% to 15%).

Fiction is the hardest to wash

Genre	Flagged (mean test)	Flagged (Bayesian)	5-grams kept
Short fiction	70%	47%	21%
Personal narrative	33%	23%	16%
Business documents	37%	3%	13%
How-to guides	33%	0%	13%
Marketing copy	30%	10%	13%
Science	27%	3%	11%
Opinion essays	20%	0%	10%
Technical explainers	20%	0%	11%
History	17%	0%	12%
News	13%	7%	12%

The models rewrote stories much more timidly than they rewrote explainers. They kept names, dialogue, and specific images, and with them the token sequences that carry the watermark. All three top-scoring rewrites in the whole test were qwen3:14b stories, each with a probability of 1.0.

Here is the opening of one, a time-travel story. The original:

It was 6:47 a.m. when Elias Varell stepped onto the cracked asphalt of Maple Street, his boots clicking against the frost-bitten sidewalk. The bus stop, a tattered wooden bench leaning at an angle like some tired old tooth, sat beneath a flickering traffic light that had failed to turn green for the last forty-one minutes.

The qwen3:14b rewrite:

It was 6:47 a.m. when Elias Varell arrived at the broken pavement of Maple Street, his boots making a tapping sound on the frozen path. The bus stop, a worn wooden bench leaning at an odd angle, stood under a flickering traffic light that had remained red for forty-one minutes.

It reads as a rewrite. But 51% of the original's five-word sequences survived, the longest copied run was 66 words, and the detector gave it a probability of 1.0. Gemma's rewrite of the same story got a probability of 0.0, with a mean g-value of 0.509 and a z of 3.3. Still above the noise, but a deployed detector would have let it through.

The n-gram proxy from last time was fair

Last time I used "share of the original's 5-grams that survive" as a stand-in for watermark removal. Now I can check it against the real thing. Across the 300 first-pass rewrites, the correlation between 5-grams kept and the watermark z-score was 0.77 (0.73 with the second pass included).

5-grams kept	Rewrites	Flagged (mean test)	Mean z
under 10%	133	11%	0.9
10–20%	112	27%	1.7
20–30%	41	78%	3.8
over 30%	14	100%	6.7

If a rewrite keeps less than a tenth of the original's five-word sequences, it is nearly clean. If it keeps more than a fifth, the watermark is probably still there.

What this means

For someone trying to hide AI use: one pass through a local 12B model removes the watermark well enough to beat the standard detector most of the time, especially on short, factual text. Long texts and stories are the exception. Rewriting twice does not close that gap; in this test it widened it slightly. What does reduce the trace is a rewriter that keeps less of the original wording, and shorter text.

For someone trying to detect AI use: the watermark is extremely robust to nothing-done and weak to paraphrase, which matches what the SynthID paper says. But "weak" is not "gone." A trace survives in nearly every rewrite. It is too faint to accuse any one short document, but strong enough to show up in a long document or across a body of work. If your detector sees 50 essays from one source, the pooled statistic will tell you something even when no single essay does.

For the "which local model" question: the trade-off from last post is now sharper. qwen3:14b keeps the meaning and keeps the watermark. gemma3:12b loses a little precision and loses the watermark. There is no free lunch in this set of three.

Caveats

One generator, one setting. Watermark strength depends on how much freedom the sampler has. Temperature 1.0 with top-k 40 is friendly to the watermark. A lower temperature, or a more deterministic task like code, would start weaker and wash out faster.
My detector is smaller than Google's recipe. The example recommends 10,000 examples per class; I used about 1,200 watermarked and 3,500 clean. Validation accuracy was 99.7%, so it is not underfit, but a bigger detector might be better calibrated on partial watermarks. The Bayesian detector's outputs were very polarized (near 0 or near 1), which is why the plain mean test caught more borderline rewrites.
The 1% false-positive cutoffs come from 500 clean texts. That is enough for a rough cutoff, not a precise one. The AUC numbers do not depend on a cutoff.
One seed, one prompt. Same as last time. The second pass was tested with one model only (Gemma on its own output). A second pass with a different model might behave differently.
The Ollama models are 4-bit quantized. Full-precision versions might rewrite slightly differently.
The detector needs the generator's tokenizer and key. That is how SynthID works in practice: the model owner runs the detector. A third party without the key cannot do any of this.

The pipeline

Everything is a few short Python scripts on top of transformers:

Generate watermarked and clean training text with model.generate(..., watermarking_config=...).
Compute g-values with SynthIDTextWatermarkLogitsProcessor.compute_g_values() and train a BayesianDetectorModel on them.
Generate the 100 test texts with the watermark, and 100 without.
Rewrite with Ollama over the local HTTP API (and once more for the second pass).
Tokenize everything with the generator's tokenizer, score with both detectors, and compare to the clean controls.

Total GPU time was a little over four hours, most of it generation and rewriting. Detection itself takes seconds.

Update: held-out cutoffs, a rewritten null, and genre at matched length

Readers pointed out three holes in the method above. All three were real. I fixed them and re-ran the numbers. The outcome did not change: a trace of the watermark survives every rewriter, and the pooled statistics are untouched. The per-text percentages moved by a few points, and each one now comes with an interval. Here is what was wrong and what the fixes showed.

Hole 1: the 1% cutoff was set and measured on the same texts

The cutoff was the 99th percentile of 500 clean texts, and the control table reported how those same 500 texts fell against it. That 1% was true by construction. With 5 texts above the line, the cutoff itself was shaky.

Fix. I pulled 1,200 more held-out Wikipedia articles, bringing the clean set to 1,700 texts, and split every clean set in half by text id. One half (850 texts) sets the cutoffs. The other half (850 texts) measures the false-positive rate. The calibrated z cutoff came out at 2.42; theory for a 1% rate says 2.33.

Held-out clean text	n	False positives (mean test)	95% CI	False positives (Bayesian)	95% CI
All	850	1.2%	0.6 to 2.1	1.3%	0.7 to 2.3
Same-domain subset	250	1.6%	0.6 to 4.0	0.8%	0.2 to 2.9

The operating point transfers. The rewrite table at the new cutoffs:

Rewriter	Flagged (Bayesian)	Flagged (mean test)	95% CI	Was
gemma3:12b	3%	25%	18 to 34	26%
qwen3:4b-instruct	7%	17%	11 to 26	23%
qwen3:14b	14%	39%	30 to 49	41%
gemma3:12b, second pass	3%	36%	27 to 46	38%

Originals with at least one rewrite caught by the mean test: 57 of 100 (was 53). The pooled z values and the AUCs do not depend on a cutoff and are unchanged. The honest reading is that every per-text percentage in this post is soft to about plus or minus 8 points. The ordering of the rewriters, and the gap between any of them and clean text, is not in doubt.

Hole 2: the clean controls were never rewritten

I compared rewritten watermarked text against clean text that had not been rewritten. If a rewriter nudges clean scores up on its own, some of the "residual watermark" would have been rewriter artifact.

Fix. I ran the 100 unwatermarked texts, which come from the same 100 prompts, through the identical pipeline: same three rewriters, same prompt, same temperature, same seed. The rewriters changed clean and watermarked text to the same degree (5-grams kept: 10% versus 10%, 10% versus 11%, 19% versus 18%).

Rewriter	Clean rewrites, mean z	Watermarked rewrites, mean z	Clean rewrites flagged	Watermarked rewrites flagged	AUC vs clean rewrites
none (originals)	0.00	12.13	2%	100%
gemma3:12b	-0.01	1.56	0%	25%	0.80
qwen3:4b-instruct	0.12	1.50	0%	17%	0.78
qwen3:14b	0.10	2.52	0%	39%	0.86

Rewriting does not move the clean distribution. The shift in mean z is at most 0.12 and a Mann-Whitney test against the unrewritten clean texts gives p of 0.40 or higher for all three. Not one of the 300 clean rewrites crossed the cutoff. The AUC against the rewritten null is the same as against the unrewritten null. Paired by prompt, the watermarked rewrite scored above its clean twin in 80% to 89% of pairs (paired t 7.5 to 10.0). Inside each length bucket the clean rewrites are flagged 0% of the time for every rewriter, while the watermarked ones climb with length as before. The residual is watermark, not rewriter.

Hole 3: the genre table mixed in a length effect

My fiction texts averaged 745 tokens, my technical explainers 513, and detection rises with length.

Fix. A least-squares fit of z on log2(tokens) plus genre over the 300 first-pass rewrites gives 1.64 z per doubling of length. Fiction's raw advantage over technical explainers is 3.22 z. After adjusting for length it is 2.38 z, with a 95% interval of 1.3 to 3.6 (10 fiction sources and 10 explainer sources, 3 rewrites each). About a quarter of the fiction effect was length. Three quarters is genre.

The interval comes from a bootstrap that resamples the 100 original texts, not the 300 rewrites. Each original has three rewrites, one per rewriter, and they share its topic, length, and watermark strength. Treating them as independent gave a standard error of 0.39 in the first version of this update. Grouped by source it is 0.57 (10,000 resamples; an earlier run with fewer resamples gave 0.58). The effect is still well clear of zero. (Edited October 9 after a reader pointed this out.)

Dropping any one of the 100 source texts and refitting leaves fiction first in all 100 refits, with a difference of 2.1 to 2.7 z. (Added October 18.)

Inside one bucket, where lengths match:

Bucket	Fiction: tokens, mean z, flagged	Technical explainer: tokens, mean z, flagged
150–250 words	336, 1.1, 0% (n=6)	334, 0.3, 0% (n=15)
300–450 words	632, 3.4, 67% (n=12)	610, 1.9, 33% (n=9)
500–700 words	937, 6.8, 100% (n=6)	812, 2.2, 33% (n=6)

Personal narrative is the only other genre with a clear length-adjusted effect (+1.3 z, interval 0.6 to 2.0, 10 sources). Every other genre sits within 0.7 z of explainers once length is accounted for, and none of their intervals excludes zero. All genres, length-adjusted, compared with technical explainers (10 sources × 3 rewrites per genre, 95% intervals from the grouped bootstrap):

Genre	Sources	Rewrites	Difference vs explainer (z)	95% CI
fiction (short story)	10	30	+2.38	1.30 to 3.56
personal narrative	10	30	+1.30	0.59 to 2.02
product description / marketing	10	30	+0.29	-0.49 to 1.09
business / professional	10	30	+0.13	-0.74 to 0.98
how-to guide	10	30	+0.01	-0.70 to 0.66
history / culture	10	30	-0.14	-0.86 to 0.52
news report	10	30	-0.30	-1.06 to 0.42
science popularization	10	30	-0.46	-1.28 to 0.37
opinion essay	10	30	-0.68	-1.38 to 0.01

These are differences from explainers, not tests of whether the watermark survived in a genre. An interval that includes zero means that, with 10 sources, I cannot tell that genre apart from explainers. It does not mean no watermark survived there. The survival claim in this post is per rewriter, pooled over all genres. The genre result is narrower: fiction and personal narrative keep more of the watermark than explainers at the same length. (Edited October 12 after the same reader asked for source counts next to the intervals.)

The clean rewrites show no genre pattern at all (fiction clean rewrites: mean z 0.20), so this is in the watermark, not in how the rewriters treat stories.

Sensitivity check: leave one source out

The same reader asked whether the fiction-versus-explainer ordering is shared across the fiction texts or carried by a few of them. A genre mean from 10 sources can move a lot if one strong source is removed. So I refit the same length-adjusted model 100 times, each time dropping one of the 100 original texts together with its three rewrites.

Fiction stays on top in all 100 refits. Its length-adjusted difference from explainers ranges from 2.14 to 2.70 z (full sample 2.38). The low end comes from dropping t093, the story with the strongest surviving trace (mean z 7.3 over its three rewrites). The high end comes from dropping t095, the story with the weakest (mean z 0.3). Even the lowest value sits above the lower bound of the grouped bootstrap interval (1.30). Personal narrative stays second in all 100 refits, between 1.12 and 1.47 z.

Fiction source dropped	Fiction difference vs explainer (z)
none (full sample)	+2.38
t091	+2.49
t092	+2.48
t093	+2.14
t094	+2.36
t095	+2.70
t096	+2.56
t097	+2.23
t098	+2.48
t099	+2.25
t100	+2.26

Dropping any one explainer source moves the fiction difference between 2.27 and 2.56. The middle genres are a different story. The how-to guide estimate changes sign in 12 of the 100 refits, and business and history each flip once. Those three sit within 0.15 z of explainers, so a sign flip there means nothing. It just restates that their intervals include zero.

This is a sensitivity check on the texts I have, not more evidence about fiction in general. With 10 fiction sources the population claim stays as wide as the bootstrap interval says. What the check adds is that the ordering inside this sample does not hinge on one or two stories. The pooled survival result per rewriter is not part of this check and does not change. (Added October 18 after the same reader asked for a leave-one-source-out view.)

UPD. Oct-21

Sensitivity check: a different length curve

The reader asked the mirror question. Leave-one-source-out tests whether the ranking depends on individual texts. It does not test whether it depends on the curve I chose for length. The model adjusts for length with a single log2(tokens) term. If fiction only comes out on top under that one choice, the ranking belongs to the curve, not the data.

So I kept the source grouping and the grouped bootstrap, and swapped only the length term. Six alternatives: no length term at all (raw genre means), tokens as a straight line, square root of tokens, log2 plus its square, the four size buckets as dummies, and a model where each genre gets its own length slope, read at the median length.

Length term	R^2	Fiction vs explainer (z)	95% CI	Fiction first in re
none (raw means)	0.22	+3.22	1.51 to 4.94	97.7%
log2(tokens), as published	0.44	+2.38	1.26 to 3.55	97.3%
tokens, linear	0.46	+2.30	1.24 to 3.41	96.6%
sqrt(tokens)	0.45	+2.34	1.27 to 3.47	97.0%
log2 + log2 squared	0.46	+2.32	1.21 to 3.45	96.4%
size bucket dummies	0.43	+2.59	1.31 to 3.84	97.6%
log2, per-genre slope, at median length	0.50	+1.98	0.88 to 2.78	87.

Fiction ranks first under every specification, and personal narrative ranks second under every one. The fiction difference sits between 1.98 and 2.59 z across the six adjusted fits. The per-genre-slope model gives the smallest value, because it lets fiction's own slope absorb part of the gap, and even there the interval stays clear of zero. The only rank changes anywhere in the table are among the genres that sit within 0.15 z of explainers: how-to guide crosses zero under the linear and square-root terms (+0.01 to -0.01), which again just restates that its interval includes zero. Dropping the length term entirely inflates fiction to 3.22 and flips the sign of four small genres, which is the length confound the original comment pointed out, now visible as a row in the table.

Same caveat as before. This probes whether the estimate is fragile to the fitted curve, not whether 10 fiction sources speak for fiction in general. (Added October 20 after the reader asked to vary the length adjustment while keeping the source grouping. Script: breakdown.py v5, log breakdown_v5.log.)

Bottom line

Same result, tighter error bars, and two doubts that used to be arguments are now data. Thanks to the readers who pushed on this.

Can a Local LLM Wash Out a Watermark Without Washing Out the Meaning? I Tested 300 Rewrites

vadim albarov — Fri, 02 Oct 2026 18:49:07 +0000

AI text watermarks hide a signal in which words a model picks. A common idea is that you can remove that signal by running the text through another model: "rewrite this in your own words."

That raises a simple question. If you rewrite a text enough to break its word patterns, do you still keep its meaning?

I tested it on a single GPU with 16 GB of VRAM. In short: yes, mostly — but the model you pick matters a lot, and there is a real trade-off between "changed a lot" and "kept everything."

The setup

Here is the pipeline:

Claude Opus 5.5 wrote 100 source texts. There were 10 genres (news, technical explainers, science, opinion, personal stories, marketing, how-to guides, business memos, history, and short fiction), with 10 topics each. Lengths were from about 190 to 900 words. Every text was packed with checkable details: names, numbers, dates, steps, and plot events.
Three local models rewrote every text with Ollama on a GPU with 16 GB VRAM:
- qwen3:4b-instruct (small)
- gemma3:12b (mid)
- qwen3:14b (upper mid, thinking mode off)
Claude Fable 5.1 judged all 300 rewrites blind. Each packet held the original and three rewrites labeled A/B/C in random order. The judge listed every lost, changed, or added fact, and gave each rewrite a score from 1 to 5.
I measured how much the wording changed by counting how many of the original's word n-grams survive in the rewrite.

All models used the same prompt:

Rewrite the text below completely in your own words.
- Change the vocabulary and the sentence structure; do not copy phrases.
- Keep ALL of the meaning: every fact, name, number, date, step, argument, and plot event.
- Do not add new information, comments, or opinions.
- Keep the same genre, tone, and roughly the same length. Keep the title line, rewritten.
Output only the rewritten text, nothing else.

Settings: temperature 0.7, num_ctx 8192, one run per text.

Why n-grams are a fair proxy for "watermark removed"

Most text watermarks (green-list schemes like Kirchenbauer et al., and Google's SynthID-Text) work at the token level. While the model writes, they nudge it toward certain tokens based on the tokens just before. A detector then counts how often these "preferred" token pairs or sequences appear.

So if most of the original word sequences are gone, most of the signal is gone too. I did not run a watermark detector, because I have no access to one for these texts. Treat the n-gram numbers as a proxy, not a proof.

Results

Meaning, by model

Model	Avg score (1–5)	"Preserved"	"Mostly preserved"	"Not preserved"	Same understanding for the reader
qwen3:14b	4.68	82%	18%	0%	100%
gemma3:12b	4.09	40%	60%	0%	98%
qwen3:4b-instruct	3.76	21%	76%	3%	81%
All 300	4.18	48%	51%	1%	93%

"Same understanding" was a yes/no question to the judge: would a reader of the rewrite come away with the same understanding as a reader of the original?

When the judge ranked the three rewrites of each text (still blind):

Model	Ranked best	Ranked worst
qwen3:14b	76	5
gemma3:12b	17	35
qwen3:4b-instruct	7	60

Average errors per rewrite, as listed by the judge:

Model	Facts lost	Facts changed	Things added
qwen3:14b	0.59	0.96	0.05
gemma3:12b	1.42	2.04	0.30
qwen3:4b-instruct	1.22	2.94	0.38

Wording change, by model

"Kept" means the share of the original's n-grams that still appear in the rewrite. Lower means a heavier rewrite.

Model	Words kept (1-gram)	2-grams kept	3-grams kept	5-grams kept	Longest copied run (words)	Length vs original
gemma3:12b	60%	34%	20%	8%	11.7	1.12×
qwen3:4b-instruct	64%	37%	23%	9%	12.4	1.05×
qwen3:14b	72%	51%	37%	21%	20.7	1.07×

The trade-off

Put the two tables side by side:

qwen3:14b kept the meaning best by far, but it also kept the most of the original wording. About 1 in 5 five-word sequences survived, and on average it copied a run of about 21 words straight from the original.
gemma3:12b changed the wording the most (only 8% of 5-grams survived) and still scored 4.09. Almost all its rewrites (98%) gave the reader the same understanding.
qwen3:4b-instruct changed the wording almost as much as Gemma, but it was the least accurate.

Across all 300 rewrites, the correlation between "meaning score" and "5-grams kept" was 0.36. So yes, rewrites that copy more wording tend to keep more meaning. But inside each model the link was weak (0.03–0.22). Most of the difference comes from which model you use, not from how hard a given model happens to rewrite a given text.

What kinds of mistakes?

The type of error mattered more than the count.

qwen3:14b mostly made tiny changes in nuance:

"two placebo pills work better than one" was hedged to "may be more effective"
"ransomware" was widened to "malware attacks"
an overall risk of "moderate" became "average"

gemma3:12b made the wording vaguer or swapped in a near-synonym:

"Most houseplants" became "Many indoor plants"
"hiring manager" became "recruiter"
Kubernetes init containers that "run to completion, one after another" became a single init container that "executes a sequence of tasks". That one is a real technical error.

Here is a typical example. Original:

SafeNest Products announced on Monday that it is recalling about 86,000 infant car seats sold in the United States because the harness buckle may fail to latch fully.

Gemma:

On Monday, SafeNest Products declared a recall of roughly 86,000 car seats intended for infants, distributed across the United States, due to a potential issue with the harness buckle's functionality.

Every fact is still there, but "may fail to latch fully" became "a potential issue with ... functionality." It reads fine, but it says less.

qwen3:4b-instruct made the dangerous mistakes: wrong names and wrong numbers.

It renamed products: "Quietwave Q7" headphones became "SilentHush S7" all through the text, and the "Elevate Pro Dual-Motor" desk became "Rise Pro Dual-Drive."
It garbled numbers: a desk height range "from 150 to 200 centimeters" became "from 150 to 20."
It changed amounts: British tea imports of "millions of pounds" a year became "hundreds of thousands of pounds."
It cut a farmer's name, "Marcos Ribeiro," to "Marcos Ribe."

A rough keyword search over the judge's notes found number-related changes in about 31 of 100 qwen3:4b rewrites, compared with about 11 for Gemma and 3 for qwen3:14b. This count is approximate, but the gap is clear.

All 3 rewrites judged "not preserved" in the whole test came from qwen3:4b. Two were the renamed products, and the third was a short story whose ending stopped making sense.

By genre and length

The genre made less difference than I expected. The average score ranged from 4.03 (marketing copy) to 4.40 (science writing). Marketing copy had the most "not preserved" verdicts. Product names and spec numbers are exactly where the small model slips, and those details are the meaning of marketing copy. Business memos and technical explainers were most often fully "preserved" (77–80%).

Length did not hurt. The 750–900 word texts scored about the same as, or a little better than, the short ones (4.33 vs 4.19). With an 8K context, all three models handled 900 words easily.

Speed

Total time for 100 rewrites on the 16 GB GPU:

Model	Total	Per text
qwen3:4b-instruct	11 min	~7 s
qwen3:14b	25 min	~15 s
gemma3:12b	25 min	~15 s

So — does the rewrite keep the meaning?

With a good 12–14B model: yes. Readers would get the same understanding from 98–100% of the rewrites, and none were judged "not preserved." The remaining issues are small shifts in nuance or precision.

With a 4B model: usually, but you can't trust the details. One in five rewrites changed what a reader would understand. Names and numbers are the weak spot, and they are often the facts that matter most.

And there is a trade-off you can't ignore:

If you want maximum fidelity, qwen3:14b is the clear winner. But it keeps more of the original wording, so if your goal is to break token-level patterns, it does the weakest job.
If you want maximum change in wording with good meaning, gemma3:12b is the sweet spot. Only 8% of 5-grams survive, and the reader gets the same understanding 98% of the time.

Caveats

No real watermark detector. N-gram overlap is a proxy. Some watermark research claims robustness to paraphrase, and semantic-level watermarks would not be hurt by word changes at all.
One judge, and it is an LLM. Fable 5.1 was strict and specific (it listed things like "price changed from X to Y"), but I did not check it against human ratings. The generator and the judge are both Claude models, which could add some bias toward Claude-like phrasing. That should affect all three local models equally, though.
One run per text at temperature 0.7. Another seed would give somewhat different rewrites.
English only, synthetic texts, one prompt. A stricter prompt ("never change any name or number") would probably help the 4B model.
The keyword counts of name and number errors are approximate.

Next time: SynthID?

This test answered only half of the question. It showed that a good local model keeps the meaning, but it only estimated watermark removal from n-gram overlap.

A real check would use SynthID-Text, Google's open-source text watermark. Would rewriting with a local model really remove a watermark that a detector can see?

Should I run that test next? Tell me in the comments.

How Much Does Your Lunch Break Cost in Claude Code Tokens?

vadim albarov — Thu, 01 Oct 2026 15:06:06 +0000

You step away for lunch with a big Claude Code session open. You come back, type one message, and continue. Nothing looks different. But that one message may have cost 20 times more than the one before it.

I wanted a real number, so I scanned a month of my own Claude Code logs. The short answer: a typical break past the one-hour mark cost me about 125k–135k tokens of cold cache writes. Over the month, these restarts added about 15% to my input usage.

This post explains why it happens, shows the script I used, and lists what I changed.

How prompt caching works in Claude Code

Every turn in a Claude Code session sends the whole conversation to the model again: system prompt, tool definitions, project files you read, tool output, everything. A long session can carry 100k–400k tokens of context.

Prompt caching makes this affordable. After the first request, the stable prefix of the prompt is stored in a cache. Later requests read it from the cache instead of processing it again. In API pricing terms, relative to the base input price:

Token type	Relative price
Uncached input	1.0x
Cache write, 1-hour TTL	2.0x
Cache read	0.1x

A warm turn reads the context at 0.1x. A cold turn writes the whole context again at 2.0x. That is where the roughly 20x comes from.

The cache has a time to live (TTL). In my Claude Code sessions it is one hour, and each request resets the timer. Tool calls count as requests, so an agent working on its own keeps the cache warm. The risk is when you are the slow part: a meeting, lunch, the end of the day.

Note: the 1-hour TTL is what my sessions used. The default API TTL is 5 minutes, and Claude Code can drop to 5 minutes in some cases, for example usage overage. The script below reports which TTL each write used, so you can check your own.

Where the data lives

Claude Code stores each session as a JSONL file under ~/.claude/projects/<project>/<session-id>.jsonl. Every assistant turn includes a usage block. Trimmed:

{
  "type": "assistant",
  "timestamp": "2026-10-01T13:28:05.993Z",
  "message": {
    "id": "msg_...",
    "usage": {
      "input_tokens": 2,
      "cache_creation_input_tokens": 32912,
      "cache_read_input_tokens": 0,
      "output_tokens": 211,
      "cache_creation": {
        "ephemeral_1h_input_tokens": 32912,
        "ephemeral_5m_input_tokens": 0
      }
    }
  }
}

Three fields matter:

cache_read_input_tokens: context served from the cache (cheap).
cache_creation_input_tokens: context written to the cache (expensive).
cache_creation.ephemeral_1h_input_tokens / ephemeral_5m_input_tokens: which TTL the write used.

On a normal warm turn, reads are large and writes are small (just the new tool output). A cache miss looks the other way round: a large write, little or no read. If the previous turn in the same file was more than an hour earlier, the idle gap is the cause.

The script

The script walks every session file, removes duplicate messages by ID, and flags turns where:

cache_creation_input_tokens is at least a threshold (default 50,000), and
the write is larger than the read.

It records the gap since the previous turn, groups the misses by gap length, and estimates a weighted cost share using the prices above.

Click to see the full script (100 lines)

import json, glob, os, sys
from datetime import datetime, timedelta
from collections import defaultdict

ROOT = os.path.expanduser("~/.claude/projects")
MIN_MISS = int(sys.argv[1]) if len(sys.argv) > 1 else 50_000

def parse_ts(s):
    return datetime.fromisoformat(s.replace("Z", "+00:00"))

misses = []
totals = defaultdict(int)
per_project = defaultdict(lambda: defaultdict(int))

for path in glob.glob(os.path.join(ROOT, "*", "*.jsonl")):
    project = os.path.basename(os.path.dirname(path))
    session = os.path.basename(path)[:8]
    prev_ts = None
    seen_ids = set()
    with open(path, encoding="utf-8", errors="replace") as f:
        for line in f:
            try:
                d = json.loads(line)
            except json.JSONDecodeError:
                continue
            if d.get("type") != "assistant":
                continue
            msg = d.get("message") or {}
            u = msg.get("usage")
            ts = d.get("timestamp")
            if not u or not ts:
                continue
            mid = msg.get("id")
            if mid in seen_ids:
                continue
            seen_ids.add(mid)
            t = parse_ts(ts)
            read = u.get("cache_read_input_tokens", 0) or 0
            created = u.get("cache_creation_input_tokens", 0) or 0
            raw = u.get("input_tokens", 0) or 0
            cc = u.get("cache_creation") or {}
            w1h = cc.get("ephemeral_1h_input_tokens", 0) or 0
            w5m = cc.get("ephemeral_5m_input_tokens", 0) or 0
            totals["turns"] += 1
            totals["read"] += read
            totals["created"] += created
            totals["raw"] += raw
            per_project[project]["read"] += read
            per_project[project]["created"] += created
            gap = (t - prev_ts) if prev_ts else None
            if created >= MIN_MISS and created > read:
                misses.append(dict(project=project, session=session, ts=t,
                                   gap=gap, created=created, read=read,
                                   w1h=w1h, w5m=w5m))
            prev_ts = t

misses.sort(key=lambda m: m["ts"])

def fmt_gap(g):
    if g is None: return "start"
    h = g.total_seconds() / 3600
    return f"{h:6.1f}h" if h >= 1 else f"{g.total_seconds()/60:5.0f}m "

print(f"Cache misses >= {MIN_MISS:,} created tokens, read < created\n")
print(f"{'date':16} {'gap':8} {'created':>9} {'read':>8} {'ttl':4} {'project':28} sess")
buckets = defaultdict(lambda: [0, 0])
for m in misses:
    ttl = "1h" if m["w1h"] >= m["w5m"] else "5m"
    print(f"{m['ts'].strftime('%Y-%m-%d %H:%M'):16} {fmt_gap(m['gap']):8} {m['created']:>9,} {m['read']:>8,} {ttl:4} {m['project'][:28]:28} {m['session']}")
    g = m["gap"]
    key = ("session start" if g is None else
           "<5m" if g < timedelta(minutes=5) else
           "5m-1h" if g < timedelta(hours=1) else
           "1h-24h" if g < timedelta(hours=24) else ">24h")
    buckets[key][0] += 1
    buckets[key][1] += m["created"]

print("\nMisses by idle gap before the request:")
for k in ["session start", "<5m", "5m-1h", "1h-24h", ">24h"]:
    n, tok = buckets[k]
    print(f"  {k:14} {n:4} misses  {tok:>12,} tokens written")

miss_tok = sum(m["created"] for m in misses)
print(f"\nAll assistant turns: {totals['turns']:,}")
print(f"  cache read tokens    : {totals['read']:>14,}")
print(f"  cache created tokens : {totals['created']:>14,}  ({miss_tok:,} in the misses listed above)")
print(f"  uncached input tokens: {totals['raw']:>14,}")
# cost weights: read 0.1, 1h write 2.0, raw 1.0
w_read = totals['read'] * 0.1
w_created = totals['created'] * 2.0
w_raw = totals['raw'] * 1.0
w_miss = miss_tok * 2.0
tot = w_read + w_created + w_raw
print(f"\nWeighted input cost share (read=0.1, write=2.0, raw=1.0):")
print(f"  cache reads : {w_read/tot*100:5.1f}%")
print(f"  cache writes: {w_created/tot*100:5.1f}%   of which listed misses: {w_miss/tot*100:5.1f}%")
print(f"  raw input   : {w_raw/tot*100:5.1f}%")

print("\nTop projects by cache-write tokens:")
for p, v in sorted(per_project.items(), key=lambda kv: -kv[1]["created"])[:8]:
    print(f"  {p[:40]:40} written {v['created']:>12,}  read {v['read']:>14,}")

Run it with an optional threshold:

python cache_misses.py          # default 50,000
python cache_misses.py 30000    # lower threshold

My results

The data covers 2026-08-31 to 2026-10-01: 2,410 assistant turns across five projects.

The misses

Restarts after an idle gap of over one hour, threshold 50k:

Idle gap	Tokens re-written
1.1h	170k
1.1h	80k
1.2h	137k
1.2h	77k
1.2h	193k
1.3h	80k
1.4h	142k
1.5h	58k
1.6h	110k
1.6h	164k
2.2h	148k
2.5h	53k
2.6h	171k
2.6h	103k
2.8h	131k
15.1h	124k
18.7h	233k
61.8h	476k

Metric	Value
Idle restarts over 1h	18
Tokens re-written cold	2.65M
Mean per restart	147k
Median per restart	134k
Median for gaps of 1.1h–1.6h (10 restarts)	123k

The mean is pulled up by one outlier: a session resumed after a 2.5-day weekend that re-wrote 476k tokens in one request. The median is the more honest "cost of a break".

The pattern

Most gaps barely passed the limit. 10 of the 18 were between 1.1 and 1.6 hours. These are lunch breaks and meetings, not abandoned sessions.
One stale session cost as much as three or four normal breaks. Resuming a huge session after days away is the worst case.
Session starts are a separate, fixed cost. At a 30k threshold the script also shows 14 session starts, each 31k–36k tokens. That is the startup context: system prompt, CLAUDE.md, memory, skills, and MCP tool definitions. Every new session pays it once, and no timing habit removes it.

The cost share

Weighted by relative price across all 2,410 turns:

Bucket	Tokens	Weighted share
Cache reads	234M	55%
Cache writes, normal incremental	6.8M	32%
Cache writes, idle misses	2.7M	13%
Uncached input	82k	0.2%

Idle misses were 13% of the total weighted input. Put another way: without them, input cost would have been 87% of what it was, so the misses added about 15% on top.

The 32% for normal writes is the ordinary cost of new tool output going into the cache each turn. Timing does not change it. Only the 13% slice can be avoided.

What I changed

1. Come back within the hour when I can. Most of my misses were 5 to 35 minutes past the limit. If I know I will be back soon, I make the next message the first thing I do.

2. Run /compact before a long break. If I will be away for more than an hour and the session is large, I compact while the cache is still warm. The summary is cheap to produce on a warm cache. The restart after the break then re-writes a small summary, not 150k tokens of history, and every turn after that is cheaper too.

3. Do not revive stale sessions. For anything older than a day, I start a fresh session with a short brief: goal, current state, key files. That costs about 35k tokens of startup context, not 400k+ of old history.

4. Do not bother with keep-alive pings. Sending a "heartbeat" message every hour does keep the cache warm, and each one costs only about 0.1x of the context. On paper it wins for gaps up to about 20 hours (20 pings at 0.1x equal one cold write at 2.0x). In practice it adds noise to the conversation, it stops when the machine sleeps, and it keeps a large context alive that you may not need. /compact is simpler.

Caveats

Plan differences. On a subscription plan, tokens count against usage limits, not a bill. The relative weights still show what eats your limits faster, but your plan may not count tokens exactly like API prices.
The weights are an estimate. Read = 0.1, 1h write = 2.0, uncached = 1.0 follows published API pricing ratios. Output tokens are left out on purpose, because they do not depend on caching.
The heuristic is simple. "Write > read and write ≥ 50k" catches idle restarts well. It also catches a few non-idle events, such as one write under 5 minutes in my data, so always look at the gap column.
One user, one month. These are numbers from my own work: several infrastructure and app repos, long agentic sessions, normal office hours.

Try it on your own logs

Save the script, run it, and look at two things: the 1h-24h bucket and the >24h bucket. If they hold a large share of your cache writes, a habit change will save you more than any prompt tweak.

Over to you: is saving ~15% of tokens worth changing how you work? Or is the extra attention not worth it? I'd like to know how you handle long sessions and breaks.

These numbers are averages from one month of my own work. Your numbers could vary.

Typos don't break LLM prompts. One missing quote mark does.

vadim albarov — Sun, 27 Sep 2026 21:02:46 +0000

Everyone who types prompts has wondered the same thing. Does the model care that I wrote "teh" or forgot a comma? Should I clean up my prompt before hitting enter?

I ran a test to find out. The short answer: stop fixing your spelling. Start checking your quotes and colons.

What I tested

I wrote a set of tasks with short, checkable answers. Arithmetic, extraction from a passage, list filtering with "except" and "not", a Python snippet's output, counting words inside quotes, and so on. Then I broke the prompts in controlled ways and sent every version to every model, several times each, in a fresh session every time.

Two prompt sets:

Easy set. 12 one-line tasks, each hand-written at five error levels, from clean to "heavy typos plus broken punctuation".
Hard set. 9 long prompts (200 to 450 words) where the question depends on a detail buried in the text. Here I separated the two kinds of damage:
- Spelling noise: a script misspells 35% or 70% of the words, but never touches the words the answer depends on.
- Structural break: the spelling is perfect, but exactly one punctuation mark that carries meaning is wrong. A missing closing quote. A colon dropped before a list. A comma moved around "except". 9.45 instead of 9:45. resign instead of re-sign.
- Two realistic profiles on top: non-native grammar (articles and verb forms off, spelling fine) and voice-to-text (no punctuation, no capitals, numbers spelled out, "resign" for "re-sign").

Models: Claude Opus 5, Opus 5.5, Sonnet 5, Haiku 4.5, Fable 5.1 (all at low and medium effort), plus seven local models through Ollama: gemma4, qwen3 14B, phi4 14B, granite 4.2, ornith-1.5 9B, llama 3.1 8B, mistral 7B.

Total: about 4,900 sessions. An answer counts as correct if the ideal answer appears in the reply, even with an explanation attached.

Result 1: spelling and grammar cost nothing

Here is the hard set. Each cell is the percent of correct answers.

model	clean	70% typos	non-native grammar	voice-to-text	one punctuation break
Opus 5	100	100	100	100	88
Opus 5.5	100	98	100	100	88
Fable 5.1	100	100	100	98	92
Haiku 4.5	100	100	100	98	88
Sonnet 5	100	100	100	89	77
granite4.2	93	67	89	56	88
phi4 14B	74	74	85	78	62
gemma4	78	78	81	44	79
qwen3 14B	67	74	85	56	62
ornith-1.5 9B	74	48	70	52	42
llama 3.1 8B	44	56	63	33	38
mistral 7B	52	26	41	56	38

Read the Claude rows first. With 70% of the words misspelled, every one of them scored the same as on clean text. Non-native grammar: 100% across the board. This is the prompt that scored 100%:

shop sellin pens 3dolar each an notbooks 5 dolar,each tom buys 4pen an 2notbook.how much he pay totaly,anser numbr only

Now read the last column. One punctuation mark, with perfect spelling everywhere else, and the same models lose 8 to 23 points.

Result 2: which punctuation mark breaks which model

Not every break matters. Most were recovered from context by every Claude model. Two were not.

break	Opus 5	Opus 5.5	Fable	Haiku	Sonnet
missing closing quote	0	33	33	0	17
colon dropped before a list	100	67	100	100	0
comma moved around "except"	100	100	100	100	100
European decimals (1.500 at 2,50)	100	100	100	100	100
9.45 instead of 9:45	100	100	100	100	100
managers' → manager's	100	100	100	100	100
resign instead of re-sign	100	100	100	100	100
comma dropped in "not closed, and assigned to Lee"	100	100	100	100	100

The missing quote. The task: count how many times "report" appears inside a quoted paragraph, not in the text after it. Remove the closing quote and every model counts the three extra "report"s after the paragraph. Answer 8 instead of 5. Opus 5.5 even wrote "the closing quote is missing, so I counted all" and still gave the wrong number. It noticed the problem and followed the broken structure anyway.

The dropped colon. "Give the plural form of each word below keeping the order mouse child foot analysis reply with the plurals only comma separated". Sonnet answered mice, children, feet, analyses, replies. Every local model did the same. The word boundary between instruction and data was gone, so "reply" became data.

Everything else was recovered, including the genuinely ambiguous one. "List the tickets that are not closed and assigned to Lee" can mean two things once the comma is gone. All five Claude models picked the intended reading every time, because the sentence started with "For Lee's standup".

Result 3: dictation is a different story

The voice-to-text column is where the small models fall apart. gemma4 goes from 78 to 44. granite from 93 to 56. Two things do the damage: spelled-out numbers ("three hundred forty pallets across twenty two trailers") and, again, the missing colon before a list. Every local model scored 0% on the dictated list task.

The frontier models mostly shrug it off, with one honest exception. "fifteen hundred units at two fifty each": Sonnet read "two fifty" as $250 in six trials out of six and answered 375,000 instead of 3,750. Fable did it once. Haiku turned "extension two zero one" into 2001 once. These are not model bugs. Spoken numbers are ambiguous, and the model has to guess.

Things that did not help

Effort level. Low versus medium made no difference for any model on any task. Structural breaks are not a reasoning problem, so more thinking does not fix them.
Warning the model. I prepended "the message below may contain spelling, grammar, and punctuation mistakes, answer according to the writer's likely intent". Zero effect on Opus 5, Haiku, and Sonnet. A few points on Opus 5.5 and Fable. Mixed or negative on the local models.

What degrades first: the format, not the answer

Sloppy prompts get slightly sloppier replies before they get wrong ones. Asked for "the number only", Fable and Opus 5.5 add a line of working in about 20% of runs on any prompt. Haiku, on the easy set, drifted into "Yes. If all bloops are razzies..." explanations as the grammar got worse, and at the heaviest level refused four times out of eight because "bloops and razzies are not real words". With clean grammar it had answered the same nonsense-word question fine.

What to actually do

Don't fix spelling or grammar. The model does not care, and neither should you.
Do check the punctuation that carries structure: closing quotes, the colon before a list, commas around "except" and "not", and anything separating your instructions from your data.
If you dictate prompts, re-read the numbers and the list boundaries before sending.
Small local models are much less forgiving. If you run a 7B to 14B model, treat every delimiter as load-bearing and avoid dictation.

Caveats

Twelve short tasks and nine long ones, three trials per cell, one random seed for the typo generator. The voice-to-text prompts are hand-simulated, not real speech engine output. Claude models were called through the Claude Code CLI with a minimal system prompt and tools disabled. Local models ran at Ollama defaults with thinking off. The harness and every prompt and reply are in the repo linked below, so you can rerun it with your own tasks.

Repo with the harness, all prompts, and every raw reply: github.com/valbarov/prompt-noise-test

12 AWS Defaults That Ship Insecure (and the One Line That Fixes Each)

vadim albarov — Sun, 27 Sep 2026 01:19:02 +0000

Every AWS default answers exactly one question: will the tutorial work?

No surprise bill. No failed API call. No "access denied" on step three. That is a great default for a tutorial. It is a terrible default for the thing you spun up "just for staging" that is now production, on the day someone asks who downloaded that bucket in March.

I run infrastructure for a healthcare company. People with clipboards read my configs. Somewhere around my third Terraform module I noticed my job had a shape, and the shape was not "design clever things". It was flipping the same switches AWS had left in the wrong position, over and over. So I wrote them down.

Here are the twelve, ranked by how likely each one is to hurt you. One rule of thumb before we start: checkov and tfsec check what you wrote. Most of this list is about what you did not write.

1. CloudTrail remembers 90 days, then forgets

Default: you get "Event history": 90 days of management events, viewable in the console. No trail to S3 unless you create one. And S3 object reads and writes (data events) are not logged at all, even when a trail exists.

How it bites: a credential gets phished in March. In September someone asks what it touched. The window scrolled off in June, and even inside the window "did they download the sensitive bucket" was never recorded. Your answer is a shrug.

The fix:

resource "aws_cloudtrail" "audit" {
  name                       = "org-audit"
  s3_bucket_name             = aws_s3_bucket.audit.id
  is_multi_region_trail      = true
  enable_log_file_validation = true

  event_selector {
    read_write_type = "All"
    data_resource {
      type   = "AWS::S3::Object"
      values = ["arn:aws:s3:::my-sensitive-bucket/"]
    }
  }
}

Data events cost real money at volume. Scope them on purpose, not by forgetting.

2. RDS storage is unencrypted, and you cannot fix that later

Default: storage_encrypted = false in Terraform and in the API. The console nudges you. Code does not. And encryption can only be enabled at creation. Later means snapshot, encrypted copy, restore, cutover.

How it bites: this is the worst one on the list because it is irreversible in place. Staging quietly becomes production, it always does, and eighteen months later a review finds your primary database in plaintext. The remediation is a maintenance window on your busiest system.

The fix:

storage_encrypted = true
kms_key_id        = aws_kms_key.data.arn

In my modules this is hardcoded, not a variable. A security invariant that callers can toggle off is not an invariant. It is a default waiting to come back.

3. Your Postgres accepts plaintext connections

Default: on RDS for PostgreSQL 14 and earlier, rds.force_ssl is 0. The server supports TLS. It also cheerfully accepts connections without it. AWS flipped this to 1 for PostgreSQL 15+, which tells you what AWS thinks of the old default.

How it bites: any client on sslmode=prefer (the libpq default) falls back to plaintext if the handshake hiccups. Nothing fails. Nothing logs. Your transmission security now depends on every developer, sidecar, and ad hoc psql from a bastion remembering a flag.

The fix: enforce it at the engine so client config stops mattering:

parameter {
  name  = "rds.force_ssl"
  value = "1"
}

4. Your load balancer still shakes hands with 2008

Default: create an HTTPS listener without a policy and you get ELBSecurityPolicy-2016-08, which accepts TLS 1.0 and 1.1. Also off by default on the same resource: access logs and deletion protection.

How it bites: nobody notices, because modern browsers pick 1.2+. Then a customer's security team runs an external scan before signing, and the deal stalls on a finding one attribute would have prevented.

The fix:

ssl_policy = "ELBSecurityPolicy-TLS13-1-2-2021-06"

And on the aws_lb itself: access_logs { enabled = true }, enable_deletion_protection = true, and drop_invalid_header_fields = true while you are there.

5. VPC Flow Logs do not exist

Default: no VPC has flow logs until you create them. No network-level record of anything.

How it bites: GuardDuty flags an instance talking to a known-bad IP. The first question is "since when, and what else did it talk to?" Without flow logs your incident response runs on vibes, and "was anything exfiltrated?" gets answered conservatively, which means expensively.

The fix:

resource "aws_flow_log" "vpc" {
  vpc_id               = aws_vpc.main.id
  traffic_type         = "ALL"
  log_destination_type = "cloud-watch-logs"
  log_destination      = aws_cloudwatch_log_group.flow.arn
  iam_role_arn         = aws_iam_role.flow.arn
}

ACCEPT only or REJECT only is half an audit trail.

6. Services create log groups you never asked for

Default: log groups default to Never Expire and an AWS-managed key. The sneaky part: RDS log exports, Container Insights, and Lambda create their own log groups on first write, with exactly those defaults, outside your Terraform state.

How it bites: you enable RDS log exports. RDS creates /aws/rds/instance/.../postgresql for you. Three years later that orphan holds three years of connection logs plus whatever your app leaked into query errors, retained forever, under a key you did not choose, and invisible to terraform destroy.

The fix: pre-create every log group a service will write to, before the service exists:

resource "aws_cloudwatch_log_group" "rds" {
  name              = "/aws/rds/instance/myapp-db/postgresql"
  retention_in_days = 2192
  kms_key_id        = aws_kms_key.logs.arn
}

7. The default security group is a party where everyone knows everyone

Default: every VPC ships with a default security group that allows all traffic between members and all outbound. Anything launched without an explicit group lands in it.

How it bites: a contractor spins up a "quick utility box" without thinking about security groups. It joins the default one, alongside everything else that drifted in over the years. That box gets popped, and lateral movement is free, because membership is the authorization.

The fix: you cannot delete the default group, but you can strip it bare. A resource with no ingress or egress blocks removes every rule, so anything landing there can talk to precisely nothing:

resource "aws_default_security_group" "this" {
  vpc_id = aws_vpc.main.id
  tags   = { Name = "default-DO-NOT-USE" }
}

Loud failure beats silent success.

8. RDS backups: one day via the API, zero via Terraform

Default: backup retention is one day via API or CLI, seven via the console, and backup_retention_period defaults to 0 in Terraform. Zero. Automated backups off. Deletion protection is also off.

How it bites: an instance defined without that line has no backups at all, and it passes plan, apply, and review, because absence does not show up in a diff. You find out during your first real restore attempt, which is the single worst moment available. Bonus: with deletion protection off, one terraform destroy in the wrong workspace takes the database and its backups together.

The fix:

backup_retention_period   = 30
deletion_protection       = true
skip_final_snapshot       = false
final_snapshot_identifier = "myapp-db-final"

Then AWS Backup with a locked vault on top, because backups attached to the instance share the instance's blast radius.

9. EBS encryption by default is off, in every region separately

Default: account-level EBS encryption by default is disabled, and it is a per-region setting. Turning it on in us-east-1 does nothing for us-west-2.

How it bites: your Terraform encrypts every volume it manages. Then someone launches a console instance for a one-off migration, copies a database dump onto it, and that volume is plaintext, because the account default governs ad hoc resources, not your module.

The fix: one resource, once per region you use, including the ones you think you do not use:

resource "aws_ebs_encryption_by_default" "this" {
  enabled = true
}

10. SNS stores your alerts in plaintext

Default: server-side encryption on SNS topics is off until you set a KMS key.

How it bites: mostly as an audit finding. Occasionally worse: a team wires appointment reminders through SNS, and now message bodies with patient data sit unencrypted in the messaging layer while every database in the stack is dutifully encrypted. One-line fix or an hours-long finding memo. Pick the line.

The fix:

resource "aws_sns_topic" "alerts" {
  name              = "security-alerts"
  kms_master_key_id = aws_kms_key.logs.arn
}

Real gotcha: once the topic is encrypted, CloudWatch and EventBridge need kms:Decrypt and kms:GenerateDataKey* in the key policy, not in IAM. Otherwise every alarm publish fails silently. Test the path end to end.

11. S3 got fixed, which is exactly the problem

Default (current): since January 2023 every new object is encrypted with SSE-S3. Since April 2023 new buckets get Block Public Access on and ACLs off. Credit where due: the two most famous S3 footguns are gone for new buckets. Still off: server access logging, versioning, and any customer-managed key.

How it bites: "S3 encrypts by default now" becomes the reason nobody configures anything further. A sensitive bucket ends up with no access log, no versioning, no policy denying plaintext transport, and a key nobody can audit.

The fix: for a bucket that matters: SSE-KMS with your own key, versioning on, aws_s3_bucket_logging to a separate log bucket, and a bucket policy denying aws:SecureTransport = false. The 2023 change is your floor, not your control.

12. ECS Exec gives you a shell, and records nothing

Default: ECS Exec logging is DEFAULT, meaning "whatever awslogs config the task has". If the container has none, session input and output are not logged at all. KMS encryption of the session channel is opt-in.

How it bites: an engineer execs into a production task to debug, runs a few queries, pastes results somewhere. Months later an access review needs that session. CloudTrail says a shell was opened at 14:32. That is the entire record.

The fix: on the cluster, execute_command_configuration with logging = "OVERRIDE", a dedicated encrypted log group, and a kms_key_id for the channel. Every session becomes a transcript.

Automate it or relive it forever

Twelve items is too many for humans to re-run reliably. That is the real lesson. What works for me:

One baseline module for the apply-once account controls: the trail, AWS Config rules like RDS_STORAGE_ENCRYPTED and VPC_FLOW_LOGS_ENABLED, EBS default encryption, the account-level S3 public access block.
Hardcoded invariants in workload modules. storage_encrypted, publicly_accessible = false, forced TLS, and log validation are not variables in mine. Exposing them as inputs just recreates the permissive default one level up, with your name on it.
Policy as code for the rest. checkov in CI catches regressions in code. AWS Config catches the console-created, script-created, "temporary" resources your code never met. You need both, because the resources that hurt most never saw a pull request.

The pattern across all twelve is the same. AWS optimizes for the first five minutes. Your auditor cares about year five, when someone asks who accessed what and the answer has to exist. Those two goals produce opposite defaults, and closing the gap is, by AWS's own shared responsibility model, your job.

Not legal or compliance advice. I am an infrastructure engineer, and defaults change, sometimes even in the right direction, so check each one against current docs.

Which default bit you? I only know the twelve that bit me.

Claude Code's Max Effort Cost Me 8x More. It Wrote the Same Code.

vadim albarov — Tue, 15 Sep 2026 21:44:18 +0000

Claude Code has an /effort command with five levels: low, medium, high, xhigh, max. The help text for low says "quick, straightforward implementation with minimal overhead." The natural assumption is that the higher levels buy you better code, and that the price is time and tokens.

I wanted to see that trade-off on a chart. So I built a small harness, gave every level the same task in a clean copy of the same folder, and measured. I did it three times with three different task shapes.

The chart never appeared. Every level got the same score on every task. Only the clock and the invoice moved.

The harness

Nothing clever. One headless Claude Code process per level, running in its own copy of the project:

claude -p "$(cat prompt.txt)" --effort low \
  --output-format stream-json --verbose \
  --permission-mode acceptEdits --allowedTools "Read,Write,Edit,Bash(python*)"

The stream-json output goes to a log file. A small viewer prints readable progress in a visible PowerShell window, so I could watch five terminals work through the same problem one after another. When all runs finish, an analyzer parses the logs for turns, tool calls, cost, and token counts, then runs a hidden test suite against each folder. The child sessions never see the hidden tests.

I ran each level once. Keep that in mind for every number below. One sample is enough to show a 9x gap. It is not enough to tell a 10 percent difference from noise.

Model was Fable 5.1 on every run. Claude Code 2.1.272.

Benchmark 1: implement from docstrings

A tiny Python order-processing library: money parsing, coupons, tax, inventory with reservations. Nine functions and one class, all stubbed out with NotImplementedError and detailed docstrings. Three public tests visible. 57 hidden tests.

The prompt:

Implement every function and method marked # TODO in the shop package exactly according to the docstrings. Do not change the docstrings, models.py, or anything in tests/. Run python -m pytest -q to check your work. When done, reply with a one-line summary.

Results:

effort	turns	wall time	cost	output tokens	hidden tests
low	12	77 s	$0.59	5,174	57/57
medium	12	69 s	$0.59	4,995	57/57
high	23	174 s	$1.22	13,009	57/57
xhigh	18	297 s	$2.04	24,919	57/57
max	31	693 s	$4.72	62,196	57/57

Every level passed everything on the first try. Max took eleven and a half minutes and $4.72 to arrive at the same 57/57 that low reached in 77 seconds for 59 cents. Output tokens grew 12x. Tool calls grew only about 3x, so most of that growth is reasoning, not doing.

Low and medium were near clones of each other. Same number of turns, same sequence: read, write three files, run pytest, one sanity check, done. The diff between their outputs is 41 lines of reordering.

The code was not identical across levels, though. Since the tests could not separate them, I diffed the implementations by hand.

Low and medium parsed money strings by stripping the sign and the dollar symbol and checking each character. They delete commas, so "1,23.45" is accepted.
High wrote one verbose regex with named groups that enforces comma groups of three.
Xhigh and max factored parsing into helpers. Max also widened Decimal precision with a localcontext for very large amounts. No test covers that.
Tax rounding: low and medium used Decimal with quantize. High, xhigh and max switched to Fraction for exact rational math and a hand-written half-up round.
Max added validation nobody asked for. The inventory constructor rejects negative stock, and reserve rejects quantities below one.

So higher effort produced more defensive, more factored code. It did not produce more correct code, because there was nothing left to be more correct about.

One detail I liked: the only actual spec deviation came from high, not low. The docstring said to raise KeyError with the coupon code as the message. High raised it with the uppercased code. My test used an all-caps code, so it passed anyway. Effort level did not predict spec fidelity.

Benchmark 2: port it to Go

The first task was too easy. A port to another language has real traps: Python has Decimal, Fraction, arbitrary ints and exceptions. Go has none of those. Errors are values, half-up rounding is manual, and the model has to decide how an OutOfStock error carries its fields.

Same harness. The base folder held the Python reference source and an empty Go module. Hidden Go tests import a fixed package path. The analyzer also runs go vet and gofmt -l for free quality signals.

I ran low, medium and high, then stopped to save budget.

effort	turns	wall time	cost	output tokens	hidden tests	gofmt	Go lines	test lines
low	18	135 s	$1.04	11,175	all pass	ok	442	148
medium	32	401 s	$2.60	31,503	all pass	test file unformatted	498	339
high	26	470 s	$3.44	38,673	all pass	ok	574	420

Again, flat on correctness. Cost roughly linear in effort. What each level did with its budget was different, and this part was genuinely interesting to watch.

Low read the sources, wrote five files, ran vet and test, and stopped.

Medium wanted to probe the Python original before writing Go, to confirm exact rounding and error behavior. Good instinct. But my tool allowlist for that run did not include Python, so it retried across PowerShell and Bash several times before giving up. That is harness noise, not effort. It also wrote a test suite more than twice the size of low's, and never ran gofmt on it.

High, with Python allowed, built its own cross-language oracle. It wrote a Python script that ran the original code over many inputs, generated a Go test file from the results, ran that against its port, and then deleted the generated files. That is exactly the grading technique I had planned to build myself. High invented it unprompted, which is the kind of behavior you would hope a higher effort setting buys.

It just did not change the score.

Benchmark 3: bug hunt with no execution

Both tasks so far had the same two properties: the spec was complete, and verification was cheap. Any level can iterate against a test suite until green. Extra thinking has nothing to buy.

So I broke both. A working Go version of the library with 8 planted bugs: a wrong rounding constant, bad comma grouping, a sign error in refund splits, case-sensitive coupon codes, percentage coupons discounting gift cards, tax applied to exempt categories, a non-atomic inventory reserve, and a double-release that inflates stock. The prompt gave 8 vague user reports, one per bug, and one hard rule:

You cannot run any commands in this session (no go build, no tests). Reason carefully from the code.

Tool allowlist: Read, Grep, Glob, Edit. Nothing else.

effort	turns	wall time	cost	output tokens	bugs fixed	diff
low	15	52 s	$0.94	3,677	7/8	+20 / -5
medium	15	55 s	$0.86	3,585	7/8	+16 / -3
high	16	69 s	$0.94	5,055	7/8	+26 / -4
xhigh	16	125 s	$1.30	10,200	7/8	+21 / -4

Finally a miss. All four levels missed the same bug, the tax one, and all four made the identical wrong fix. When I looked closely, that was my fault.

The correct rule was proportional: remove the exempt fraction of the pre-discount subtotal from the post-discount taxable amount. That rule lived only in the Python docstring, which was not in the Go project. From the Go code and the report "grocery and book orders are being taxed," subtracting the exempt amount directly is a completely reasonable fix. Four independent runs converging on it is evidence that the intended rule was not inferable from what they were given. It is not evidence that effort failed.

So the fair score is 7 of 7 for every level. Flat again.

What effort did change here was, once more, output tokens: 3.6k, 3.6k, 5k, 10k. Wall time and cost followed. And scope creep showed up at low, high and xhigh, each of which invented a fuzzy gift-card category matcher that handles "Gift Card", "gift_card" and "GIFT-CARD" when the original code used an exact string. Medium alone made the exact minimal change the prompt asked for.

What I take from this

Effort controls how much the model deliberates, not what it knows. On every task here the answer was fully determined and reachable in one careful pass. Given that, low found it. Max found it too, after thinking about it for eleven minutes.

Verification loops make effort irrelevant. If the session can run tests, any level will iterate until green. The Python and Go benchmarks measured persistence, and persistence was free at every level.

Higher effort spends its budget on self-checks and defensive code. Oracles, throwaway verification scripts, extra validation, regexes that reject malformed input the spec never mentioned. Sometimes that is exactly what you want. In a codebase with a scope rule, it is unrequested behavior you now have to review.

Medium was the quiet winner on discipline. It never won on speed or cost, but on the bug hunt it was the only level that did precisely what the prompt asked and nothing more.

The total bill for this experiment was about $20. Roughly $9 for the Python runs, $7 for the Go runs, $4 for the bug hunt. The max Python run alone was almost a quarter of it.

Where effort probably should matter

I have not proven that effort is useless. I have shown three tasks where it did not help, and I think I know why. Effort should matter when deliberation itself is the bottleneck:

Two-step inference. The symptom is in one file and the cause is in another, and the code near the symptom looks fine. A fix at the symptom passes the reported case and fails hidden siblings.
A decoy fix that breaks a hidden invariant. An inventory patch that resolves the report and introduces a race under go test -race.
A constraint hidden in a README that conflicts with the obvious fix.
A codebase too large to read. Around 2,000 lines, where effort shows in search strategy and in what gets verified before editing.
Repeat runs. Three samples per level at minimum. My single-sample wall times have variance comparable to the gaps between adjacent levels.

That is the next benchmark. Until then, my working rule: for a well-specified implementation task where the session can run tests, low or medium. Reserve high and above for tasks where the model cannot check itself and the answer is not sitting in the docstring.

And watch the clock. Nothing on max was wrong. It just took nine times longer to be right.