DEV Community: jidonglab

Refmade Shipped Without Git: 2,764 Files in One Snapshot

jidonglab — Thu, 05 Nov 2026 20:46:46 +0000

On October 7 I asked an agent why Refmade was not on GitHub. The answer was short: it had never been in Git. Seventeen days of work, from September 20 to October 7, and every installer I had published up to 0.3.2, existed only as a folder on a laptop.

This is part 13 of "Building Refmade", a series about a desktop app I am building mostly with Claude Code and Codex. I plan, decide and review. The agents write most of the code. Refmade draws the Claude Code and Codex sessions running on your computer as an office, so you can see which sessions are working and which are waiting for you. It is an independent app, not affiliated with Anthropic or OpenAI. The UI is Korean right now; that changes later in this series.

How a project goes 17 days without `git init`

Nobody decided to skip Git. The folder started on September 20 as a personal structure for organizing agents, not as a product. Nothing in it said "repository", so nothing ever asked for one.

I do have a standing rule that code gets committed and pushed automatically after verification. That rule runs inside repositories. This folder was not one, so the rule never fired and never complained.

Deploys hid the gap too. The website went out with the Vercel CLI straight from a local folder. For rollback I noted the previous deployment each time, so "go back" meant "redeploy the older deployment", not "check out an older commit". On September 24 I told Codex to wrap up because I was moving the work to Claude Code. The handoff note it wrote said, in plain text, that the folder was not a Git repository. Nobody acted on that line, including me.

The only version control I had was a tarball. Before the big office rewrite on September 25, an agent packed _backup/cli-before-office-p0-2026-09-25.tgz. That is a snapshot, not a history.

Date	What existed	Where the history lived
Sep 20	Agent team folder	Nowhere
Sep 25	Before the office rewrite	One tarball
Sep 28	Desktop app 0.2.x installers	File timestamps
Oct 1	0.3.0 and 0.3.1 public	Deployment list
Oct 2	0.3.2 public	Same
Oct 7, 21:44	First commit	Git, finally

A second Mac, and source inside an installer

The strangest part came between October 1 and October 6. I had a second Mac where Codex worked on a separate line of experiments: an easier first screen, a model-and-effort router and some automation ideas. The latest source was not on that machine. It only had what had been installed.

The agent found a workaround. A packaged Electron app carries its own JavaScript, so the installed 0.3.1 build contained the app source. The agent copied 1,034 files out of it, confirmed the originals' SHA-256 hashes did not change, and built the experiments on that copy.

It worked, and it meant reading my own product back out of an installer.

On October 6 an agent searched GitHub for the real source. An older public repository with the same name holds a gallery site last touched in July, a completely different product that I had stopped working on. The desktop app was not there. The status recorded that day was "source location pending". On October 8 the experiments from the second Mac were merged into the main tree as one commit: integrate easy-start workspace into Refmade desktop.

The underlying problem was that two machines each held part of the work. As far as I can reconstruct it, my main Mac had the latest source but not the experiments, and the second Mac had the experiments but not the latest source. A branch and a remote would have given both a single place to meet. "Source location pending" is a strange status to write about your own product.

The first commit

On October 7 an agent created a private repository and pushed, at my request. The first commit has 2,764 files and about 105 MB. It is one snapshot commit that says "start Git here", with no history behind it. The commit is dated October 7 and does not pretend to be older.

What I kept out mattered as much as what went in:

Excluded	Why
`runtime/`	Run logs and 33 copies of old deployments, about 5.4 GB
`_e2e/`	End-to-end output that included a browser's cookies
`_backup/`	Tarballs of earlier states
`dist/`, `node_modules/`	Rebuilt by the build
Screenshots, raw generated images	Large

The .gitignore groups its rules by what they protect against. An excerpt, with the comments translated from Korean and other lines left out:

# 1. secrets: env files, keys, credentials, token caches, browser profiles
.env
.env.*
credentials*.json
Cookies
# 3. run records, state, caches, build output
runtime/
_e2e/
_backup/

The cookie patterns have a reason. The end-to-end folder held a browser's cookies, so the browser-profile patterns sit in the file and _e2e/ is ignored whole.

17 gitleaks findings

Before the push, gitleaks ran over the tree and reported 17 findings. Each one was opened and compared against its source. All 17 were fake values inside tests: JWT-shaped strings, token-shaped strings and invented key names used as fixtures. None was a real credential.

A scanner cannot tell a fixture from a leak, so the comparison is the useful part. I keep that step on every first push, even for a private repository.

What I lost, and what I changed

The cost is history. I cannot restore the September 21 dashboard from Git, because Git starts on October 7. What remains of that day is file modification times and screenshots. The same goes for every 0.2.x and 0.3.0 to 0.3.2 build: I have notes and deployment records, not the diffs.

After the snapshot, the rhythm changed fast. The Git log shows 15 commits dated October 8 alone, the day five releases went out (that day is the subject of part 14). I could not have done that safely without being able to look at yesterday's state of a file.

Refmade was not the only one. The same day, 13 folders on my machine, this one included, went into private repositories, each with its own ignore file covering secrets, operational data, run logs and build output.

The standing rule is now stricter: any folder with work in it gets git init, a gitleaks pass and a private remote before it ships anything. Today the working tree tracks 3,062 files across 38 commits.

Refmade 0.3.9, Korean UI, public demo data. A decision card: "Tell me just this", then "What tone should I use to reply to the customer?", with a highlighted "AI recommended: Kind and short" option. This is the app that the tracked files build.

What I would check now

Git was optional for me only because every other step (build, deploy, rollback) had a substitute that felt like version control. Tarballs and deployment lists do not give you diffs. A quick test: can you answer "what changed between yesterday's build and today's" without the person who built it?

For 17 days I could not do that reliably. Part 14 covers the releases that came right after, and the checks that caught bugs in them. The app link comes in the last post of the series.

If you work with agents that create files for you, when do you make the first commit: before the first line of generated code, or once there is something worth keeping?

Claude Code Env Vars Don't Persist: 3 of 80 Runs Fell Back to Prod

jidonglab — Thu, 05 Nov 2026 00:43:59 +0000

I told Claude Code export APP_ENV=staging. My next message said to run the script. It printed target = production.

Nothing got deployed. target.py is a one-line stub in a throwaway repo, built around os.environ.get("APP_ENV", "production"), a default you've probably seen in real code. But Claude Code env vars don't persist between Bash tool calls, and I wanted to know how often that actually lands. So I ran 97 headless sessions and counted.

TL;DR

Every Bash tool call in Claude Code gets a fresh shell. export, source .venv/bin/activate, functions, aliases and umask all vanish before the next call.
cd is the exception, and only when the command succeeds and stays inside the project. In subagents it didn't carry at all.
Claude usually works around this by repeating the export. In 80 task runs it missed 3 times, and all 3 ran the script with the fallback value.
Each miss was reported honestly, after the script had already run.
For a real fix, set state at session start: env in settings.json for plain values, a SessionStart hook writing to $CLAUDE_ENV_FILE for a virtualenv.

Why don't Claude Code env vars persist between Bash calls?

Claude Code env vars don't persist because each Bash tool call starts a new shell, so anything held in shell memory dies with that call. Only the working directory is tracked across calls.

I set eight kinds of state in one call and read them back in the next. All 3 runs gave identical results.

Set in call 1	Seen in call 2
`export LAB_EXPORT=1`	empty
`LAB_PLAIN=2`	empty
`source .venv/bin/activate`	`VIRTUAL_ENV` empty, no `python` on PATH
`lab_fn() { ... }`	not found
`alias lab_alias=...`	not found
`umask 077`	back to `022`
`set -o noclobber`	off
`cd sub`	kept

You can see it for yourself with one command:

claude -p 'Make two separate Bash calls, one at a time: first "export FOO=1", then "echo FOO=$FOO". Show me the second output verbatim.' --allowedTools Bash

It comes back FOO= with nothing after the equals sign.

The trap is that row at the bottom. cd working makes the tool feel like one long terminal session, so you assume the rest of your shell came along too.

When does cd persist in Claude Code?

cd persists in the main session when the command succeeds and ends inside the project. Break either condition and the shell stays where it was.

Command	Next `pwd`
`cd sub`	`proj/sub`
`cd /tmp` (outside the project)	back at `proj`, with the note `Shell cwd was reset to ...`
`cd sub && false` (exit 1)	still `proj`
`cd sub` inside a subagent, then `pwd`	still `proj` (2 of 2 runs)

The subagent row is the sneaky one. cd sub prints nothing and exits 0, then the next call is back at the root. A relative path written after that lands somewhere you didn't plan. Subagents do start in the main session's current directory, though: after cd sub in the parent, a subagent's first pwd was proj/sub in 2 of 2 runs.

How often does Claude actually trip on this?

Rarely. Across 80 runs of ordinary-sounding tasks on Sonnet 5.5, Claude ended up with the wrong environment 3 times, and all 3 involved a variable with a silent default.

Task (10 runs each)	Correct	Wrong
Export, then run the script	10	0
Export, list files, read script, run it	10	0
Export, then a subagent runs the script	10	0
Export, 8 unrelated Bash calls, then run	9	1
Export in one message, run in the next	8	2
Activate venv, then run	10	0
Activate venv, then a subagent runs it	10	0
Activate venv in one message, run in the next	10	0

The model knows this rule. It wrote export APP_ENV=staging && python3 target.py, re-exported in each command, and briefed subagents with the variable spelled out in their prompt. That last part surprised me. I expected delegation to be the weak spot, and it scored 20 of 20.

The misses came from two places:

Distance. In one run Claude issued export APP_ENV=staging as its own call, did eight other things, then ran a bare python3 target.py.
A message boundary. In all 10 of these runs, Claude skipped the bare export and warned that it wouldn't last. My next message ignored the warning and asked for the run. Eight times it prefixed the command with APP_ENV=staging on its own. Twice it ran the bare command.

That second miss is partly on me, and I think it's the realistic one. You skim the reply, type the next instruction, and move on.

All 3 misses were reported in the final reply. "It was not set when target.py ran." That sentence is useful after a read-only script. After terraform apply it's a postmortem.

Ten runs per pattern is too small to quote as a failure rate. The finding I'd trust is the split: 3 misses in 50 variable runs, 0 in 30 venv runs. A missing dependency throws ModuleNotFoundError. A missing variable with a default throws nothing.

How do you make Claude Code env vars persist?

Set them before the session starts, since nothing Claude runs mid-session survives the call. I tested four ways.

Method	`APP_ENV`	venv	Subagent
`env` in `.claude/settings.json`	works	no	not tested
Activate, then launch `claude` from that shell	works	works	not tested
`SessionStart` hook writing to `$CLAUDE_ENV_FILE`	works	works	works
`CLAUDE_ENV_FILE=./env.sh claude` at launch	works	works	`APP_ENV` works

Plain values: use env in settings. It's the smallest fix.

{ "env": { "APP_ENV": "staging" } }

It can't run anything, though. With this alone, python3 still resolved to /usr/bin/python3 and check.py died with ModuleNotFoundError.

Virtualenvs and anything that needs a script: use a hook. Whatever it appends to $CLAUDE_ENV_FILE applies to every later Bash call, subagents included.

{
  "hooks": {
    "SessionStart": [{
      "hooks": [{
        "type": "command",
        "command": "printf 'source \"%s/.venv/bin/activate\"\\nexport APP_ENV=staging\\n' \"$CLAUDE_PROJECT_DIR\" >> \"$CLAUDE_ENV_FILE\""
      }]
    }]
  }
}

Use >>, so you don't wipe what another hook wrote. Put per-developer values in .claude/settings.local.json and keep secrets out of anything committed.

Can Claude change the environment during a session?

No, not in anything I tried. The env file behaves as a snapshot taken at session start.

With the hook setup, $CLAUDE_ENV_FILE was empty inside the Bash tool, so Claude had nothing to append to.
With the launch variable, the path was visible and the file could be edited. I appended export APP_ENV=edited-midsession, confirmed it with cat, and the next call still said staging. Same result in 2 of 2 runs.

If you need to switch from staging to prod, restart the session. For a single command, ask for the inline form: APP_ENV=prod python3 target.py.

Should scripts default to production at all?

No, and this is the cheapest fix in the post. All 3 misses needed a fallback to go wrong quietly.

# silently picks prod when the variable is missing
target = os.environ.get("APP_ENV", "production")

# stops right here with a KeyError
target = os.environ["APP_ENV"]

The second line turns every miss in that table into a crash before any work happens. Default to the safe environment if you must have a default. Anything an agent can run should fail loudly on missing config.

What did this test cover?

Claude Code 2.1.318 on macOS with zsh, headless claude -p with Sonnet 5.5, project settings only, 97 sessions at about $2.08 of list-price usage. I didn't measure interactive sessions, other models or Linux, so run the one-liner above on your own setup first.

Do Claude Code env vars persist between Bash calls?

No. Claude Code env vars set with export in one Bash tool call are gone by the next, along with activated virtualenvs, functions, aliases and umask, because every call starts a fresh shell. Only cd carries over, and only in the main session on a successful command that stays inside the project. Claude works around this most of the time (77 of 80 runs here), but for anything that matters, define the state before launch with env in .claude/settings.json or a SessionStart hook that appends to $CLAUDE_ENV_FILE, and remove production defaults from your scripts.

Written by the developer behind Preterview, an interview prep platform.

Claude Code Token Usage: 97% of 44.6B Tokens Were Cache Re-Reads

jidonglab — Wed, 04 Nov 2026 12:35:35 +0000

Claude wrote 122 million tokens for me last month. To write them, it read 43 billion.

I got that ratio by parsing 30 days of my Claude Code token usage straight from the JSONL session logs on my laptop: 192,178 API requests, 889 main sessions and 5,696 subagents. I wanted to know which habit was expensive. The answer was mostly not the one I would have bet on, and the fix I liked best turned out to be weak.

TL;DR

97.03% of my 44.57B tokens were cache reads. Output was 0.27%.
At API list prices the month comes to $15,290: 56.6% reads, 31.7% cache writes, 11.6% output. I'm on a subscription, so this is a measuring stick, not my bill.
Subagents were 58% of the cost. Each one started with a median of 68,815 tokens before its first tool call.
Bash drove 67% of the re-reads I could pin on a tool. Claude ran cat/sed-style commands 42,453 times, more than the Read tool's 29,968.
Capping tool output at 5K tokens would save about 2.4%. Trimming 10K tokens from the startup context would save about 3.8%, with no change to any tool call.

Where does Claude Code token usage actually go?

Almost all of it goes to re-reading the conversation. Every agent step sends the whole context again, so a long session pays for its history on every request.

Bucket	Tokens	Share of tokens	List cost	Share of cost
Cache reads	43.24B	97.03%	$8,661	56.6%
Cache writes	1.20B	2.70%	$4,846	31.7%
Output	122.3M	0.27%	$1,781	11.6%
Uncached input	454K	under 0.01%	$2	under 0.1%

Before anyone reads this as "caching is bad": the same 43.24B tokens at normal input prices would cost about $133,000. Caching cut that line by roughly 15x. It's the cheapest token I buy, and it's still the biggest line on the sheet.

Size matters too. Requests above 200K tokens of context were 41.7% of all requests and 59.6% of the cost. My longest session ran 2,436 requests and took 2.3% of the month alone.

How do you measure Claude Code token usage from the logs?

Read ~/.claude/projects/**/*.jsonl and sum message.usage, but key it by message.id. One API response is written as several JSONL lines, one per content block, and each line repeats the same usage object.

That trap is big. My window had 400,880 assistant lines for 192,178 real requests. The naive sum said 93.0B tokens, 2.09x the true 44.6B. Resumed sessions add a smaller version of the same bug: 287 requests appeared in two files.

import glob, json, os, time
from collections import Counter

cutoff = time.time() - 30 * 86400
requests = {}  # message.id -> usage
root = os.path.expanduser("~/.claude/projects/**/*.jsonl")

for path in glob.glob(root, recursive=True):
    if os.path.getmtime(path) < cutoff:
        continue
    for line in open(path, errors="ignore"):
        if '"usage"' not in line:
            continue
        try:
            row = json.loads(line)
        except json.JSONDecodeError:
            continue
        msg = row.get("message") or {}
        if row.get("type") == "assistant" and msg.get("usage") and msg.get("id"):
            requests[msg["id"]] = msg["usage"]  # last line wins

totals = Counter()
for usage in requests.values():
    for key in ("input_tokens", "cache_creation_input_tokens",
                "cache_read_input_tokens", "output_tokens"):
        totals[key] += usage.get(key) or 0

print(len(requests), totals)

This takes about 35 seconds here, against 21GB of logs. The mtime filter keeps whole files, so totals land slightly above a strict 30-day cut. For dollars, I priced each request by its own model: Opus 5.5 and Sonnet 5.5 cache reads are both $0.20 per million, and 5-minute and 1-hour writes are 1.25x and 2x input.

Why does one tool result get billed dozens of times?

A tool result enters the context once and is then re-read on every later request until the context resets. Think of it as rent, not a one-time fee.

The median tool result in my logs was only 865 tokens, and it was re-read 23 times. Mean lifetime was 42.8 requests. The worst row was a 15,183-token MCP resource read that sat in a main session for 334 more requests: 5.07 million cache-read tokens, about $1, from a single call.

Rent is concentrated. The top 1% of tool calls produced 17.4% of it, and the top 10% produced 58.4%.

To estimate it, I took the context growth between consecutive requests, split it across that step's tool calls by character count, capped each at characters / 2.5, and multiplied by the requests left before the context dropped by more than 30%. It's an estimate, and a deliberately low one.

Which Claude Code tool costs the most?

Bash, by a wide margin. It wasn't because each result was huge. There were just a lot of them.

Tool	Calls	Avg tokens added	Est. re-read tokens	Share of tool rent
`Bash`	155,817	1,459	9.98B	67.2%
`Read`	29,968	1,956	2.60B	17.5%
MCP tools	13,227	877	0.78B	5.3%
`Write`	2,156	2,906	0.33B	2.2%

The surprise was inside Bash. I bucketed commands by their first verb, and file-reading verbs (cat, sed, head, tail, awk, nl, wc) won: 42,453 calls and 4.22B re-read tokens. So the shell read files 1.4x as often as Read did and generated 1.6x the rent.

Write is worth a look as well. The file body sits in the tool input, so a big generated file stays in context after it lands on disk.

How much do Claude Code subagents cost before doing any work?

About 19% of my month. The median subagent opened with 68,815 tokens of context and ran only 15 requests, so startup was a large part of its whole life.

The math on all 5,696 subagents:

First-request cache writes: $923. Only about 22% of first-request tokens were cache hits.
Re-reading that startup context on later requests: $1,934.
First-request cache reads: $17.

Total: $2,874, or 18.8% of the month, spent on context that existed before any subagent ran a command. Main sessions added another $856 of startup re-reads. Across everything, startup context accounted for 32.2% of all cache reads. Tool results took 34.4%, and the other 33.4% is assistant text, injected reminders and whatever my low estimator declined to assign.

My global CLAUDE.md is 25KB. Every agent carries it, along with tool definitions and the skill list. That file felt free because I only ever wrote it once.

Is idle cache expiry the real cost problem?

No. It's the obvious suspect, and it came in under 5%.

I counted a cache rebuild as any request that kept at least 70% of the context but read less than half of it from cache. There were 649 of those, rewriting 183.7M tokens. That's 15.3% of all cache writes and roughly $740 at my average write price, or 4.8% of the month.

Idle time doesn't even explain all of them. 257 rebuilds, the largest single group, happened less than 5 minutes after the previous request, and 61 lined up with a model switch. I didn't trace the rest. Worth fixing, but this isn't where the money is.

What would actually reduce Claude Code token usage?

Startup context first. I simulated three changes against the same 30 days, counting both re-reads and writes:

Change	Calls touched	Simulated saving	Share of month
Cap tool results at 5K tokens	10,691 (4.55%)	$369	2.4%
Trim 10K tokens from startup context	0	$582	3.8%
Cap tool results at 2K tokens	43,924 (18.71%)	$1,283	8.4%

The output cap was the fix I wanted to win. It's easy to explain and easy to build. It lost to deleting 10K tokens of standing instructions, and it would truncate over 10,000 results to do it.

The 2K cap looks best and is the one I trust least. Nearly one call in five would lose data, and an agent missing data runs another command. That's the part this simulation can't see.

The caveats, stated plainly:

This is one person's month, with a lot of automated subagent work. Your ratios will differ.
Dollars are API list prices, not what I paid.
Rent is an estimate, and a third of reads are unattributed.
The savings are simulated. I did not A/B them, so treat them as a ranking, not a promise.

So where does Claude Code token usage go? In my 30 days of logs, 97% of 44.6B tokens were cache re-reads of context the model had already seen, and output was 0.27%. Cost split 56.6% reads, 31.7% writes and 11.6% output, with subagents at 58% of the total. The biggest levers are how much context every agent starts with and how long sessions run past 200K tokens, ahead of output caps or idle cache expiry.

Written by the developer behind Preterview, an interview prep platform.

RoPE in bfloat16: 8,192 Token Positions Collapsed Into 897

jidonglab — Wed, 04 Nov 2026 00:28:57 +0000

Cast the integers 8,190 through 8,200 to bfloat16 and you get eleven copies of 8,192.

That one line is why RoPE in bfloat16 is a trap. Rotary position embeddings tell attention where each token sits, and the number format most LLMs run in can't count past 256 without rounding. There's no NaN and no crash. A 128-token unit test comes back bit-identical. I kept seeing "force float32" around rotary code, so I built a small NumPy simulator to find out how bad the damage really is.

Across an 8,192-token sequence, only 897 distinct positions survived. A head whose entire job is "look at the previous token" dropped from 100% to 3.5%.

TL;DR

bfloat16 has 8 bits of significand precision, so integers are exact only up to 256. Between 4,096 and 8,192, representable values sit 32 apart.
Casting position_ids to bfloat16 turns 8,192 positions into 897 distinct values. 89.1% of adjacent token pairs end up at distance zero.
In a synthetic previous-token head, top-1 accuracy fell from 8,191 of 8,191 queries to 286 (3.5%).
Rounding the final cos/sin tables, q and k to bfloat16 cost nothing measurable: still 8,191 of 8,191. Only the steps before cos and sin need float32.
Fix: keep inv_freq, positions and angles in float32, disable autocast around them, then cast cos/sin to the model dtype.

What does RoPE in bfloat16 actually break?

RoPE in bfloat16 breaks position information once a sequence passes 256 tokens, while every short test keeps passing. My setup used head_dim 128 and theta 10,000, with a synthetic attention head built so the correct answer is always token i - 1.

I tested all 8,191 query positions with 3 random heads. A hit means the previous token strictly beat every other key, so ties count as misses.

Query position	Hit rate with `position_ids` in bfloat16
1 to 255	100.0%
256 to 511	12.2%
512 to 1,023	0.0%
1,024 to 8,191	0.0%

Your tests live in the first row. Then it falls off a cliff.

Why can't bfloat16 count past 256?

bfloat16 keeps float32's 8 exponent bits but only 7 stored mantissa bits, which gives 8 bits of precision with the implicit leading bit. The gap between neighboring values doubles at every power of two.

Value range	Gap between bfloat16 values
0 to 256	1 or smaller
256 to 512	2
4,096 to 8,192	32
8,192 to 16,384	64

The first integer that can't be stored is 257, which rounds to 256. By 131,072 tokens you have 1,409 distinct positions and a worst-case error of 256.

Why does a rounded position wreck attention?

RoPE works because the attention score between two tokens depends only on their distance. It rotates q and k by angles proportional to position, and when you take the dot product, the absolute positions cancel and i - j remains.

Rounding destroys that cancellation, because round(i) - round(j) isn't i - j. Here's what happened to adjacent tokens in my 8,192-token run:

89.1% of neighbors had a distance of exactly 0
3.1% kept the correct distance of 1
The rest jumped, up to a gap of 32

So "the previous token" and "this token" are usually the same place. Attention can't separate keys sitting on one rounded position, and a whole block of 32 tokens gets identical rotation.

Which bfloat16 cast in RoPE actually hurts?

The casts before the trig functions hurt and the casts after them don't. I put bfloat16 rounding at one stage at a time and averaged 3 random heads. Attention weight is on the correct token, using a softmax sharpness I picked, so compare those rows to each other and ignore the absolute value.

What's in bfloat16	Hits of 8,191	Hit rate	Attention on right token
Nothing	8,191	100%	0.201
cos/sin tables, q and k	8,191	100%	0.201
`inv_freq` only	8,191	100%	0.201
Angles (`pos * inv_freq`)	504	6.2%	0.110
`position_ids` only	286	3.5%	0.098
Autocast style: inputs and angles	267	3.3%	0.094

A cos value lives between -1 and 1, where bfloat16's relative error is harmless. An angle lives in the thousands of radians, and at 8,000 radians a gap of 32 is about five full rotations. The output of cos tolerates rounding. Its input doesn't.

A sampled run at Llama 3's rope_theta of 500,000 showed the same collapse, so a larger base doesn't rescue you.

Why is inv_freq in bfloat16 harder to catch?

An inv_freq table in bfloat16 keeps RoPE's relative property, so nearby attention looks perfect while long-range attention drifts. That 100% row above is misleading.

Each frequency is off by up to 0.37%, the same at every position. Distance still cancels cleanly, but the model now rotates at frequencies it wasn't trained on, and the phase error grows with distance: 0.078 radians at 64 tokens, 1.25 at 1,024, 5.01 at 4,096.

I tested heads aimed at a fixed offset, using 500 random heads per distance:

Target offset	float32 hits	`inv_freq` bfloat16 hits
64 to 512	500 of 500	500 of 500
1,024	500 of 500	439 of 500
2,048	500 of 500	286 of 500
4,096	500 of 500	232 of 500

This is the nastiest variant. Local grammar stays fine and only the far end of your context gets quietly worse.

How does this bug get into real code?

It usually gets in through three ordinary lines, and none of them look like precision decisions.

A dtype borrowed from the activations:

# fine for x in float32, broken past 256 when x is bfloat16
t = torch.arange(seq_len, device=x.device, dtype=x.dtype)
freqs = torch.outer(t, self.inv_freq)

A buffer that gets downcast with the model. model.to(torch.bfloat16) casts floating-point buffers too, so a registered inv_freq is rounded on the spot. Calling .float() on it later changes the dtype and can't bring the lost bits back.

Autocast. If you build angles with a matmul of inv_freq and positions, autocast can run it in bfloat16. Hugging Face's Llama rotary code disables autocast around this exact computation, with a "Force float32" comment. Unsloth's early-2024 Gemma bug writeup also listed RoPE needing float32 instead of bfloat16.

How do you fix RoPE precision in PyTorch?

Compute everything up to cos and sin in float32 with autocast off, and cast only the result.

import torch

def rope_cos_sin(position_ids, head_dim, theta=10000.0, out_dtype=torch.bfloat16):
    device = position_ids.device
    with torch.autocast(device_type=device.type, enabled=False):
        exps = torch.arange(0, head_dim, 2, device=device, dtype=torch.float32) / head_dim
        inv_freq = 1.0 / (theta ** exps)
        angles = position_ids.to(torch.float32)[..., None] * inv_freq
        angles = torch.cat((angles, angles), dim=-1)
        return angles.cos().to(out_dtype), angles.sin().to(out_dtype)

inv_freq is rebuilt inside the function on purpose, so model.to() has no buffer to round.

Then add a test long enough to fail:

def test_rope_positions_stay_distinct():
    cos, _ = rope_cos_sin(torch.arange(8192), head_dim=128)
    assert torch.unique(cos.float(), dim=0).shape[0] == 8192

def test_inv_freq_stays_float32(model):
    for name, buf in model.named_buffers():
        if "inv_freq" in name:
            assert buf.dtype == torch.float32, name

In the simulator, that row count gave 8,192 for the correct path, 897 for bfloat16 positions and 7,533 for bfloat16 angles. It can't see the inv_freq drift, which is why the second test exists.

Is float16 safer for RoPE than bfloat16?

float16 is more precise and fails louder, and it's still the wrong choice. Its integers are exact up to 2,048, it keeps 4,097 distinct positions out of 8,192, and at 65,520 a position overflows to infinity. cos(inf) is NaN, so at least you'd notice.

bfloat16 has float32's range, so it never overflows. It just keeps rounding, and that's what makes it the quieter bug.

What did this test not measure?

This is a synthetic attention head in NumPy, with bfloat16 emulated by round-to-nearest-even on float32. I didn't run a trained model. Real networks spread position across many heads and lean on content too, so don't read 3.5% as a prediction for your model's accuracy. The PyTorch snippets above are the translation of that logic and weren't what produced these numbers.

To check a real model, run one long prompt through your RoPE path and through a float32 reference, then compare the logits at positions past 1,000.

So why does RoPE in bfloat16 collapse token positions?

RoPE in bfloat16 collapses token positions because bfloat16 has 8 bits of precision, so integers above 256 round to the nearest representable value, and the gap reaches 32 between 4,096 and 8,192. That left 897 distinct positions out of 8,192 in my test, and it broke the i - j cancellation RoPE depends on, taking a synthetic previous-token head from 100% to 3.5%. Keep inv_freq, positions and angles in float32 with autocast off, and cast only the final cos/sin tables to bfloat16.

Written by the developer behind Preterview, an interview prep platform.

Shipping a Mac Electron App Without Apple Notarization

jidonglab — Tue, 03 Nov 2026 20:15:17 +0000

Every new version of Refmade asks every Mac user to click "Open Anyway" in System Settings again. I chose that. The macOS signature check on the app reports one fatal finding, a missing notarization, and I decided to leave it there.

Refmade is a desktop app that wraps the Claude Code and Codex command-line tools and shows their work as an office floor. It is an independent app, not affiliated with Anthropic or OpenAI. The UI is Korean right now; that changes later in this series. Claude Code and Codex agents wrote most of the code at my direction. The decisions in this post are mine.

What the update notice does

The app does not update itself. About ten seconds after launch, and then once a day, it makes one GET request to a fixed address and reads a version number from the JSON that comes back. If the number is newer than its own, it shows a notice with a button that opens the download page in your browser. You can turn the check off from the menu.

The check is written to learn as little as possible about you and to trust the answer as little as possible. No app version, no ID and no cookie go out (the server sees your IP address, like any visit). The request has an 8-second limit for the whole answer, reads at most 16 KB, and follows no redirects. The app never opens or downloads an address that came from the answer, so a wrong answer can only produce a wrong number.

The answer can list a version per platform, and that matters for the Windows build later:

export function latestVersionOf(body, key = platformKey()) {
  if (!body || typeof body !== 'object' || Array.isArray(body)) return null;
  const platforms = body.platforms && typeof body.platforms === 'object' && !Array.isArray(body.platforms) ? body.platforms : null;
  if (platforms && Object.hasOwn(platforms, key)) return parseVersion(platforms[key]) ? platforms[key] : null;
  if (platforms) return null;   // a list without this platform: nothing for it yet
  return parseVersion(body.version) ? body.version : null;
}

On October 1 I saw it work for real. After I published 0.3.1, an installed 0.3.0 app picked it up as available from the production file.

The site's update log for Mac 0.3.9, October 10, 2026 (Korean). Headline: "When it needs my answer, the letters bounce." Three notes: my turn gets a color and two bounces, working gets a wave, news gets one bounce. Below: "This update is for Mac. The public Windows version is 0.3.2." Button: "Get latest version". The decision card on the right is public demo data.

Why notify and not auto-update

I decided on September 30 that a notice is enough for the first public version. I did not write down a motive beyond that, so here is the effect of the design. With an auto-updater, my server's reply would decide what runs on your machine. With a notice, the reply is one number, which the app can check against its own.

Installers are never overwritten. When I changed things after 0.3.5 was published, I shipped 0.3.6 instead of replacing the 0.3.5 file. Each build has a versioned file name and is served with immutable cache headers. Only the current version stays downloadable, so an old link returns 404. That is a real cost for anyone who bookmarked it.

The three ways to ship on a Mac

On October 1 I decided between these three.

Option	What users see	What it needs	My verdict
Notarized	Opens normally	Paid Apple Developer membership, notarization step per build	Not now
Ad-hoc signed, not notarized	"Open Anyway" once per version	Nothing paid (`identity: "-"` in electron-builder)	Chosen
Mac App Store	Store install	Sandbox, store review	Doesn't fit, see below

Ad-hoc signing seals the bundle (1,042 files in 0.2.3) so macOS can tell it was not modified, but it does not identify a developer. After it, syspolicy_check dropped from three fatal findings to one. The one left is the missing notarization.

The download page says plainly why the warning appears and where to click. The page explains that macOS blocks the first launch because the app has no Apple notarization, that it is not a malfunction, and which two clicks to make. It also repeats that the step comes back with every new version, so nobody is surprised by it after an update. Windows has the same shape: an unsigned installer, a "More info" link and a Run button, once per version.

Why the App Store is out

This is my reading of Apple's App Review Guidelines, not an official ruling. Guideline 2.5.2 says apps "may not read or write data outside the designated container area" and may not "execute code which introduces or changes features or functionality of the app". Refmade reads session logs that Claude Code and Codex keep in the user's home folder, and it launches those tools. 2.4.5(i) also requires Mac App Store apps to be sandboxed. Both go against what the app is for.

Building the Windows installer without Rosetta

I build on a Mac that runs macOS 27.0.1 and has no Rosetta. When I first tried a Windows installer there (for 0.2.2), electron-builder's NSIS step spawned makensis, an x86_64 binary, and failed with spawn -86.

The workaround was a flag that pulls a native NSIS toolset, -c.toolsets.nsis=1.2.1, passed to electron-builder --win nsis --x64. That produced Refmade-0.3.1-x64.exe, 133,331,616 bytes, with no code signature. Up to that point the file had never run on Windows. A 15-minute checklist was run on a real Windows PC. It passed, and only then did I publish 0.3.1.

Windows is still behind. The public Windows build is 0.3.2. The Mac build is 0.3.9. A 0.3.8 Windows installer exists, cross-built, but I did not publish it because I could not run it. That is what the per-platform list in latest.json is for. Windows users are not told about 0.3.9, because there is nothing for them to get.

The yellow stripe

Version 0.3.1 also fixed a visual bug. The game theme drew a yellow stripe along the left edge of every card title. The cause was a 12×7 pixel PNG, frame-title, whose left 9 pixels are stretched as a slice. A gold jewel sat inside those 9 pixels and got stretched to the card's height.

I asked for the stripe to go. The fix is a script, drop_title_jewel.py, that erases the jewel from four images and from two CSS data URIs. A test, game-art.test.mjs, now decodes the PNGs directly and counts gold pixels. The expected count is zero.

Takeaways

A notice that reads one number is a smaller promise than an updater. I can keep it.
Never overwrite a published installer. A new number costs nothing.
Say which platform got tested and on which version.
Read the store rules before building toward a store. Two guidelines were enough to rule it out for me.

The app link comes in the last post of the series.

If you ship an unsigned or ad-hoc signed desktop app, what did you put on the download page to make the "Open Anyway" step less scary?

json.dumps Writes NaN by Default: 31 LLM Reports JSON.parse Rejected

jidonglab — Tue, 03 Nov 2026 12:22:33 +0000

Nine people emailed me the same screenshot: a spinner and the line "Your report is being prepared." One of them had been looking at it for 51 hours.

The report was done. It had been done since about 40 seconds after their session ended. It was sitting in object storage as a perfectly normal-looking file. The problem was three characters inside it: NaN. Python json.dumps writes NaN by default, the browser's JSON.parse refuses to read it, and my pipeline had no step anywhere that cared about the difference.

TL;DR

Python's json.dumps has allow_nan=True by default, so float('nan') becomes a bare NaN token. That is not valid JSON.
Python's own json.loads reads NaN back without complaint, so round-trip tests pass. JavaScript's JSON.parse throws a SyntaxError.
My NaN came from np.nanmean over a rubric category where the LLM scorer returned null for every answer. An all-NaN slice returns NaN and only emits a RuntimeWarning.
31 of 1,940 reports (1.6%) over 16 days were unreadable. 26 of the 31 came from one interviewer style.
Fix: score fields as Optional[float] with allow_inf_nan=False in Pydantic, plus json.dumps(..., allow_nan=False) as a tripwire at every write.

What broke?

The system is Preterview, an interview practice platform I run (full disclosure: I built it, preterview.com/en). A candidate does a voice interview with one of three interviewer styles, an LLM scores each answer against a rubric, and a worker assembles everything into a written report.

The report pipeline looks like this:

The LLM scores each answer per rubric category (communication, technical depth, structure, and so on) and returns structured JSON.
A Python worker aggregates scores per category.
The worker writes report.json to object storage.
The frontend fetches that file directly and renders it.

Step 4 is where 31 reports died. The fetch succeeded, JSON.parse threw, my error handler caught it, and the error handler's fallback was "show the preparing state and poll again." Forever. I had built a very polite infinite loop.

Where did the NaN come from?

The NaN came from averaging a category that had no scores at all. The scorer prompt allows the model to say "I can't assess this category from this answer" by returning null. That's deliberate. I'd rather get a null than a confident 6/10 for system design on an answer about a team conflict.

The aggregation code turned those nulls into NaN on purpose so they'd be skipped:

import numpy as np

def category_mean(scores: list[float | None]) -> float:
    arr = np.array([np.nan if s is None else s for s in scores], dtype=float)
    return float(np.nanmean(arr))

np.nanmean ignores NaN values. That's what I wanted. What I hadn't thought through is what it does when every value is NaN:

>>> np.nanmean([np.nan, np.nan, np.nan])
RuntimeWarning: Mean of empty slice
nan

No exception. A warning and a NaN.

When does every answer in a category get null? When the interviewer never asks a question that touches that category. One of my three interviewer styles is a conversational, culture-fit style. It often runs a whole session without a single technical question. So the "technical depth" category came back all-null, nanmean returned NaN, and the NaN flowed downstream. That's why 26 of the 31 broken reports came from that one style.

Why does Python json.dumps write NaN by default?

Because allow_nan defaults to True. The Python docs are upfront about it: with allow_nan=True, out-of-range floats are written as their JavaScript equivalents (NaN, Infinity, -Infinity), and the docs note this is not strictly JSON compliant. RFC 8259 has no representation for NaN at all.

>>> import json
>>> json.dumps({"technical_depth": float("nan")})
'{"technical_depth": NaN}'

The part that let this survive testing: Python reads it back just fine.

>>> json.loads('{"technical_depth": NaN}')
{'technical_depth': nan}

My test suite had a round-trip test: build a report, dump it, load it, compare. Python on both ends, so it passed. The only consumer that enforced the actual JSON spec was the browser:

JSON.parse('{"technical_depth": NaN}')
// SyntaxError: Unexpected token 'N'

Why didn't my logs show anything?

Two reasons, and both are defaults.

First, Python's default warnings filter shows a given warning once per code location, not once per occurrence. My workers are long-running processes. The first all-null category after a deploy printed Mean of empty slice to stderr. The next 30 printed nothing. Across 16 days I found 11 instances of that warning in the logs, one per worker start, sitting between thousands of normal lines. Nothing about it said "a user's report is broken."

Second, the worker considered the job successful. json.dumps returned a string, the upload returned 200, the job was marked done. From the backend's point of view, every one of the 1,940 reports succeeded.

There was one more trap I tripped over while debugging. My first attempt to display the score did this:

>>> round(float("nan"))
ValueError: cannot convert float NaN to integer

But round(float("nan"), 1) quietly returns nan. The report used one decimal place, so this path never raised either. Every layer had a way to pass NaN along without anyone noticing.

Which serializers accept NaN and which reject it?

This is the table I wish I'd had. Same input, float("nan"), very different behavior:

Where	What happens
`json.dumps` (stdlib)	Writes `NaN`. Invalid JSON, no error
`json.dumps(..., allow_nan=False)`	Raises `ValueError`
`json.loads` (stdlib)	Accepts `NaN`, returns `nan`
Starlette / FastAPI `JSONResponse`	Raises `ValueError` (it uses `allow_nan=False`)
Pydantic v2 `model_dump_json()`	Writes `null` by default (`ser_json_inf_nan`)
orjson	Writes `null`
Postgres `jsonb`	Rejects the input as invalid JSON
Browser `JSON.parse`	Throws `SyntaxError`

The irony: if this report had gone through my FastAPI endpoint, it would have been a loud 500 on day one. If I'd stored it in a jsonb column, the insert would have failed. I chose to write it straight to object storage with the stdlib serializer, which happens to be the most permissive path in the whole table.

Also notice that "writes null" is not automatically safe. Pydantic and orjson would have produced valid JSON, but the frontend would then have rendered "Technical depth: null" or crashed on null.toFixed(1). Valid syntax, wrong meaning.

How do you stop NaN from reaching your JSON?

Make "not assessed" a real value instead of a float that pretends to be a number. Then make every serializer refuse NaN so the next mistake fails at write time, not in a user's browser.

The model change:

from pydantic import BaseModel, Field

class CategoryScore(BaseModel):
    category: str
    mean: float | None = Field(default=None, allow_inf_nan=False)
    assessed_answers: int

And the aggregation stops producing NaN in the first place:

def category_mean(scores: list[float | None]) -> float | None:
    vals = [s for s in scores if s is not None]
    return sum(vals) / len(vals) if vals else None

No numpy, no warning, and the empty case is a branch you can read instead of a side effect you have to remember.

Then the tripwire, at the single function that writes report files:

payload = json.dumps(report.model_dump(), allow_nan=False)

If anything upstream ever sneaks a NaN or an Infinity back in, this raises, the job fails, and my alerting sees a failed job instead of a successful one.

Two smaller changes:

Tests run with warnings as errors. filterwarnings = error::RuntimeWarning in pytest config. The "Mean of empty slice" warning would have failed CI on the first test with a skipped category, and I had one. It just never asserted anything about the output.
The frontend stops polling on parse errors. A SyntaxError is not "still preparing." It now shows an error state and pings me.

The frontend change was really the bigger lesson. The NaN was a bug. The spinner turned that bug into 51 hours of a user waiting for something that was never going to arrive.

How did I recover the 31 broken reports?

I re-ran aggregation only. The raw per-answer LLM scores were stored separately from the assembled report, so I didn't need a single new model call. All 31 reports rebuilt and rendered in under two minutes, with "Technical depth: not assessed in this session" where the NaN used to be. I emailed the nine people who'd written in, plus the other 22 who'd quietly given up.

If I'd only stored the final report, I'd have had to re-score 31 sessions and explain why their scores changed. Keep your raw LLM outputs. You will need to re-derive something from them eventually.

Does Python json.dumps write NaN by default?

Yes. Python's json.dumps uses allow_nan=True by default, so float('nan') is written as a bare NaN token, which is not valid JSON under RFC 8259. Python's json.loads accepts it back, so Python-only tests won't catch it, but JavaScript's JSON.parse, Postgres jsonb, and most strict parsers will reject the file. Pass allow_nan=False to make serialization raise a ValueError instead, and represent missing numeric values as None/null explicitly, especially for LLM-generated scores where "couldn't assess" is a legitimate answer that averaging code will happily turn into NaN.

Written by the developer behind Preterview, an interview prep platform.

asyncpg + PgBouncer Transaction Mode: 61 Prepared Statement Errors

jidonglab — Tue, 03 Nov 2026 00:19:58 +0000

On a weeknight around 9:40, fourteen people finished a session in my app inside the same three minutes. Three of them never got their report. The logs had the same line over and over: prepared statement "__asyncpg_stmt_1a__" already exists.

I had added PgBouncer two days earlier to fix a connection limit problem. It fixed that one and introduced a new one. If you run asyncpg + PgBouncer transaction mode, this post is the autopsy I wish I had read first: why it fails only under load, why the obvious fix only gets you 85% of the way, and what actually got me to zero.

TL;DR

asyncpg uses named prepared statements (__asyncpg_stmt_N__). Those live on one Postgres server connection.
PgBouncer transaction mode hands your next transaction to whichever server connection is free. Your statement name is now on the wrong connection, or collides with someone else's.
statement_cache_size=0 alone was not enough in my SQLAlchemy setup. Failures dropped from 61 to 9, not to 0, because SQLAlchemy's asyncpg dialect still calls prepare() with sequential names.
The fix that worked: disable both caches and give every prepared statement a unique name with prepared_statement_name_func. Zero errors across the next 11,800 writes.
Cost: p50 query time went from 3.1 ms to 3.6 ms because Postgres now re-parses every statement. I'll take it.

What was the setup?

The backend is FastAPI, SQLAlchemy 2.x async, and asyncpg, talking to a managed Postgres instance with a hard cap of 100 connections. Each app worker held its own SQLAlchemy pool. Scale out to enough workers and the pools alone eat the cap, so I put PgBouncer in front in pool_mode = transaction.

The system is Preterview, an interview practice app I run (full disclosure: I built it). A session is a live voice conversation, and when it ends the backend writes the transcript, per-question scores, and a written report in one burst of inserts. That burst is the write path that broke.

Locally, everything passed. Staging passed. Production at quiet hours passed. The failures only showed up when a dozen sessions ended at once, which is exactly the moment you can't debug interactively.

How bad was it?

In the first 48 hours behind PgBouncer, out of 2,140 session-end writes:

Error	Count
`DuplicatePreparedStatementError: ... already exists`	44
`InvalidSQLStatementNameError: ... does not exist`	17
Total failed writes	61 (2.9%)

Every single failure happened when concurrent session-end writes were above roughly 20 in flight. Below that, zero. That pattern is the whole clue.

Why does asyncpg break behind PgBouncer transaction mode?

asyncpg breaks behind PgBouncer transaction mode because it prepares statements by name on a server connection, and PgBouncer doesn't promise you the same server connection twice. The statement name is client-side state pointing at server-side memory that may not be yours anymore.

Here's the sequence in slow motion:

Worker A runs INSERT INTO reports .... asyncpg sends PARSE with the name __asyncpg_stmt_1a__ on server connection S1. Postgres stores it on S1.
The transaction commits. PgBouncer returns S1 to its pool.
Worker A's next transaction gets server connection S2. asyncpg's cache says "I already prepared this as __asyncpg_stmt_1a__", so it sends BIND with that name. S2 has never heard of it. "does not exist."
Meanwhile Worker B, on its own fresh client connection, also counts from the same starting point. It lands on S1 and tries to PARSE a different query as __asyncpg_stmt_1a__. S1 already has one. "already exists."

With low concurrency, PgBouncer tends to hand back the same idle server connection, so the names line up by luck. Under load, connections get shuffled and the luck runs out. That's why it looked like a flaky bug instead of a configuration bug.

The asyncpg error message even says this. The hint attached to the exception tells you that PgBouncer in transaction or statement mode doesn't support prepared statements properly and suggests statement_cache_size=0. I had been reading only the first line of the traceback for two days.

Does statement_cache_size=0 fix it?

Partly. Setting statement_cache_size=0 stops asyncpg from reusing cached named statements across transactions, which kills most of the "does not exist" errors. In my setup it took failures from 61 to 9 over a comparable 48-hour window. Better, not fixed.

First attempt:

engine = create_async_engine(
    DATABASE_URL,
    connect_args={"statement_cache_size": 0},
)

The remaining 9 were all "already exists." The reason: SQLAlchemy's asyncpg dialect keeps its own prepared statement cache (prepared_statement_cache_size), and it calls asyncpg's prepare() explicitly. Explicitly prepared statements still get the sequential __asyncpg_stmt_N__ names. Two client connections counting from the same starting point, landing on the same server connection, still collide.

So there are two caches and one naming scheme to deal with. I had fixed one cache.

What is the fix for asyncpg + PgBouncer prepared statement errors?

Disable both caches, and make every prepared statement name globally unique so collisions are impossible even when PgBouncer shuffles connections. SQLAlchemy 2.0 has a parameter built for exactly this: prepared_statement_name_func.

from uuid import uuid4
from sqlalchemy.ext.asyncio import create_async_engine
from sqlalchemy.pool import NullPool

engine = create_async_engine(
    DATABASE_URL,  # points at PgBouncer, not Postgres
    poolclass=NullPool,  # PgBouncer is the pool now
    connect_args={
        "statement_cache_size": 0,              # asyncpg's cache
        "prepared_statement_cache_size": 0,     # SQLAlchemy's cache
        "prepared_statement_name_func": lambda: f"__asyncpg_{uuid4()}__",
    },
)

Three changes, each for a separate reason:

statement_cache_size=0 stops asyncpg from assuming a statement prepared earlier still exists.
prepared_statement_cache_size=0 does the same for SQLAlchemy's layer.
prepared_statement_name_func replaces __asyncpg_stmt_1a__ with a UUID. Two workers can no longer pick the same name, so "already exists" becomes mathematically boring.

I also switched to NullPool. Running a SQLAlchemy pool on top of PgBouncer is double pooling: app-side connections sit idle holding PgBouncer client slots, and you get none of the multiplexing you added PgBouncer for. With NullPool, each checkout opens a cheap client connection to PgBouncer and closes it after.

Result: 0 prepared statement errors across the next 7 days and 11,800 session-end writes, including two evenings with more than 30 concurrent writes.

What does disabling prepared statements cost?

Every query now gets parsed and planned fresh instead of reusing a plan. For my workload, short inserts and indexed lookups, p50 query time went from 3.1 ms to 3.6 ms and p99 from 14 ms to 17 ms. The session-end burst is about 40 statements, so that's roughly 20 ms added to a write the user never waits on.

If your workload is heavy, repeated, complex queries, the parse cost could matter more. Measure your own p99 before and after rather than trusting my numbers.

Is there a fix on the PgBouncer side?

Yes. Newer PgBouncer versions (1.21 and later) can track protocol-level named prepared statements in transaction mode via max_prepared_statements. PgBouncer rewrites statement names and re-prepares them on whichever server connection you land on. I had to set it explicitly; it did nothing until I did.

[pgbouncer]
pool_mode = transaction
max_prepared_statements = 200

I tested this on staging and it also got the errors to zero, with the plan-caching benefit back. I still ship the client-side config in production for one reason: my managed Postgres provider controls the PgBouncer version, and I don't want correctness depending on a setting I can't see. If you own your PgBouncer, the server-side option is the better long-term answer.

What I'd check first next time

Read the whole exception, including the hint. asyncpg told me the answer on day one.
Reproduce with concurrency, not with retries. A single client never triggers this. I wrote a 40-coroutine script that fires the session-end write in parallel against staging PgBouncer, and it reproduced the bug in under a minute. Before that I had spent an evening adding retry logic, which turned 3% failures into 3% duplicate partial writes.
Count your caches. asyncpg has one, SQLAlchemy has one. Any ORM wrapping a driver may add another.
Never let session-level state cross PgBouncer transaction mode. Prepared statements are the famous one. SET without LOCAL, advisory locks, LISTEN, and temp tables break the same way.

So why does asyncpg throw "prepared statement already exists" behind PgBouncer?

Because asyncpg + PgBouncer transaction mode puts named, per-connection prepared statements on top of a pooler that reassigns server connections between transactions. Your statement name points at a connection you no longer own, or collides with another client's identical sequential name on the connection you got. The fix that took my production errors from 61 to 0: set statement_cache_size=0 and prepared_statement_cache_size=0, give prepared statements unique names with prepared_statement_name_func (UUIDs work), and drop app-side pooling with NullPool. If you control PgBouncer 1.21+, max_prepared_statements solves it on the server side and keeps plan caching. Expect a small parse cost; mine was 0.5 ms at p50.

Written by the developer behind Preterview, an interview prep platform.

GitHub Actions GITHUB_TOKEN Doesn't Trigger Workflows: 11 Tags, 0 Runs

jidonglab — Mon, 02 Nov 2026 12:18:17 +0000

A user opened an issue on one of my side projects: "Is this package abandoned? The README says 1.4.6 but npm says 1.3.0."

I checked GitHub. The tags were there: v1.3.1 through v1.4.6, eleven of them across nineteen days, each cut by my release workflow. I checked npm. Still 1.3.0. Then I checked the Actions tab for the publish workflow that was supposed to fire on every tag.

It had not run once. It hadn't failed or been skipped. There were simply no runs at all.

The cause is a GitHub Actions rule that is documented, deliberate, and still catches people every week: GITHUB_TOKEN doesn't trigger workflows. Anything your workflow does with the default token (push a commit, push a tag, open a PR, publish a release) will not start another workflow run.

TL;DR

Events created with the default GITHUB_TOKEN do not start new workflow runs. The only exceptions are workflow_dispatch and repository_dispatch.
This covers pushes, tags, pull requests, releases and comments made by github-actions[bot].
GitHub does this on purpose, to stop workflows from triggering themselves in an infinite loop.
Fixes: push with a GitHub App token or a PAT, call gh workflow run explicitly, or turn the second workflow into a reusable workflow called by the first.
If you switch tokens, pass the new token to actions/checkout. Otherwise git push keeps using GITHUB_TOKEN and nothing changes.

What did my broken release setup look like?

Two workflows. The first one I ran by hand when I wanted a release:

# .github/workflows/release.yml
name: release
on:
  workflow_dispatch:

permissions:
  contents: write

jobs:
  tag:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: |
          git config user.name "github-actions[bot]"
          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
          npm version patch
          git push --follow-tags

The second one publishes whenever a version tag shows up:

# .github/workflows/publish.yml
name: publish
on:
  push:
    tags: ["v*"]

jobs:
  npm:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          registry-url: https://registry.npmjs.org
      - run: npm ci && npm publish
        env:
          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

Each half works on its own. When I tagged locally and pushed from my laptop, publish ran perfectly. That's how the first few releases went out, and it's why I never suspected anything when I moved tagging into CI.

Once the tag push came from inside a workflow, publish went quiet. There was no error or warning anywhere, and the run list stayed empty.

Why doesn't GITHUB_TOKEN trigger workflows?

GITHUB_TOKEN doesn't trigger workflows because GitHub blocks recursion at the token level. Any event caused by the repository's automatic GITHUB_TOKEN is not allowed to create a new workflow run, except workflow_dispatch and repository_dispatch.

Imagine a workflow on push that formats code and pushes the result. If that push triggered workflows, it would trigger itself, format again, push again, and burn your minutes forever. GitHub's answer is blunt: the built-in token can change your repo, but those changes are invisible to the trigger system.

The rule is broader than "pushes," which is where people get bitten twice. All of these are silent when done with GITHUB_TOKEN:

Action done with GITHUB_TOKEN	Workflow that won't run
`git push` to a branch	`on: push`
`git push` a tag	`on: push: tags`
Open a PR (e.g. an auto-update bot)	`on: pull_request` CI checks
Create a release via API or `gh release create`	`on: release`
Comment on an issue	`on: issue_comment`

The PR case is the nastiest one. A dependency-update or codegen workflow opens a PR with GITHUB_TOKEN, your required checks never start, and the PR sits forever at "Expected — Waiting for status to be reported." It looks like CI is broken, when really CI was never asked to run.

How do I tell if this is my problem?

Look for the absence of a run, not a failed run. If this rule is your problem, the downstream workflow has no entry at all for that commit or tag.

Three quick checks:

Open the tag or commit on GitHub. If it was pushed by github-actions[bot], suspect this rule first.
Open the downstream workflow in the Actions tab. Zero runs for that ref means it was never triggered. A skipped run would show up as skipped, so a missing one points at the token.
Push the same kind of tag from your laptop. If the workflow runs instantly, your trigger config is fine and the token is the problem.

That last test took me 30 seconds and would have saved nineteen days.

How do I make a workflow trigger another workflow?

You have four options. I'll go from the one I now use to the one I'd only use in a pinch.

Fix 1: Merge them with a reusable workflow

Often the cleanest fix skips the second trigger entirely. Make publish.yml callable:

# publish.yml
on:
  push:
    tags: ["v*"]
  workflow_call:

Then call it as a job from release.yml:

jobs:
  tag:
    # ...same as before...
  publish:
    needs: tag
    uses: ./.github/workflows/publish.yml
    secrets: inherit

Nothing has to be "triggered," since publishing is just the next job. One catch: the called workflow checks out the ref of the caller, not the new tag, so check out the tag explicitly if your build reads the version from git.

Fix 2: Dispatch it explicitly

workflow_dispatch is one of the two exceptions, so GITHUB_TOKEN is allowed to start it on purpose:

permissions:
  contents: write
  actions: write

steps:
  # ...after git push --follow-tags...
  - run: gh workflow run publish.yml --ref "v$(node -p 'require(\"./package.json\").version')"
    env:
      GH_TOKEN: ${{ github.token }}

Add workflow_dispatch: to the on: block of publish.yml, or this call fails. And don't forget actions: write, which the default permission set often doesn't include.

Fix 3: Push with a GitHub App token

If you really need the push itself to count as a normal event (other people's workflows, PR checks, release triggers), push with a token that isn't GITHUB_TOKEN. A GitHub App installation token is the best option. It isn't tied to a person, it expires on its own, and commits show up as your-app[bot].

steps:
  - uses: actions/create-github-app-token@v1
    id: app
    with:
      app-id: ${{ vars.RELEASE_APP_ID }}
      private-key: ${{ secrets.RELEASE_APP_KEY }}

  - uses: actions/checkout@v4
    with:
      token: ${{ steps.app.outputs.token }}

  - run: |
      npm version patch
      git push --follow-tags

This is the setup that finally published 1.4.7. It also published nothing else, because I had already burned eleven version numbers.

Fix 4: Push with a fine-grained PAT

Same idea, less setup: create a fine-grained personal access token with Contents: read and write on that one repo, store it as a secret, pass it to checkout. It works. The downsides are that it acts as you, it expires on a date you'll forget, and it breaks when you leave the team.

Why did switching tokens still not work for me?

The first time I tried Fix 3, I generated the App token, exported it as an env var, and ran git push. Nothing triggered.

The reason is actions/checkout. By default it saves the token it used into the repo's local git config (persist-credentials: true), and later git commands in the job authenticate with that saved credential. If checkout ran with the default token, your git push goes out as GITHUB_TOKEN no matter what other secrets sit in the environment.

The fix is the line in the example above: pass the new token to checkout via token:. Checkout has to authenticate with the token you want the push to use.

Won't a real token bring back the infinite loop?

Yes. Once you push with an App token or PAT, you've opted out of GitHub's loop protection, so you need your own guard. If a push-triggered workflow commits and pushes with a real token, it will trigger itself.

Two guards that work:

Put [skip ci] in the bot's commit message. GitHub skips push and pull_request workflows for commits containing it.
Add a job condition such as if: github.actor != 'my-release-app[bot]'.

My release workflow runs on workflow_dispatch, so it can't loop. But the moment you combine "on push" with "push with a PAT," write the guard first.

What I changed

release.yml now uses a GitHub App token, passed to actions/checkout.
publish.yml also has workflow_dispatch, so I can re-run a missed tag by hand with gh workflow run publish.yml --ref v1.4.7.
A scheduled job compares the latest git tag with the npm latest version and fails loudly if they differ. That check would have caught this on day one, not day nineteen.

Does GITHUB_TOKEN trigger workflows?

No. In GitHub Actions, GITHUB_TOKEN doesn't trigger workflows: pushes, tags, pull requests, releases and comments created with the default GITHUB_TOKEN never start new workflow runs, except for workflow_dispatch and repository_dispatch events. GitHub does this to prevent recursive workflow loops. To chain workflows, call the second one as a reusable workflow, start it explicitly with gh workflow run, or push using a GitHub App installation token or a fine-grained PAT that you also pass to actions/checkout, and add a loop guard such as [skip ci] when the triggering event is push.

Written by the developer behind Preterview, an interview prep platform.

Python requests timeout=60 Let My LLM Stream Run for 38 Minutes

jidonglab — Mon, 02 Nov 2026 00:16:44 +0000

My nightly summarization job had timeout=60 on every request. One call ran for 38 minutes and 12 seconds anyway. Nothing crashed and nothing logged an error. The 6:00 a.m. report just didn't show up.

The Python requests timeout is not a deadline for the whole call. It caps how long you wait for the next bytes. Point it at a streaming LLM endpoint that emits a token every 70 milliseconds and that timeout never fires.

This post covers what I found over 14 nights and 33,600 calls, how I tracked it down, and the 20 lines that fixed it.

TL;DR

requests.post(..., timeout=60) sets a connect timeout and a read timeout. The read timeout is the longest allowed gap between bytes, not the total duration.
A streaming LLM response sends bytes constantly, so it never trips the read timeout, even when the model is stuck in a repetition loop for half an hour.
httpx works the same way: Timeout(60) covers connect, read, write and pool separately. There is no total-time cap by default.
Fix: set max_tokens, keep a short read timeout, and enforce a wall-clock deadline yourself (a check inside the stream loop, or asyncio.timeout() around the whole call).
After the fix: 0 missed reports in the next 14 nights. The longest call dropped from 38m12s to 1m51s.

The setup

Nothing exotic here:

A 7B instruct model served by vLLM through its OpenAI-compatible endpoint
A Python batch job with 6 worker threads
About 2,400 support tickets per night, each summarized into a short JSON blob
Streaming on, so I could log time-to-first-token

The call looked like this, and I'd guess half the LLM scripts on GitHub look the same:

r = requests.post(URL, json=payload, stream=True, timeout=60)
for line in r.iter_lines():
    if line.startswith(b"data: ") and line != b"data: [DONE]":
        chunk = json.loads(line[6:])
        parts.append(chunk["choices"][0]["delta"].get("content") or "")

I read timeout=60 as "this call dies after a minute." That was wrong.

What does the timeout parameter in Python requests actually do?

The Python requests timeout sets two limits: how long to wait for the TCP connection, and how long to wait between bytes on the socket. Passing one number sets both to that value. Passing a tuple like (5, 30) sets them separately. Neither one limits total request time.

The requests docs say this outright. The read timeout is the number of seconds the client will wait for the server to send a byte. Every byte that arrives resets that clock.

For a normal JSON API this almost never matters. The server is silent until it has an answer, so "time between bytes" and "total time" are basically the same number.

Streaming changes that. A server streaming tokens is never silent for long. My model produced a chunk roughly every 70 ms under load, so the read timeout restarted about 14 times a second. A 60-second read timeout couldn't fire unless the model stopped completely.

Why would an LLM stream run for 38 minutes?

It ran that long because the model got stuck repeating itself, and nothing told it to stop. I hadn't set max_tokens. vLLM's OpenAI-compatible server then allows generation up to whatever context window is left after the prompt. My context was 32k tokens, the prompts were about 1.5k, and that left roughly 30,000 tokens of room.

At around 15 tokens per second per request under batch load, 30,000 tokens takes about 33 minutes. My worst call ran 38. The numbers line up.

The outputs from those calls looked like what you'd expect:

"summary": "Customer reports login failure. Customer reports login failure.
Customer reports login failure. Customer reports login failure. ...

That repeated a few thousand times. Some tickets (very long ones, pasted stack traces, logs full of near-identical lines) push small models into these loops. It's rare, but over 2,400 calls a night it happens.

How rare, and how bad?

Here's what 14 nights of logs showed:

Metric	Value
Total calls	33,600
p50 duration	4.1 s
Calls over 5 minutes	61 (0.18%)
Longest call	38m 12s
GPU time spent on those 61 calls	~9.2 hours
Nights the 6:00 a.m. report was late	3

0.18% sounds harmless until you remember there were only 6 workers. When 2 or 3 runaway calls landed on the same night, a third to half of my capacity spent half an hour generating "Customer reports login failure." The other workers also slowed down, because every runaway sequence took a slot in vLLM's batch and a chunk of KV cache.

On the 3 late nights, 4 or more workers were stuck at once.

How I found it

The job wasn't failing, so I had no errors to read. What I had:

py-spy dump on the stuck process. All 6 threads were sitting in iter_lines() → urllib3 → socket.recv. So they weren't deadlocked. They were busy receiving data.
vLLM's /metrics endpoint. vllm:num_requests_running sat at 6 with nothing waiting. The server thought it was doing useful work.
Logging output length. Normal summaries came out at 150 to 400 tokens. The 61 slow calls averaged about 21,000.

Then I wrote a 15-line test server that sent one SSE line every 30 seconds and pointed requests.post(..., timeout=60) at it. It sat there for an hour until I killed it. That settled it.

How do you set a total timeout for a streaming request in Python?

Set a total timeout by enforcing a wall-clock deadline in your own code. Neither requests nor httpx gives you one. Check elapsed time on every chunk, cap generation with max_tokens, and close the connection when you give up so the server stops generating.

Here's what I run now:

import json
import time
import requests

def stream_completion(payload, deadline_s=90, read_timeout=20):
    payload = {**payload, "max_tokens": 1024, "stream": True}
    start = time.monotonic()
    parts, finish = [], None

    with requests.post(URL, json=payload, stream=True,
                       timeout=(5, read_timeout)) as r:
        r.raise_for_status()
        for line in r.iter_lines():
            if time.monotonic() - start > deadline_s:
                raise TimeoutError(f"exceeded {deadline_s}s wall clock")
            if not line.startswith(b"data: ") or line == b"data: [DONE]":
                continue
            choice = json.loads(line[6:])["choices"][0]
            parts.append(choice["delta"].get("content") or "")
            finish = choice.get("finish_reason") or finish

    return "".join(parts), finish

Four things changed, and each one matters:

max_tokens=1024. My longest legitimate summary was about 600 tokens. This alone puts a ceiling on the runaway case at roughly 70 seconds.
timeout=(5, 20). Short connect timeout, moderate read timeout. This still catches a server that goes completely silent.
The deadline check inside the loop. This is what actually caps total time. It only runs when a chunk arrives, so the true worst case is deadline_s + read_timeout, here 110 seconds. Know that number before you promise anyone an SLA.
The with block. Leaving it closes the socket. vLLM notices the disconnect and aborts the request, so the GPU stops generating junk for nobody. My server logs showed the aborts.

Is there a hard cap that doesn't depend on chunks arriving?

Yes. With async code, wrap the entire call in asyncio.timeout() (Python 3.11+). It cancels the task at the deadline whether or not bytes are flowing:

import asyncio
import httpx

async def stream_completion(client: httpx.AsyncClient, payload):
    async with asyncio.timeout(90):
        async with client.stream("POST", URL, json=payload) as r:
            async for line in r.aiter_lines():
                ...

Cancellation unwinds the async with and closes the stream. This is my preferred version. I kept the sync one above because the batch job is threaded and I didn't want to rewrite it at 2 a.m.

If you use an SDK built on httpx, like the OpenAI Python client, read what its timeout argument means for streams in the version you have installed. Don't assume it's a total deadline. Wrap the call in your own deadline either way. It costs one line.

What about the 58 calls that hit max_tokens?

Capping generation doesn't fix the loop. It just makes the loop cheap. In the 14 nights after the fix, 58 calls ended with finish_reason == "length", which in this job always meant a repetition loop.

I handle those explicitly now:

Never parse them as valid output. A truncated JSON blob full of repeated sentences is garbage.
Retry once at a slightly higher temperature (0.3 instead of 0). 49 of 58 came back clean.
Send the other 9 to a manual review queue with the ticket ID.

That's 9 tickets a fortnight for a person to look at, instead of 9 GPU-hours of "Customer reports login failure."

Results after the fix

Same job, same model, next 14 nights:

Longest call: 1m 51s (it hit max_tokens, then went through the retry path)
Calls over 5 minutes: 0
Late reports: 0
Total nightly runtime dropped by about 11%, mostly because runaway sequences stopped taking batch slots from healthy requests

The checklist I now apply to every LLM HTTP call

Always set max_tokens. A server default of "whatever context is left" is not a limit.
Use a tuple timeout: short connect, moderate read.
Add a wall-clock deadline that you enforce yourself.
Close the stream when you give up so the server stops generating.
Treat finish_reason == "length" as a failure until proven otherwise.
Log output token counts. That's how the 21,000-token outliers showed up.

So why didn't timeout=60 stop my request?

Because the Python requests timeout only limits connecting and the gap between received bytes. It never limits total request duration. A streaming LLM response sends a byte every few milliseconds, so the read timeout keeps resetting and a call can run as long as the model keeps generating: 38 minutes in my case, with no max_tokens set. To actually bound a streaming call, set max_tokens, keep a short read timeout, enforce a wall-clock deadline in your own loop or with asyncio.timeout(), and close the connection when the deadline hits so the server stops generating.

Written by the developer behind Preterview, an interview prep platform.

My Claude Code Sign-in Fix Needed Two Fixes of Its Own

jidonglab — Sun, 01 Nov 2026 18:29:43 +0000

On September 29 I removed a sign-in refusal from my app. The next day a review pass whose only job was to break that change found that it left some sign-ins reported as ready that then failed, and my repair of that opened a second hole. Both were caught before the first public release on October 1.

Refmade is a desktop app that wraps the Claude Code and Codex command-line tools and draws what they are doing as an office floor. It is an independent app, not affiliated with Anthropic or OpenAI. The UI is Korean right now; that changes later in this series. Claude Code and Codex agents wrote most of the code, working from specs I gave them. I planned the work, made the calls and reviewed the results. This post is about one call that turned out to be half right.

The problem: the app refused a sign-in the CLI accepts

The app has a "team" mode, where several agents plan, write and check a task. Until September 29 it only worked when Claude Code was signed in with a Claude account (the plan login). If you had an API key instead, the app answered with a reason code, AUTH_MODE_UNSUPPORTED, and stopped.

I read Anthropic's documentation around that refusal. My conclusion was that an app wrapping the CLI should not refuse a sign-in method the CLI itself supports. I'm a developer, not a lawyer, and this is not legal advice. It is simply how I decided to design it, so I asked for the refusal to be removed.

The app also tells people what it does with their data. Here is the first screen as it looked in a capture from the end of September:

Preview-mode first screen, late September (Korean UI, demo state). Title: "First, let's get an AI ready." Cards: Claude Code "ready", Codex "login needed". Notes below, translated: the app itself sends nothing to the internet; when you hand it a task, the Claude Code or Codex on your computer sends your request and needed file contents to Anthropic, OpenAI or wherever you connected, and checking that the AI is ready can also reach Anthropic; sign-in is managed by each AI tool and the app only checks whether and how you are signed in; tasks use up your plan's usage; AI answers can be wrong. A fifth note, cropped here, says the app is not made or endorsed by Anthropic or OpenAI.

One correction to that first note. It was true when I took the capture. Since version 0.3.0 the app also asks for a version number once a day, and since 0.3.3 it fetches banners for my other services in the office view. Neither request carries task content, but "sends nothing" is no longer an accurate sentence.

Review round one: the status check and the run disagree

Removing the refusal was a small change. I asked for an adversarial review (September 30), a few agents whose only task is to find how the change fails, and it found this.

Team mode starts Claude Code with --restricted, which keeps settings files out of the run. That includes your own user settings file. But before a run, the app asks claude auth status --json whether you are ready, and that command does read the user settings file. Some sign-ins live only there: an apiKeyHelper, or the Bedrock and Google Cloud login wizards.

Step	Reads user settings file?	What it sees
Readiness check (`auth status`)	yes	"signed in, ready"
Team run (`--restricted`)	no	no sign-in at all
Result		run fails, or quietly uses another stored login

I reproduced it with a fake API key and a local test server standing in for the API. The run either failed after the screen said "ready" or went out under a different Claude account that happened to be stored on the machine. For those people the screen now showed a green light, followed by a failure or a silent account switch.

The fix, and the second problem inside it

--settings still applies under --restricted, so the app can hand the file back by path. I never read its contents. The app checks that it exists, is at most 2 MB and is readable, and Claude Code reads the rest. This is the condition from providers/claude.mjs (one branch for third-party providers trimmed):

function signInFromSettings(signIn, env, platform) {
  if (!signIn) return false;
  const has = name => Boolean(envValue(env, name, platform));
  if (signIn.method === 'api_key_helper' || signIn.keySource === 'apiKeyHelper') return true;
  if (signIn.keySource === 'ANTHROPIC_API_KEY') return !has('ANTHROPIC_API_KEY');
  if (signIn.method === 'oauth_token') return !has('ANTHROPIC_AUTH_TOKEN') && !has('CLAUDE_CODE_OAUTH_TOKEN');
  return false;
}

My first version passed the file whenever a sign-in existed. A second review pass, one agent this time, found that the same file can contain permissions.additionalDirectories. Handing it back would let a run reach folders outside the one you picked. The restriction I was relying on was gone. So the file is passed only when the sign-in actually comes from it.

That condition has a trap. auth status reports a key from the environment and a key from the settings file with the same label, api_key and ANTHROPIC_API_KEY. The only way I found to tell them apart was to check whether the run's own environment holds the key, which is what has(...) does above. A related lesson from the Codex side: its -c override values are type-checked, so a value that looks right can still be rejected.

Other things the same review turned up

Team runs with Codex forced the provider to OpenAI. People using Azure or another provider had their tasks sent to OpenAI. I removed the forcing.
Claude Code and Codex could each receive the other's API key through the environment. Each tool now gets only its own family of variables (ENV_FAMILIES in the code).
The label "Claude account" appeared for any login. It now appears only for a plan login.

What passed, and what that proves

After the fixes: app tests 405, CLI tests 323, engine tests 401 plus one. I also ran mutation checks, which deliberately break the code and confirm a test fails. Real team runs through the actual CLIs finished in 6.1 seconds for Claude and 14.3 seconds for Codex.

Those numbers say the code does what my specs say. They say nothing about whether real users find sign-in easy. I have not run a usability test with real users.

A correction that belongs here

On September 29 I reported that the app's macOS signature check had zero fatal findings. That was wrong. When I ran syspolicy_check again, it listed three: two about the signature and one about missing notarization. Version 0.2.3 (September 30, 08:07) went out ad-hoc signed with 1,042 files sealed, and the count dropped from three to one. The remaining one is notarization, which I chose not to do. The next post covers that choice.

Takeaways

A status check and a run must read the same inputs. If they differ, "ready" is a promise the run can't keep.
Review the fix, not only the original change. Both regressions here came from the repairs, not from the code that was already there.
Reproduce with fake credentials and a local server before you trust a login fix.
Say what a test count measures. 405 passing tests is a statement about the specs.

The app link comes in the last post of the series.

If you wrap a CLI that has several ways to sign in, how do you keep your readiness check and your actual run reading the same configuration?

LLM-as-Judge Position Bias: Swapping A/B Flipped 87 of 300 Verdicts

jidonglab — Sun, 01 Nov 2026 12:15:13 +0000

In a head-to-head eval, my new system prompt beat the old one in 66% of comparisons. I already had the "we're shipping v2" message typed.

Then I swapped which answer went first and reran the exact same 300 pairs. The new prompt won 43%.

The judge, the answers, the rubric and temperature 0 were all the same. The only change was which answer sat in the slot labeled "A". That is LLM-as-judge position bias, and if your eval script calls judge(new, old) in a fixed order, your headline number is probably measuring it too.

TL;DR

LLM-as-judge position bias means a pairwise judge prefers whichever answer sits in a particular slot (usually the first) regardless of quality.
In my run, swapping A and B flipped 87 of 300 verdicts (29%), and 78 of those flips went to whichever answer was in slot A.
The bias shows up in close calls. When two answers are near-equal, slot position acts as the tiebreaker.
Fix: judge every pair in both orders. Count a win only when both runs agree, and treat disagreement as a tie.
Before trusting a judge, run the mirror test: put the identical answer in both slots. A fair judge has no reason to prefer either one.

What is LLM-as-judge position bias?

LLM-as-judge position bias is a judge model's tendency to pick an answer because of where it appears in the prompt, not because of what it says. It's one of the first biases documented for LLM judges (the MT-Bench / Chatbot Arena paper from Zheng et al. called it out back in 2023), and it's still easy to walk straight into.

It hits pairwise evals hardest, the "here are Answer A and Answer B, which is better?" setup. That's the setup most of us use to compare prompts, models or RAG configs, because it's easier than defining an absolute 1-10 scale.

How I caught it

My eval script looked like every eval script:

for q in questions:
    verdict = ask_judge(q.text, new_answer[q.id], old_answer[q.id])
    wins += verdict == "A"

The new answer always went in slot A. I didn't pick that order for any reason. It was just the order of the function arguments.

A coworker asked "did you swap them?" and I had no good answer. So I ran it again with the old answer in slot A. Here is what the 300 pairs did across both runs:

Outcome across both orders	Pairs
New prompt won both times	120
Old prompt won both times	93
Slot A won both times (verdict flipped)	78
Slot B won both times (verdict flipped)	9

Count the slot picks: slot A won 369 of 600 judgments, or 61.5%. A judge with no position preference would land somewhere near 50%.

The 66% headline was really 120 consistent wins plus 78 pairs where the new prompt just happened to sit in the favored seat.

Why does an LLM judge prefer the first answer?

An LLM judge prefers the first answer mostly because of how it reads: one token at a time, left to right. Answer A gets evaluated with nothing to compare it to. By the time the model reaches Answer B, it has already formed a picture of what "good" looks like for this question, and that picture came from A.

A few things stack on top of that:

Close calls get decided by the seat. I read the 87 flipped pairs myself. In most of them I couldn't pick a winner either: both answers were correct, worded differently, and about the same length. With no real quality signal to go on, the position prior decides.
Temperature 0 doesn't help. It makes the bias reproducible, which is worse in a way, because rerunning the same order gives you the same wrong answer and feels like confirmation.
The direction isn't universal. Some judge and prompt combinations favor slot B, especially with long answers where the most recent text is freshest. My judge favored A. Yours might favor B. Measure it before you assume a direction.
Format priors. "A" is the first option in countless multiple-choice examples the model has seen. Label tokens carry their own baggage before the model reads a word of the answers.

None of this means the judge is useless. On the 213 pairs where it agreed with itself, its picks matched mine on almost every one I spot-checked. The judge is fine at clear differences. On near-ties it isn't measuring quality at all.

How do you test an LLM judge for position bias?

Run the mirror test: build pairs where Answer A and Answer B are the exact same text, then ask the judge to pick. A fair judge has no basis to prefer either, so it should say TIE or split close to 50/50.

I ran 50 mirror pairs. My judge picked A 34 times, B 9 times and TIE 7 times. It was confidently picking a winner between two copies of the same text, and it usually picked the first one.

This takes ten minutes and tells you more about your judge than a week of reading verdicts. I now run it every time I change the judge model or the judge prompt.

How do you fix position bias in pairwise LLM evals?

Judge every pair twice, once in each order, and count a win only when both runs agree. If the judge contradicts itself, call it a tie. That's the whole fix:

JUDGE_PROMPT = """You are comparing two answers to the same question.

Question:
{question}

Answer A:
{a}

Answer B:
{b}

Briefly list the strengths and weaknesses of each answer.
Then end with exactly one line: VERDICT: A, VERDICT: B, or VERDICT: TIE"""


def ask_judge(question, a, b):
    out = call_llm(JUDGE_PROMPT.format(question=question, a=a, b=b), temperature=0)
    return parse_verdict(out)  # "A" | "B" | "TIE"


def judge_pair(question, new, old):
    run1 = ask_judge(question, new, old)  # new in slot A
    run2 = ask_judge(question, old, new)  # new in slot B

    if run1 == "A" and run2 == "B":
        return "new"
    if run1 == "B" and run2 == "A":
        return "old"
    # Self-contradiction, or a TIE in either run: no signal.
    return "tie"

Then report three numbers instead of one:

results = [judge_pair(q.text, new[q.id], old[q.id]) for q in questions]
n = len(results)

new_wins = results.count("new")
old_wins = results.count("old")
ties = results.count("tie")

print(f"new {new_wins/n:.0%}  old {old_wins/n:.0%}  tie {ties/n:.0%}")
print(f"new win rate on decisive pairs: {new_wins/(new_wins+old_wins):.0%}")

The tie rate is the important number here. It tells you how much of your eval set the judge can't actually separate. If it's 30%, a 5-point swing in your headline doesn't mean much.

Some other things that helped:

Give the judge a TIE option. Without it, you're forcing a coin flip on every near-equal pair, and the position prior is what decides the flip.
Ask for reasoning before the verdict. Put the verdict line last so the model writes its comparison before it commits to a letter. It doesn't remove the bias, but it cut my flip rate.
Use logprobs if your API exposes them. Take P("A") in one order and P("B") in the swapped order, then average. You get a soft score that's symmetric by construction.

Isn't randomizing the A/B order enough?

Randomizing the order per pair is better than a fixed order, but it doesn't fix the problem. Random assignment spreads the position bias evenly across both systems, so your aggregate win rate stops being skewed in one direction. That's real progress.

What it hides is the noise. Those 87 coin-flip pairs are still in your data, now split randomly between "new won" and "old won." Your headline drifts toward 50% and looks like an honest close race, when actually a third of your eval set says nothing at all. Rerun it with a different random seed and the number moves.

Running both orders costs twice the judge calls. On 300 pairs, that's 300 extra API calls. Shipping a prompt that was never better costs more.

Which other LLM judge biases show up alongside position bias?

Position bias rarely travels alone. Watch for these two:

Verbosity bias. Judges tend to favor the longer answer, even when the extra length is padding. If your new prompt makes answers 40% longer, some of its "wins" are word count. Check the win rate split by length difference.
Self-preference. A judge can favor outputs that sound like its own writing. If you're comparing Model X against Model Y and Model X is also the judge, be suspicious of a big X win.

Swapping the order fixes neither of these. They need their own checks.

So was my new prompt actually better?

Yes, a little. On the 213 pairs where the judge agreed with itself, the new prompt won 120 to 93, which is 56%. That's a real but modest improvement. It isn't the 66% blowout I nearly announced. I shipped it with a much quieter message and a note that 29% of the eval set was a toss-up.

How does LLM-as-judge position bias distort pairwise evals?

LLM-as-judge position bias makes a pairwise judge favor whichever answer occupies a particular slot, usually the first. It mostly decides near-tie comparisons, which can quietly inflate or deflate a win rate by double digits. In my eval, swapping A and B flipped 87 of 300 verdicts and turned a 66% win into 43%. The fix is to judge every pair in both orders, count only verdicts that agree, treat contradictions as ties, and report the tie rate next to the win rate. Run the mirror test, with the identical answer in both slots, before you trust any judge.

Written by the developer behind Preterview, an interview prep platform.

git push --force-with-lease Is Unsafe If Your Editor Auto-Fetches

jidonglab — Sun, 01 Nov 2026 00:13:30 +0000

I have typed git push --force-with-lease for years with the smug calm of someone wearing a seatbelt. Then one afternoon it overwrote two of my teammate's commits on our shared feature branch, and git didn't print a single warning.

No typo, no stale alias, no plain --force. The lease checked out fine. The problem was my editor, quietly running git fetch in the background every few minutes, which turned my seatbelt into a decorative strap.

This post explains exactly why git push --force-with-lease is unsafe when something auto-fetches, how to reproduce it in two minutes, and the flag that actually closes the hole.

TL;DR

git push --force-with-lease with no argument compares the remote branch to your remote-tracking ref (origin/feature), not to the commit you actually rebased on.
Any background git fetch (VS Code's git.autofetch, GitHub Desktop, a cron job, a prompt script) updates origin/feature to include your teammate's new commits. The lease now matches, and your push deletes their work.
The fix is --force-if-includes (Git 2.30+), which refuses the push if the remote tip was never part of your local branch's history.
Set it once: git config --global push.useForceIfIncludes true. It only kicks in alongside --force-with-lease, so plain --force stays dangerous.
Lost commits are usually recoverable from git reflog show origin/<branch>.

What does git push --force-with-lease actually check?

git push --force-with-lease checks one thing: that the branch on the remote still points to the same commit your local remote-tracking ref points to. If they match, the force push goes through. If they differ, git rejects it with "stale info".

That's it. It doesn't know which commit you started your rebase from. It doesn't know what you've looked at. It reads refs/remotes/origin/feature and assumes that's the last state you saw.

That assumption holds when you are the only thing that runs git fetch. It quietly breaks the moment anything else does.

Why does a background fetch break the lease?

A background fetch breaks the lease because it updates the exact ref the lease compares against, without touching your branch. Your view of the world goes stale, but git's record of "what you last saw" gets refreshed behind your back.

Here's the timeline from my afternoon:

14:02. I start an interactive rebase on feature/billing to squash some fixup commits. My local branch and origin/feature/billing both point at commit A.
14:09. My teammate pushes two commits, B and C, to the same branch.
14:11. VS Code's autofetch runs. origin/feature/billing now points at C. My local branch is still my rewritten history on top of the old base.
14:15. I finish the rebase and run git push --force-with-lease.
Git compares: remote says C, my tracking ref says C. Lease matches. Push accepted. B and C are gone from the remote.

The lease worked perfectly. It protected me against a state I had technically "seen". I hadn't seen anything. My editor had.

The git documentation actually warns about this in the --force-with-lease section: if something updates your remote-tracking refs in the background, the default lease form gives you no protection. Almost nobody reads that paragraph, because the flag's name sounds like it already handled it.

How do I reproduce the force-with-lease overwrite?

You can reproduce it locally with a bare repo and two clones, no network needed. Paste this into a scratch directory:

git init --bare origin.git
git clone origin.git me
git clone origin.git them

# me: create a branch with one commit
cd me
echo a > f && git add f && git commit -m "base"
git push origin HEAD:main
git branch -u origin/main

# them: pull it, add a fix, push
cd ../them
git pull origin main
echo fix > g && git add g && git commit -m "teammate fix"
git push origin HEAD:main

# me: rewrite my history, then a "background" fetch happens
cd ../me
git commit --amend -m "base (amended)"
git fetch            # pretend this was your editor

git push --force-with-lease
git log --oneline origin/main   # "teammate fix" is gone

The push succeeds. Now repeat the last section with the fix:

git push --force-with-lease --force-if-includes

This time git rejects the push, because the remote tip (teammate fix) never appeared in the history of your local branch. The rejection reason says the remote ref was updated since checkout, which is precisely the truth.

What does --force-if-includes do?

--force-if-includes makes git verify that the commit your remote-tracking ref points to is something your local branch has actually included at some point. It checks the reflog of your local branch: if the remote tip isn't reachable from any entry, git assumes you never integrated it and refuses the force push.

That's the missing check. The lease asks "did the remote change since my last fetch?" The include check asks "did I ever incorporate what's on the remote?" A background fetch can fake the first answer. It can't fake the second, because it never touches your local branch.

A few rules worth knowing:

It only works together with --force-with-lease. With plain --force it does nothing.
It's ignored if you pass an explicit lease value, like --force-with-lease=feature:abc123, since you've already told git exactly what to expect.
It requires Git 2.30 or newer. Run git --version before you rely on it.

How do I make --force-if-includes the default?

Set one config option and keep typing what you already type:

git config --global push.useForceIfIncludes true

Now every git push --force-with-lease also runs the include check. I also added an alias so my muscle memory never reaches for plain --force:

git config --global alias.pushf "push --force-with-lease --force-if-includes"

The explicit flag in the alias is redundant with the config, but it means the alias still works on a machine where I forgot to set the config. Cheap insurance.

When does --force-if-includes still let you overwrite work?

--force-if-includes still lets you overwrite work if your teammate's commits were in your local branch at any point, even briefly. Because the check looks at the reflog, "I pulled it once" counts as "I included it".

The classic case: you git pull --rebase, see your teammate's commits, then start an interactive rebase and drop them (on purpose or by fat-fingering d instead of p). The tip was in your reflog, so the check passes, and the push removes their work. Git is trusting that you made a decision. It can't tell the difference between a decision and an accident.

If you want the strongest guarantee, use an explicit lease captured before you start rewriting:

BASE=$(git rev-parse origin/feature/billing)
git rebase -i main
git push --force-with-lease=feature/billing:$BASE

Now the push only succeeds if the remote is exactly the commit you started from. Background fetches can't touch a shell variable. This is what I use for long-lived shared branches. For my own short branches, the config option is enough.

How do I recover commits lost to a force push?

You can usually recover them from the reflog of your remote-tracking ref, because the background fetch that caused the problem also recorded your teammate's commits locally. Run:

git reflog show origin/feature/billing

You'll see an entry from the fetch pointing at the old remote tip (in my case, C). Cherry-pick the lost commits back onto your branch:

git cherry-pick <sha-of-B> <sha-of-C>
git push --force-with-lease --force-if-includes

If your reflog doesn't have them, your teammate's local clone almost certainly does. That's how we recovered: a slightly awkward "hey, can you push again?" message and five minutes of cleanup. Lucky. On a branch with three people and a CI bot, it would have been a much longer afternoon.

Should I stop auto-fetching instead?

No. Auto-fetch is genuinely useful: it keeps your branch status accurate and shows incoming changes before you start a rebase. Turning it off trades a precise fix for a vague one, and you'd still get bitten by the next tool that fetches on your behalf.

The better move is to stop trusting the default lease form. Fetching should never be what decides whether a destructive push is safe.

So is git push --force-with-lease safe?

git push --force-with-lease is only safe if nothing else runs git fetch in your repo, because by default it compares the remote against your remote-tracking ref, and a background fetch silently updates that ref to include commits you never saw. With VS Code autofetch, GitHub Desktop, or any scheduled fetch, the lease will happily approve a push that deletes your teammate's work. Add --force-if-includes (or set git config --global push.useForceIfIncludes true on Git 2.30+) so git also checks that the remote tip was ever part of your local branch, and use an explicit --force-with-lease=branch:sha when you need a hard guarantee on shared branches.

Written by the developer behind Preterview, an interview prep platform.

DEV Community: jidonglab

Refmade Shipped Without Git: 2,764 Files in One Snapshot

How a project goes 17 days without git init

A second Mac, and source inside an installer

The first commit

17 gitleaks findings

What I lost, and what I changed

What I would check now

Claude Code Env Vars Don't Persist: 3 of 80 Runs Fell Back to Prod

TL;DR

Why don't Claude Code env vars persist between Bash calls?

When does cd persist in Claude Code?

How often does Claude actually trip on this?

How do you make Claude Code env vars persist?

Can Claude change the environment during a session?

Should scripts default to production at all?

What did this test cover?

Do Claude Code env vars persist between Bash calls?

Claude Code Token Usage: 97% of 44.6B Tokens Were Cache Re-Reads

TL;DR

Where does Claude Code token usage actually go?

How do you measure Claude Code token usage from the logs?

Why does one tool result get billed dozens of times?

Which Claude Code tool costs the most?

How much do Claude Code subagents cost before doing any work?

Is idle cache expiry the real cost problem?

What would actually reduce Claude Code token usage?

RoPE in bfloat16: 8,192 Token Positions Collapsed Into 897

TL;DR

What does RoPE in bfloat16 actually break?

Why can't bfloat16 count past 256?

Why does a rounded position wreck attention?

Which bfloat16 cast in RoPE actually hurts?

Why is inv_freq in bfloat16 harder to catch?

How does this bug get into real code?

How do you fix RoPE precision in PyTorch?

Is float16 safer for RoPE than bfloat16?

What did this test not measure?

So why does RoPE in bfloat16 collapse token positions?

Shipping a Mac Electron App Without Apple Notarization

What the update notice does

Why notify and not auto-update

The three ways to ship on a Mac

Why the App Store is out

Building the Windows installer without Rosetta

The yellow stripe

Takeaways

json.dumps Writes NaN by Default: 31 LLM Reports JSON.parse Rejected

TL;DR

What broke?

Where did the NaN come from?

Why does Python json.dumps write NaN by default?

Why didn't my logs show anything?

Which serializers accept NaN and which reject it?

How do you stop NaN from reaching your JSON?

How did I recover the 31 broken reports?

Does Python json.dumps write NaN by default?

asyncpg + PgBouncer Transaction Mode: 61 Prepared Statement Errors

TL;DR

What was the setup?

How bad was it?

Why does asyncpg break behind PgBouncer transaction mode?

Does statement_cache_size=0 fix it?

What is the fix for asyncpg + PgBouncer prepared statement errors?

What does disabling prepared statements cost?

Is there a fix on the PgBouncer side?

What I'd check first next time

So why does asyncpg throw "prepared statement already exists" behind PgBouncer?

GitHub Actions GITHUB_TOKEN Doesn't Trigger Workflows: 11 Tags, 0 Runs

TL;DR

What did my broken release setup look like?

Why doesn't GITHUB_TOKEN trigger workflows?

How do I tell if this is my problem?

How do I make a workflow trigger another workflow?

Fix 1: Merge them with a reusable workflow

Fix 2: Dispatch it explicitly

Fix 3: Push with a GitHub App token

Fix 4: Push with a fine-grained PAT

Why did switching tokens still not work for me?

Won't a real token bring back the infinite loop?

How a project goes 17 days without `git init`