DEV Community: jidonglab

A Local-First Plan, Then 4 Days of Scope Creep

jidonglab — Wed, 14 Oct 2026 18:08:23 +0000

The product plan my agent team wrote on September 21, 2026 had a rule in it: never tell users their data stays on their PC. By September 25 the same project had guestbooks, gifts and a cosmetics gacha with 266 items.

This is post 2 of "Building Refmade". Refmade is an independent desktop app (not affiliated with Anthropic or OpenAI) that draws your Claude Code and Codex sessions as a pixel-art office. Post 1 covered the agent team I couldn't see and the dashboard that went empty whenever an agent skipped its status report. Claude Code and Codex agents wrote most of the code and documents here at my direction; I made the calls and reviewed the results.

The plan was good

On September 21 I asked for a plan for a customer-facing version: other people install it locally, watch their agents on the web, and maybe send work to their own CLI from the web. My agents produced a plan, an architecture note, a user guide, a logic review, a source list, a verification list and two PDFs.

Two sentences in it still look right to me.

The first says the product must not tell users that their data never leaves the PC. A local CLI sends prompts and file contents to a model provider, so that promise would be false the first time someone ran a task. The second says the service must not pool customers' subscription tokens to run other people's work. The customer's PC opens an outbound connection to the server, and the server never exposes anything on the local machine.

I also decided early that the office is private by default, and that letting others visit it and showing task content are two separate switches. Price and launch date were left open, and the plan says in plain words that it grants no permission to sell.

The first-run screen still follows the first rule. Here is the preview-mode screen from the September 29 build.

Refmade first-run screen, preview mode, a late-September 0.2.x build, UI in Korean. Title: "I'll get the AI ready first". Cards: "Claude Code: ready" and "Codex: login needed". Notes below: the app itself sends nothing to the internet; when you hand over a job, the Claude Code or Codex on this computer sends the request and needed file contents to Anthropic, OpenAI or a provider you connected, and Claude Code may also contact Anthropic when the app checks readiness; login is managed by each AI tool; the work counts against your own plan; AI answers can be wrong.

That caption has an expiry date. This build said the app sends nothing itself. Later builds added a version check against my server and an ad-board request, and the current first-run screen says so in its first note. The note about what Claude Code and Codex send is still there. I measure what actually leaves the machine in a later post.

I also had a naming guide written for the Claude and Codex references the app needs. Product names and domains don't contain them, and there are no logos or look-alike mascots. The guide cites the official brand and usage pages and I designed around it. That's my reading, not legal advice.

Then the plan grew

What nobody wrote was a list of things the product would not do. Between September 22 and 25 the following got added, each one a reasonable step from the one before.

Added Sept 22 to 25	Detail
Standalone dashboard mode	PC pairing; only an anonymous ID, state and timestamps are sent; three sharing settings default to off
Web requests to a local CLI	natural-language request that can edit up to 8 tracked documents or code files
Social layer	mini home pages, guestbooks, gifts, decorating
Cosmetics gacha	266 items; rarity D/C/B/A/S at 50/30/15/4/1 percent
Two "world" layouts	a token pyramid and a construction-site village
Languages	Korean, Japanese, Chinese and English UI, with automatic post translation

The plan itself recommended visits, gifts and decorating for the first beta, so the drift was partly written into the document I had asked for. The gacha plan file lists its rarity odds as approved on September 21, the day of the plan. As of the September 24 report none of it was deployed. It ran locally and lived in review screenshots.

The September 24 report counted 62 confirmed defects dropping to 0, 104 final screens (13 screens, 4 languages, 2 widths), and passing test suites in the hundreds. Those numbers show how far agents can polish a feature that nobody has decided to keep. On September 25 I stopped and asked for a redesign from scratch. That's post 3.

The race I reproduced on a real database

One piece of that week is worth keeping on its own. The remote-execution feature had a setting to turn execution off. I wanted proof that "off" meant off, and unit tests against a mock couldn't show that.

On September 24 I had my agents start a throwaway PostgreSQL 17.10 cluster on a random loopback port, apply the three SQL files, and use only synthetic users and devices. Production was never touched. Seven checks ran on independent connections. Four passed, two reproduced a race, and one couldn't be judged.

A: BEGIN; lock device row; set execution = off      (not committed yet)
B: queue a task -> wrapper reads execution = on     (no lock taken)
B: waits for A's row lock (pg_stat_activity: Lock / transactionid)
A: COMMIT
B: resumes -> inserts a queued task                 <- should be refused

Simplified timeline of the two failing checks, from the September 24 test report.

The wrapper functions read the setting without locking, then called an original function that took the row lock. "Off" was committed first, and a new task (or a new lease, in the claim variant) appeared anyway. The report doesn't call this a strict ordering violation, because B started while A's change was still in flight. It does say what happened: after the off-switch was committed, new work was still created. The fix was to take FOR UPDATE on the device row first and read the setting afterward, inside one transaction. I had the two wrappers changed and reran the same seven checks on a fresh throwaway database: 7 of 7 passed. The unfixed SQL under the same harness still reproduced both races. Nothing was applied to production.

The seventh check taught me about my own tooling. The harness updated a table directly as a service role, which the schema rightly forbids, and got "permission denied". That was a test bug, so the report left the check as "not determined" instead of counting it either way. In the rerun it calls the product's own revoke function instead. An earlier failure was also a harness bug: it skipped the "started" state and sent "completed" straight away. And the real Claude delegation test stopped after four calls on a token budget, which I recorded as stopped, not as a pass.

What I'd change

The plan was the right document. It had no section called "not in scope". An agent will happily build the next adjacent thing, and a 266-item gacha is adjacent to a guestbook, which is adjacent to a profile page. The September 25 redesign came with exactly that: a written list of what to remove.

Most of the week is gone now. Today's app has no social layer, no gacha, no worlds and no web-to-PC execution; the code sits in an archive folder. What survived is the first rule and an office that is local-only.

Do you keep a written "not in scope" list for agent-built projects, and who is allowed to overrule it?

GitHub Actions fetch-depth: 1 Made My CI Skip Tests for 41 PRs

jidonglab — Tue, 13 Oct 2026 17:01:25 +0000

A PR broke main on a Tuesday. CI had been green. I opened the run log and found the test job had finished in 4 seconds, with one line of output: No code changes, skipping tests.

The PR changed 14 files in src/.

I scrolled back through the history. For three weeks, 41 pull requests had skipped the test suite. Every one of them was green. The cause was one default in actions/checkout: fetch-depth: 1, plus a || true I had written myself and felt clever about.

TL;DR

actions/checkout defaults to fetch-depth: 1. You get one commit, with no parents, no other branches and no tags.
That means git diff origin/main...HEAD, git merge-base, git describe, HEAD~1 and git rev-list --count HEAD all fail or return the wrong answer.
If your step has no shell: key, GitHub runs it with bash -e {0}, which has no pipefail. Add a || true and a fatal git error turns into an empty string.
Empty string means "nothing changed", so the tests get skipped and the job still passes.
Fix: use fetch-depth: 0, or on pull_request use fetch-depth: 2 with git diff HEAD^1 HEAD. Then make git failures stop the job.

What does fetch-depth: 1 actually fetch in actions/checkout?

It fetches exactly one commit: the one the workflow runs on. There are no parent commits, no remote branches besides the one checked out, and no tags. The result is a shallow repository whose history is a single node.

On a pull_request event that commit isn't even your branch tip. It's the merge commit GitHub creates at refs/pull/<N>/merge, which merges your PR head into the current base. That detail matters for the fix later.

You can see it for yourself with a throwaway workflow:

name: shallow-check
on: pull_request
jobs:
  look:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: |
          git rev-parse --is-shallow-repository
          git log --oneline | wc -l
          git branch -a
          git tag | wc -l
          git rev-parse HEAD~1 || echo "no parent"
          git rev-parse origin/main || echo "no origin/main"

On a repo with years of history, mine printed true, 1, a single remote ref for the PR, 0 tags, and both fallback messages. One commit is the whole repo as far as git knows.

Why did my changed-files check return nothing instead of failing?

The git command did fail, loudly, in stderr. The shell just never treated that as a failure. Here's the step I had written, which I'd bet a lot of monorepos have some version of:

- name: Detect code changes
  id: changes
  run: |
    CHANGED=$(git diff --name-only origin/main...HEAD | grep -E '^(src|tests)/' || true)
    if [ -z "$CHANGED" ]; then
      echo "No code changes, skipping tests"
      echo "skip=true" >> "$GITHUB_OUTPUT"
    fi

Three things went wrong at the same time.

1. origin/main doesn't exist. With a depth-1 checkout there's no origin/main ref, so git prints fatal: ambiguous argument 'origin/main...HEAD': unknown revision or path not in the working tree and exits 128.

2. No pipefail. If a run: step doesn't set shell:, GitHub Actions on Linux runs it as bash -e {0}. That's errexit without pipefail. A pipeline's exit status is the status of its last command, so the pipeline's result came from grep, not from git.

3. The || true. I added it because grep exits 1 when nothing matches, and "a docs-only PR" is a normal case I didn't want to fail. But || true doesn't care why the command failed. It swallowed git's fatal error together with grep's no-match.

So CHANGED came back empty. Empty means skip. The fatal message was right there in the log, gray text above a green checkmark, and I never read it, because nobody reads the logs of passing jobs.

The nasty part is that this works fine on your laptop. Your local clone has origin/main and full history. It only breaks on the runner.

Which git commands break in a shallow GitHub Actions checkout?

Pretty much anything that needs history, a second branch, or tags. Here's what I found when I went looking through my other workflows:

Command	What happens at depth 1
`git diff origin/main...HEAD`	fatal: unknown revision
`git merge-base origin/main HEAD`	fails, no base ref
`git diff HEAD~1`	fatal: unknown revision
`git describe --tags`	fatal: No names found
`git rev-list --count HEAD`	returns `1`, so build numbers reset to 1
`git log v1.2.0..HEAD` (changelog)	fails, tag not present
`git diff ${{ github.event.before }}` on push	fails, the before commit isn't fetched

The rev-list --count one is my favorite because it doesn't even error. It just quietly says your project has one commit. Any tool that builds a version string from tags or commit counts will fall back to some default version, and you'll ship artifacts labeled like a first release.

Affected-only test runners in monorepo tools hit the same wall when they compare against a base branch. If yours has a "fall back to running nothing" mode, check it.

How do I fix fetch-depth: 1 for git diff in GitHub Actions?

You've got three workable options. Pick based on what you actually need from history.

Option 1: `fetch-depth: 0` (full history)

- uses: actions/checkout@v4
  with:
    fetch-depth: 0

This fetches all history, all branches and all tags. origin/main...HEAD, git describe and changelog generation all work like they do locally. It's the right choice when you need tags or versioning.

The cost is clone time, and it grows with the repo. On my repo (around 6,000 commits) it added roughly 8 seconds. On a huge monorepo with a decade of history it can be a lot more, and you pay it on every job that checks out code. Only set it on the jobs that need it, not across the board.

Option 2: `fetch-depth: 2` with the merge commit (PRs only)

This is the cheap and exact option for "what did this PR change", and it's the one I use now.

- uses: actions/checkout@v4
  with:
    fetch-depth: 2
- run: git diff --name-only HEAD^1 HEAD

It works because of the merge-commit detail from earlier. On pull_request, HEAD is the merge of your PR into the base. Its first parent, HEAD^1, is the tip of the base branch. Depth 2 fetches the merge commit plus its parents, so HEAD^1 HEAD is exactly the set of changes the PR brings into base.

Two caveats. This only holds on the default pull_request checkout of the merge ref. If you check out github.event.pull_request.head.sha instead, HEAD^1 is just the previous commit on your branch. And it doesn't work for push events at all.

Option 3: fetch the base branch yourself

If you need a true merge-base on a deep branch but don't want full history:

git fetch --no-tags --depth=200 origin "$GITHUB_BASE_REF"
git merge-base "origin/$GITHUB_BASE_REF" HEAD

This still fails if the branch split off more than 200 commits ago. You can loop with git fetch --deepen=200 until merge-base succeeds, but at that point you're writing a clone strategy, and Option 1 is usually simpler.

How do I make git failures fail the job instead of passing?

Changing the depth fixes today's bug. Making the step loud fixes the next one. I added three guards.

Set shell: bash explicitly. With an explicit shell: bash, GitHub runs bash --noprofile --norc -eo pipefail {0}. Same shell, but now pipefail is on. You can set it once for the whole workflow:

defaults:
  run:
    shell: bash

Split git from grep. Let git fail on its own line, and only tolerate grep's no-match:

FILES=$(git diff --name-only HEAD^1 HEAD)
CHANGED=$(printf '%s\n' "$FILES" | grep -E '^(src|tests)/' || true)

If git diff fails now, errexit kills the step right there.

Assert the ref exists before diffing:

git rev-parse --verify --quiet HEAD^1 >/dev/null \
  || { echo "::error::Base commit missing. Check fetch-depth."; exit 1; }

After this, I broke it on purpose by setting the depth back to 1. The job went red on the first line with a message telling me exactly what to fix. That's the behavior I'd wanted all along.

Should "skip tests" ever be the default path?

My actual mistake wasn't the depth. It was designing a check where "I couldn't figure out what changed" and "nothing changed" produced the same output. When detection is uncertain, run everything. A wasted 6-minute test run costs you 6 minutes. A skipped one cost me 41 unverified PRs and a broken main.

If your workflow decides to skip work, it should be able to prove it has nothing to do. An empty string isn't proof.

So why does GitHub Actions fetch-depth: 1 break git diff?

Because actions/checkout with the default fetch-depth: 1 clones a single commit with no parents, no other branches and no tags, so any git command that compares against origin/main, HEAD~1 or a tag has nothing to compare against and exits with a fatal error. In a run: step without an explicit shell:, bash runs without pipefail, so a | grep ... || true pipeline turns that error into an empty result, and a changed-files check reads it as "nothing changed". Use fetch-depth: 0 when you need tags or full history, use fetch-depth: 2 with git diff HEAD^1 HEAD for pull request diffs, set shell: bash so pipefail is on, and make any skip logic fail closed.

Written by the developer behind Preterview, an interview prep platform.

I Built an Agent Team, Then Couldn't See What It Did

jidonglab — Mon, 12 Oct 2026 21:49:53 +0000

The first files of my agent team were written at 22:47 on September 20, 2026. By the next day I had a web dashboard to watch it. The dashboard was only as full as the agents were diligent: when one skipped its status report, that part of the office was empty.

This is the first of 16 posts about building Refmade, one every two days. Refmade is a desktop app that reads the session logs Claude Code and Codex already write on your machine and draws them as a small pixel-art office, so you can see which agent is doing what. It's an independent app, not affiliated with Anthropic or OpenAI. I'm a solo developer based in Korea, and I should say up front how the work is split: Claude Code and Codex agents wrote most of the code at my direction. I planned, decided and reviewed. The posts are about the decisions, the reversals and the bugs.

The series follows the real order of events. The next few posts cover a product plan that grew out of control, the switch to reading logs, and a map of how Claude Code and Codex write those logs. Later ones cover shipping a Mac app without notarization, a first Git commit that arrived far too late, and a gate that stopped a broken release. Failures get as much space as launches.

The app's UI is Korean right now. That changes later in this series, so every screenshot has an English caption.

The name belonged to a dead project

"Refmade" is older than the app. From March 20 to July 2, 2026, Refmade was a different product: a web design reference catalog with a gallery front end. It was a Next.js app with about 83 references, 131 commits and a payment integration wired in. One April 1 commit fixed a mobile out-of-memory crash in the gallery.

Then I stopped working on it. On August 7 I cleaned up my hosting account and cut 29 projects down to 12. The old Refmade project survived for one reason: a custom domain was attached to it, so I left it out of the deletion. About six weeks later, on September 21, that domain was pointed at the new office dashboard.

None of the gallery's code carried over. The new app started fresh on September 20. What carried over was a name and a domain that had been spared by accident.

The team I couldn't see

The September 20 files were not a product. They were a personal operating setup for Codex: eleven subagent definitions (PM, product, business, marketing, design, Figma, engineering, implementation, research, data analysis and review, with the team lead being the main conversation) plus a short README that says it was made at my request as a personal operating setup. I had a team. I could not tell what it was doing.

With several agents running in separate terminals, the question "who is working on what, and who is waiting for me?" had no good answer. So on September 21 I asked Codex for an "office": a web dashboard with one floor per task, each person's current job, the next step, what they're waiting for, meetings and records. Codex built it, and I reviewed the screens and sent back changes.

	v1 office (Sept 21)
Runs at	`127.0.0.1:4318` on my Mac
Remote view	password-protected page through a quick tunnel; works only while the Mac and the bridge are running
Refresh	every 5 seconds, paused in hidden tabs
Art	pixel-art floors, shelves and 12 seated characters, drawn with GPT Image
Checks passed	25 Python tests and 16 Node auth/deploy tests

The remote view was not a hosted service. A quick tunnel exposed my Mac's state read-only, so with the Mac off the office was unreachable. That was fine for a tool only I used. It also meant the first version of "see my agents from anywhere" was really "see my agents while my laptop is on".

The pixel art worked as intended. The data model is where it failed.

Where it broke

The first version asked each agent to record its own state. A small recorder script wrote those reports to files and the dashboard drew them. A separate observer could tell that a session window was open, but it listed those on their own layer, with no role, next step or meeting attached.

agent -> writes status report -> file -> dashboard -> "office"
              |
              +-- skips the report -> nothing to draw -> empty floor

An agent in the middle of a long task has no particular reason to stop and write a status note. When it didn't, that agent had nothing on the floor, and a floor with nobody on it looks like an idle team. The dashboard could not tell "nobody is working" from "nobody reported".

On September 21 the dashboard looked right, the tests passed and the screens matched the design. By September 25 I had decided the approach itself was wrong. How I replaced self-reports with log reading is the subject of post 3.

The lesson I took was narrow. A dashboard that depends on the observed system describing itself will be wrong in exactly the moments you most want it right. Anything I could read without the agent's cooperation was more trustworthy than anything the agent chose to tell me.

There was a smaller problem in the same dashboard: the logo's white background stuck out of the sign. I kept the original logo file untouched and fixed its position and blend mode inside the sign.

An illustration generated with Codex (gpt-image-2) for an earlier version of the product site: five characters, desks and a whiteboard with sticky notes. It is AI-generated art, not an app screenshot.

What the office looks like today

The current app draws a team per session from local logs instead of from reports. Here is a crop of version 0.3.9 with three teams.

Refmade 0.3.9, public demo data, UI in Korean. Left: area "Neighborhood workshop" (Claude, 1 team), task "Improve homepage booking screen", working with 2 art-team members. Middle and right: the two teams of area "Spring-light cafe" (Codex). Middle: task "Prepare new-menu promotion", working with 3 marketing-team members. Right: task "Sort customer-inquiry replies", marked "I need your answer". The workshop and cafe names are invented sample data.

Where it stands today

Refmade today runs on Mac (Apple Silicon) and Windows, it is free, and it needs Claude Code or Codex installed and logged in on your machine. Version 0.3.9 is the current Mac build. The later posts cover how each of those pieces got there, including the parts that went wrong.

The app link comes in the last post of the series.

Next up is the product plan my agent team wrote on September 21, and the days in which it grew guestbooks and a gacha.

If your agents run in separate terminals or sessions, how do you currently know which one is stuck waiting for you?

Sync Anthropic Client Blocked My asyncio Loop: Voice p95 Hit 11.4s

jidonglab — Mon, 12 Oct 2026 16:59:26 +0000

With one user on the server, my voice AI answered in 0.9 seconds. With eight users, the last person in line waited 11.4 seconds, and it sounded broken. The model, the network and the speech vendor were all fine. The problem was one line, client = Anthropic(), called from inside an async def. A sync Anthropic client was blocking the asyncio event loop, and every other session on the box stopped while it ran.

I spent two days blaming the wrong things. Here's the whole story, with the numbers.

TL;DR

Calling the sync Anthropic() client inside an async def handler blocks the asyncio event loop for the whole generation. No other coroutine runs until it finishes: no WebSocket frames, no audio, nothing.
Testing with one user hides it completely. Lag scales with how many sessions are active at once: my p95 time-to-first-audio went from 0.9s with 1 session to 11.4s with 8.
Find it with loop.set_debug(True) plus loop.slow_callback_duration = 0.1, or with a 10-line loop-lag monitor in production.
Fix: switch to AsyncAnthropic and async for over the stream. Wrapping the sync call in asyncio.to_thread only helps until the default thread pool fills (6 threads on a 2-vCPU box).
CPU-bound work like PDF parsing belongs in a ProcessPoolExecutor, not on the loop and not in a thread.

What was I building?

The server runs voice interview sessions. Full disclosure: I built it. It's Preterview, which runs realistic voice interviews with three interviewer styles, scores a resume or portfolio, and writes a report at the end. Each session is a WebSocket: audio comes in, a transcript comes back from speech-to-text, Claude writes the interviewer's next line, and text-to-speech streams it out.

It's one FastAPI process under uvicorn on a 2-vCPU machine. That's one event loop serving every session at once. If you've written async Python, you probably know where this is going.

Why did single-user testing look perfect?

Blocking the event loop costs nothing when there's nothing else waiting. With one user, a 2.3-second blocking call just looks like a 2.3-second LLM call. Nobody else is queued behind it, so nothing looks wrong.

My local runs, my staging runs and my demo calls all had one person on the line: me. Everything measured 0.9s to first audio. I shipped it.

What did the bug look like in production?

The first complaints were "the interviewer froze for a few seconds" and "the audio stuttered." Not every session. Only at certain times of day.

My first guesses, in order:

The speech-to-text vendor was slow at peak hours. (I moved regions. Nothing changed.)
TTS chunks were too small. (I made them bigger. Stutter got worse.)
The LLM was slow. (Claude's own timing for each call was flat across the day.)

That third clue was the one that mattered. The model took the same time at 3 a.m. and at 8 p.m. But the user waited much longer at 8 p.m. So the extra time was being spent inside my process, before or after the API call.

How do you detect a blocked asyncio event loop?

Turn on asyncio debug mode and lower the slow-callback threshold. asyncio will then log every step that holds the loop longer than the threshold, with the task that did it.

import asyncio
import logging

logging.basicConfig(level=logging.WARNING)

async def on_startup():
    loop = asyncio.get_running_loop()
    loop.set_debug(True)
    loop.slow_callback_duration = 0.1  # seconds; default is 0.1 in debug mode

On staging, under a replayed load test, the logs filled with lines like:

Executing <Task pending name='Task-412' coro=<session_turn() running at app/session.py:88>>
took 2.312 seconds

Debug mode adds overhead, so I didn't want it in production. Instead I added a lag monitor. It sleeps for 100ms and checks how late it woke up. If the loop is free, it wakes on time. If something is hogging the loop, it wakes late, and the delay tells you how long the loop was blocked.

async def loop_lag_monitor(interval: float = 0.1, warn_ms: float = 50):
    loop = asyncio.get_running_loop()
    while True:
        start = loop.time()
        await asyncio.sleep(interval)
        lag_ms = (loop.time() - start - interval) * 1000
        if lag_ms > warn_ms:
            log.warning("event loop lag %.0f ms", lag_ms)

In one week of production logs, I got 1,412 warnings over 100ms. When I matched them to the debug traces from staging, they broke down like this:

Source of the block	Warnings	Share
Sync Anthropic client calls	868	61%
`pypdf` resume parsing	325	23%
A sync `requests.post` webhook + a `time.sleep` in a retry helper	219	16%

The longest single block was 7.9 seconds: pypdf grinding through a 14-page resume while every live interview on the box sat silent.

Why does a sync client inside async def block everything?

An async def only yields control at an await. A sync network call has no await, so the loop is stuck until the socket read returns. My code looked harmless:

client = Anthropic()  # sync client

async def session_turn(ws, history):
    with client.messages.stream(model=MODEL, max_tokens=400, messages=history) as stream:
        for text in stream.text_stream:      # blocks the loop waiting for each chunk
            await ws.send_text(text)         # yields, but only for a moment

The await ws.send_text made it look async. But for text in stream.text_stream is a sync iterator doing a blocking socket read between every chunk. In practice, the loop was frozen for the whole generation, apart from a few microseconds per send.

Every other session's incoming audio frames, outgoing TTS chunks and heartbeats waited in line. With eight people mid-answer at once, the last one in line waited for seven generations before their own started.

How bad was it under load?

I wrote a load test that replays recorded answers across N concurrent fake sessions and measures time-to-first-audio (TTFA). It counts from the moment the user stops talking to the moment the first TTS byte goes out. Here's p95 over 200 turns at each level:

Concurrent sessions	Sync client	Sync client in `to_thread`	`AsyncAnthropic`
1	0.9s	0.9s	0.9s
4	5.2s	1.0s	0.9s
8	11.4s	3.4s	1.0s

The middle column is the part I nearly got wrong.

Is asyncio.to_thread enough to fix a blocking SDK call?

Only up to a point. asyncio.to_thread runs the call on the loop's default ThreadPoolExecutor, which since Python 3.8 has min(32, os.cpu_count() + 4) workers. On my 2-vCPU box, that's 6 threads.

My first fix was to wrap the sync stream in to_thread, and at 4 sessions it looked solved. At 8 sessions, two turns sat in the executor queue waiting for a free thread, and p95 rose to 3.4s. The loop was no longer frozen, but a new bottleneck had appeared: the thread pool.

You can make the pool bigger. But at that point you're running a thread per in-flight LLM call so you can keep using a client that wasn't built for your runtime. The SDK ships an async client. Use it.

What's the correct fix?

Use AsyncAnthropic and iterate the stream with async for, so each wait for the next chunk is a real await:

from anthropic import AsyncAnthropic

client = AsyncAnthropic()

async def session_turn(ws, history):
    async with client.messages.stream(model=MODEL, max_tokens=400, messages=history) as stream:
        async for text in stream.text_stream:
            await ws.send_text(text)

That's a three-word diff, and it took p95 at 8 sessions from 11.4s to 1.0s.

The other 39% of the warnings needed different fixes:

pypdf resume parsing is CPU-bound pure Python. Moving it to to_thread keeps the loop mostly responsive, but it still competes for the GIL with the loop thread. I moved it to a ProcessPoolExecutor(max_workers=2) with loop.run_in_executor. Parsing a resume now costs other sessions nothing.
requests.post to a webhook: replaced it with httpx.AsyncClient.
time.sleep(2) in a retry helper: changed to await asyncio.sleep(2). That one was mine, it was in a "utils" file, and it had been freezing the server for two full seconds on every retry.

After the changes, the lag monitor logged 9 warnings over 100ms in the following week, all under 180ms, all from garbage collection pauses. That's down from 1,412.

How do I stop this from coming back?

Make it fail loudly instead of degrading quietly:

Keep the lag monitor in production and alert when the p99 loop lag goes over 100ms. It's 10 lines and costs almost nothing.
Load test with concurrency, not just latency. A single-session test can't catch this kind of bug. My CI now runs the replay at 1, 4 and 8 sessions and fails if p95 at 8 is more than 1.5x the value at 1.
Grep for the usual suspects in async code paths: Anthropic(, OpenAI(, requests., time.sleep, and sync file or PDF parsing. A lint rule that flags the sync clients in modules full of async def caught two more before they shipped.

The lesson I keep relearning: in async Python, the slowest blocking call sets the latency for every user on the box, not just the one who triggered it.

So why did a sync Anthropic client block my asyncio event loop?

A sync Anthropic client blocks the asyncio event loop because its network reads are ordinary blocking calls with no await. Even inside an async def, the single loop thread is stuck until the response or the next stream chunk arrives, and every other coroutine waits. One user never notices. Concurrent users queue up behind each other: my voice server went from 0.9s to 11.4s p95 time-to-first-audio at 8 sessions. Detect it with loop.set_debug(True) and slow_callback_duration, or a loop-lag monitor. Fix it with AsyncAnthropic and async for, not asyncio.to_thread, which just moves the queue into a 6-thread pool. Send CPU-bound work to a process pool.

Written by the developer behind Preterview, an interview prep platform.

Python json.dumps ensure_ascii: 24% More Claude Tokens, Wrong Kanji

jidonglab — Mon, 12 Oct 2026 04:57:43 +0000

The Python port of my support-ticket classifier used 24% more input tokens than the TypeScript version it replaced. Same model. Same system prompt. Same 2,000 tickets in the replay set.

I spent an afternoon diffing prompts before I saw it. The TypeScript prompt said 東京. The Python prompt said 東京.

One default argument in json.dumps, ensure_ascii=True, was quietly rewriting every non-ASCII character in my prompts into hex escapes. It cost tokens, and worse, it made Claude misread Japanese customer names.

TL;DR

Python's json.dumps defaults to ensure_ascii=True, which turns é into é and 東 into 東. JavaScript's JSON.stringify doesn't do this.
If you paste json.dumps(...) output into a prompt string, the model sees literal backslash-u hex, not the characters.
On my 2,000-ticket replay set, that raised average input tokens from 1,180 to 1,461 per request (+24%). Japanese ticket payloads alone grew about 3.1x.
Extraction quality dropped too: 29 of 340 Japanese tickets came back with a wrong character in a name or address, versus 3 of 340 with raw UTF-8.
Fix: json.dumps(obj, ensure_ascii=False), plus force_ascii=False in pandas to_json, plus a test that fails if \u shows up in a rendered prompt.

What does json.dumps ensure_ascii actually do?

ensure_ascii=True (the default) makes json.dumps escape every character outside ASCII as a \uXXXX sequence, and characters outside the Basic Multilingual Plane (most emoji) as a surrogate pair of two escapes. The output is valid JSON and decodes back to the same data. It's a transport-safety feature from an era when you couldn't trust a pipe to carry UTF-8.

>>> import json
>>> json.dumps({"city": "München", "name": "東京"})
'{"city": "M\\u00fcnchen", "name": "\\u6771\\u4eac"}'
>>> json.dumps({"city": "München", "name": "東京"}, ensure_ascii=False)
'{"city": "München", "name": "東京"}'

For a JSON file or an HTTP body, this is harmless. The receiver parses the JSON and gets 東京 back. The problem only starts when the JSON is not the transport but the content.

Why does ensure_ascii only break prompts and not API calls?

Because a prompt is a string that nobody parses as JSON before the model reads it. Here's the line that bit me:

prompt = f"""Classify this ticket and extract the customer's name and city.

<ticket>
{json.dumps(ticket, indent=2)}
</ticket>"""

client.messages.create(
    model=MODEL,
    max_tokens=512,
    system=SYSTEM_PROMPT,
    messages=[{"role": "user", "content": prompt}],
)

The SDK then serializes the whole request body to send over HTTP. That outer layer is fine. But the inner 東 is now part of a string value, so the SDK escapes the backslash, the API decodes it once, and the model receives the six literal characters \, u, 6, 7, 7, 1.

The model doesn't get 東. It gets a hex puzzle and has to solve it in its head.

The TypeScript version did the same thing with JSON.stringify(ticket, null, 2), which leaves non-ASCII characters alone. That's why nobody noticed during the port: the code looked identical line for line.

How many extra tokens does ensure_ascii cost?

On my data, 24% more input tokens per request on average, and about 3x more on the ticket payload for Japanese text. I measured it with Anthropic's count_tokens endpoint against the same 2,000 tickets rendered both ways:

Ticket language	Share of set	Payload tokens, escaped vs raw
English	38%	1.00x
German	22%	1.06x
Spanish / Portuguese	13%	1.09x
Japanese	17%	3.1x
Other (French, Polish, emoji-heavy)	10%	about 2x

Average input per request went from 1,180 tokens to 1,461. About 600 of those are a fixed system prompt, so the payload part grew by almost half.

The pattern makes sense once you look at what the tokenizer is fed. A German ticket only escapes the occasional ü or ß, so the damage is a few percent. A Japanese ticket escapes every single character, and each one becomes a dozen-ish ASCII characters of hex that tokenize badly. Emoji are the worst per character: one 😀 becomes 😀, twelve characters of surrogate pair.

At about 40,000 tickets a month, the extra 281 tokens per request comes to roughly 11 million input tokens I was paying for so the model could read hex. Multiply by your own per-token price; mine was not a fun number.

Does escaping non-ASCII hurt LLM accuracy?

Yes, at least for extraction. Classification barely moved, but copying exact characters out of escaped text got noticeably worse.

The bug actually surfaced through accuracy, not cost. A support agent flagged a ticket where the extracted customer name contained a kanji that appeared nowhere in the original message. I grepped the raw ticket for the extracted name: zero hits.

So I scored the 340 Japanese tickets from the replay set. A result counted as wrong if the extracted name or city string didn't appear verbatim in the original ticket text:

Escaped prompts: 29 of 340 had at least one wrong character (8.5%)
Raw UTF-8 prompts: 3 of 340 (0.9%)
Category labels: within one point of each other across both versions

My working theory: common characters like 東京 show up escaped all over the training data, so the model decodes them reliably. Rare kanji in personal names don't, and the model lands on a character whose code point is close but whose meaning and shape are unrelated. Hex nearness is not visual nearness. A one-digit slip in 髙 gives you a completely different character, and nothing in the surrounding text looks wrong to the model.

Caveats, since I'd want them if I were reading this: one model, one run per version, 340 tickets, exact-match scoring. It's a strong signal on my data, not a benchmark. But the direction is not subtle.

Where else does ensure_ascii hide in a Python LLM pipeline?

Anywhere you serialize data into prompt text. I found three more spots after fixing the first one:

pandas.DataFrame.to_json() defaults to force_ascii=True. I used it to inline a few rows of order history into prompts. Same bug, different parameter name.
Few-shot examples loaded from a file and re-dumped. The loader read UTF-8 JSON correctly, then a helper called json.dumps(example) to format it. Every Japanese few-shot example was escaped.
Tool results. My agent returned tool outputs as json.dumps(result) in tool_result content. Search results with French product names went to the model as café.

Places that were already fine: Pydantic v2's model_dump_json() writes raw UTF-8, and orjson.dumps() always outputs UTF-8. If you're on either, you're probably safe. Check anyway; I was sure about the TypeScript-to-Python port too.

How do you fix it and keep it fixed?

Pass ensure_ascii=False everywhere JSON ends up in a prompt, then add a test so the next helper function can't bring it back.

I centralized prompt serialization in one function:

def to_prompt_json(obj) -> str:
    # Compact separators, raw UTF-8. This goes to a model, not a wire.
    return json.dumps(obj, ensure_ascii=False, separators=(",", ":"))

Dropping indent=2 for compact separators saved another 6% on payload tokens. I checked extraction accuracy on the same set and saw no difference, but test that on your own data before copying it; some prompts read better indented.

Then a guard that runs in CI against every prompt template with a fixture that includes Japanese, German, and an emoji:

import re

ESCAPE = re.compile(r"\\u[0-9a-fA-F]{4}")

def test_prompts_contain_no_unicode_escapes():
    for name, render in PROMPT_TEMPLATES.items():
        prompt = render(MULTILINGUAL_FIXTURE)
        assert not ESCAPE.search(prompt), f"{name} leaks \\u escapes"

It caught the pandas call on its first run. And a grep for the lazy version of the audit:

grep -rn "json.dumps(" src/ | grep -v "ensure_ascii=False"
grep -rn "to_json(" src/ | grep -v "force_ascii=False"

After the fix, the Python version came in at 1,148 average input tokens: 3% under the TypeScript original, thanks to the compact separators. Japanese extraction errors dropped back to 3 of 340.

The lesson I'm keeping

The scary part isn't that ensure_ascii exists. It's that every layer looked correct. The JSON was valid. The request succeeded. The logs printed fine, because I was logging the ticket, not the rendered prompt. The model answered confidently. The only visible symptoms were a slightly bigger bill and a wrong character in a name that a human happened to check.

Log the exact string the model receives, at least for a sample. Read it once with your own eyes. Most prompt bugs I've had were visible in about five seconds of looking at the real thing.

So does json.dumps ensure_ascii waste LLM tokens?

Yes. Python's json.dumps escapes every non-ASCII character as \uXXXX by default, and when that output is pasted into a prompt, the model reads literal hex instead of the original text. In my 2,000-ticket test with Claude, it raised average input tokens by 24%, tripled the payload size for Japanese tickets, and raised Japanese name-extraction errors from 3 to 29 out of 340. Use json.dumps(obj, ensure_ascii=False) (and force_ascii=False for pandas to_json) anywhere JSON becomes prompt text, then add a test that fails if \u escapes appear in a rendered prompt.

Written by the developer behind Preterview, an interview prep platform.

Cosine Similarity Threshold 0.8: Embedding Anisotropy Broke My RAG

jidonglab — Sun, 11 Oct 2026 16:55:54 +0000

My RAG bot had exactly one safety rule: if the best chunk scores below 0.8 cosine similarity, answer "I don't know." That cosine similarity threshold ran for months without complaint. Then I swapped the embedding model, redeployed, and the bot said "I don't know" to everything. Including "how do I reset my password," which has its own dedicated doc page.

Nothing crashed. No errors. Retrieval still returned the right chunk at rank 1. My gate just threw it away, every single time.

The worst part: when I went digging, I found the old setup had been broken too. In the other direction.

TL;DR

A cosine similarity score has no universal meaning. 0.8 can be "near-duplicate" for one embedding model and "random noise" for another.
Embedding anisotropy is the cause: most models pack all vectors into a narrow cone, so even unrelated texts score high. How narrow depends on the model.
Contrastive training only teaches ranking, not absolute scores. Adding a constant to every similarity doesn't change the loss, so nothing pins 0.8 to "relevant."
Fix: measure the baseline similarity of unrelated pairs in your corpus, calibrate the threshold on a small labeled set, and recalibrate on every model change.
Better still, gate on relative signals (score vs. baseline percentile, top-1 margin) or a reranker, not a raw cosine number.

What happened when I switched embedding models?

The rank order survived the migration, but the score scale collapsed. Every relevant match dropped well below my 0.8 threshold, so the gate rejected all of them.

Here's what my logs showed, roughly, on the same internal docs corpus:

	Old model (ada-002)	New model (text-embedding-3-small)
Top-1 score, questions with a real answer	~0.84 median	~0.55 median
Score between two random unrelated chunks	~0.72 median	~0.18 median

Look at the right column first. The new model was better at separating relevant from unrelated. The gap between "real answer" and "noise" got wider. My threshold just sat above the whole distribution.

Now look at the left column. With the old model, random junk scored 0.72 and real answers scored 0.84. My "safety" threshold of 0.8 was sitting in a 0.12-wide band between garbage and gold. I pulled an off-topic test question about pizza in Naples and it scored 0.77 against a Kubernetes runbook. My guardrail was holding by three hundredths of a point.

So the threshold was never a guardrail. It was a coincidence that happened to work for one model.

Why is cosine similarity between unrelated texts so high?

Because most embedding models are anisotropic: their vectors don't spread over the whole sphere, they crowd into a narrow cone. Any two vectors inside a narrow cone have a small angle between them, so cosine similarity is high no matter what the texts say.

This isn't a quirk of one vendor. It showed up in research on BERT and GPT-2 representations years ago (Ethayarajh, 2019, found contextual embeddings occupy a narrow cone), and "representation degeneration" has been studied in language models since.

You can see why with ten seconds of math. Model every embedding as a shared component plus a unique one:

v = m + r
|m|^2 = a^2   (direction every text shares)
|r|^2 = b^2   (what makes this text unique)

If the unique parts are roughly orthogonal to each other and to m, the cosine of two unrelated texts is:

cos(v1, v2) ≈ a^2 / (a^2 + b^2)

If the shared direction carries 70% of each vector's squared length, two texts with nothing in common score about 0.7. All the actual meaning has to fit in the remaining 0.3 of the range. That's exactly what my ada-002 numbers looked like.

Different models end up with different a^2 / (a^2 + b^2). That ratio is your noise floor, and it's not printed on the box.

Why does the score scale change between embedding models?

Because the training objective never asks for a specific score. Most embedding models are trained with a contrastive loss (InfoNCE or a close cousin) that only cares whether the right pair beats the wrong pairs.

The loss for one query looks like:

loss = -log( exp(s_pos / τ) / Σ_i exp(s_i / τ) )

Add a constant c to every similarity s_i and the softmax doesn't change. The exp(c / τ) factor cancels top and bottom. So the model is free to put "unrelated" at 0.1 or 0.7. Nothing in training cares.

The temperature τ and the training data decide how spread out the scores end up. Different model, different recipe, different scale. Your 0.8 doesn't survive the trip.

One more trap: if you shorten embeddings yourself (slicing a Matryoshka-style vector down to fewer dimensions), the vector is no longer unit length. Renormalize after slicing, or your "cosine" is really a dot product with drifting norms.

Is a cosine similarity threshold ever safe to use?

Yes, if it was measured on your corpus, with your exact model, and you re-measure it whenever either changes. A threshold copied from a blog post, a tutorial default, or a previous model is a guess.

The ranking from vector search is usually trustworthy. The absolute number next to it is not, until you calibrate it.

How do I calibrate a cosine similarity threshold?

Two steps: measure the noise floor, then fit the threshold on labeled examples.

Step 1: the noise floor. Sample random pairs of chunks from your own index and look at the distribution. This takes a minute and tells you where "unrelated" lives for this model:

import numpy as np

def noise_floor(E, n=5000, seed=0):
    """E: (N, d) array of unit-normalized chunk embeddings."""
    rng = np.random.default_rng(seed)
    i = rng.integers(0, len(E), n)
    j = rng.integers(0, len(E), n)
    keep = i != j
    sims = np.einsum("nd,nd->n", E[i[keep]], E[j[keep]])
    return np.percentile(sims, [5, 50, 95, 99])

Random pairs from the same corpus overestimate "unrelated" a bit (chunks share topic and jargon), which is fine. You want a conservative floor.

Step 2: a labeled set. Write 40-50 questions your docs answer and 40-50 they don't. Record the top-1 score for each. Then pick the threshold that hits the precision you need:

def pick_threshold(pos_scores, neg_scores, min_precision=0.95):
    candidates = sorted(set(pos_scores) | set(neg_scores))
    for t in candidates:
        tp = sum(s >= t for s in pos_scores)
        fp = sum(s >= t for s in neg_scores)
        if tp and tp / (tp + fp) >= min_precision:
            return t, tp / len(pos_scores)  # threshold, recall
    return None, 0.0

If no threshold gets you there with decent recall, the problem isn't the threshold. Your embeddings can't separate those questions, and you need a reranker or better chunks.

Store the model name and version next to the threshold in config. When someone bumps the model, the stale threshold should fail loudly, not silently.

Does mean-centering embeddings fix anisotropy?

It helps a lot for models with a strong shared direction. Subtract the corpus mean vector, renormalize, and the cone opens up. This is the idea behind "all-but-the-top" post-processing (Mu and Viswanath, 2018), which also removes a few dominant principal components.

mu = E.mean(axis=0)

def center(x):
    x = x - mu
    return x / np.linalg.norm(x, axis=-1, keepdims=True)

E_c = center(E)
q_c = center(q)  # queries get the SAME mu, always

The catches are operational, not mathematical:

The query must be centered with the exact same mu as the corpus.
mu drifts as the corpus grows. Recompute it on a schedule and re-embed or re-center the index.
Your vector index (pgvector, FAISS, whatever) has to store centered vectors, so this is a reindex, not a flag.

Centering makes scores easier to read. It doesn't make them calibrated. You still run step 2.

What should I gate on instead of a raw score?

Gate on signals that survive a model swap. Three that have worked for me:

Score vs. noise floor. Accept the top hit only if it beats the 99th percentile of random-pair similarity. When the model changes, recompute the floor and the gate moves with it.
Top-1 margin. When the query matches something real, the top hit usually stands out from the next few. When it matches nothing, scores come out flat. top1 - median(top2..top10) is a cheap, scale-relative signal.
A cross-encoder reranker for the final relevance call. It reads query and chunk together, so it judges relevance directly instead of comparing two independent vectors. Its scores need calibration too (many output raw logits), but it separates relevant from irrelevant much more cleanly.

The checklist I run now on every embedding change

Re-embed a sample, plot the random-pair distribution. Write down p50 and p99.
Re-score the labeled set. Re-pick the threshold. Commit it with the model name.
Check vector norms. If they aren't ~1.0, normalize or switch to dot product on purpose.
Diff top-1 results old vs. new on the labeled set. Rank changes are the real migration risk.
Alert when "I don't know" rate jumps. That metric would have caught my outage in an hour instead of a day.

So, does a cosine similarity of 0.8 mean anything?

On its own, no. A cosine similarity threshold like 0.8 only means something relative to one specific embedding model on one specific corpus, because embedding anisotropy and contrastive training leave each model with its own score scale and its own noise floor. Unrelated texts can score 0.7 in one model and 0.15 in another. Trust the ranking that vector search gives you. Treat the number as uncalibrated until you've measured random-pair similarity in your own index and picked the threshold from labeled examples. Then redo both every time the model changes.

Written by the developer behind Preterview, an interview prep platform.

Docker Compose depends_on Won't Wait for Postgres: 11/50 Runs Failed

jidonglab — Sun, 11 Oct 2026 04:53:41 +0000

My integration tests failed on a Tuesday with ECONNREFUSED 127.0.0.1:5432. I hit re-run and they passed. Then they failed again on Thursday. I had depends_on: [db] right there in my docker-compose.yml, so I assumed the database was up before my app started. That assumption was wrong, and the way Docker Compose depends_on actually works explained every flaky run I'd been blaming on "CI being CI."

So I stopped guessing. I wrote a loop that tore everything down with a fresh volume and booted the stack 50 times. 11 of 50 runs failed on the first database connection. That's 22%, which is far too high to call bad luck.

Then I fixed it the way every Stack Overflow answer says to, and it still failed. Twice. That second bug is the part almost nobody writes about.

TL;DR

depends_on: [db] (short form) only waits for the db container to start. It does not wait for Postgres to accept connections.
To wait for readiness, use the long form with condition: service_healthy and define a healthcheck on the database service.
The default healthcheck interval is 30 seconds, so a naive healthcheck can make your stack boot slowly. Set interval: 2s.
On a fresh volume, the official Postgres image runs a temporary init server that listens only on a Unix socket. Plain pg_isready can report "ready" during that phase. Use pg_isready -h 127.0.0.1 to force a TCP check.
Keep retry logic in your app anyway. Healthchecks gate startup only, not the rest of the container's life.

What does Docker Compose depends_on actually wait for?

Docker Compose depends_on, in its short form, waits for the dependency container to be created and started. That's it. "Started" means the process inside the container has launched, not that it's listening on a port or finished initializing.

Here's the setup I had:

services:
  db:
    image: postgres:16
    environment:
      POSTGRES_USER: app
      POSTGRES_PASSWORD: app
      POSTGRES_DB: app
  tests:
    build: .
    command: npm test
    depends_on:
      - db

Compose starts db, sees the container running, and immediately starts tests. Postgres at that moment might be anywhere: still running initdb, still executing your init scripts, or restarting after initialization. My test runner opened a connection within about a second of boot. Sometimes Postgres was ready. Usually it was. 22% of the time it wasn't.

The reason it "usually works" is what makes this bug so annoying. On your laptop, the volume already exists, Postgres skips initialization, and it's ready in well under a second. In CI, every run gets a fresh volume, so every run pays the full init cost. The race only loses where you can't watch it.

How do I reproduce the depends_on race?

Run the stack in a loop with a fresh volume each time. This is the script I used:

fails=0
for i in $(seq 1 50); do
  docker compose down -v --remove-orphans >/dev/null 2>&1
  if ! docker compose up --abort-on-container-exit --exit-code-from tests >/dev/null 2>&1; then
    fails=$((fails+1))
  fi
done
echo "failed: $fails / 50"

The -v matters. Without it, the named volume survives, Postgres skips init, and you'll convince yourself the bug doesn't exist. --exit-code-from tests makes docker compose up return the test container's exit code, so the loop can count failures.

How do I make Docker Compose wait until Postgres is ready?

Use the long form of depends_on with condition: service_healthy, and give the database a healthcheck. Compose will hold the dependent service until the healthcheck passes.

services:
  db:
    image: postgres:16
    environment:
      POSTGRES_USER: app
      POSTGRES_PASSWORD: app
      POSTGRES_DB: app
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
      interval: 2s
      timeout: 3s
      retries: 30
  tests:
    build: .
    command: npm test
    depends_on:
      db:
        condition: service_healthy

Three details here bite people:

$$ is not a typo. Compose interpolates ${VAR} from your host environment when it parses the file. $$ escapes it so the literal ${POSTGRES_USER} reaches the container shell, where the variable actually exists.
CMD-SHELL, not CMD. CMD runs the binary directly with no shell, so ${POSTGRES_USER} would never expand.
Set interval yourself. The Docker default is 30 seconds. With the default, your tests can sit idle for half a minute waiting for the first check, even though Postgres was ready after three seconds. People "fix" the race this way and then complain their CI got slower.

I reran the 50-boot loop. Failures dropped from 11 to 2.

Two. Not zero.

Why does pg_isready say ready before Postgres accepts connections?

Because on a fresh volume, the official Postgres image starts Postgres twice, and the first one is lying to you, technically.

Here's the sequence the image's entrypoint runs on first boot:

Run initdb to create the data directory.
Start a temporary server with TCP disabled (it listens on the Unix socket only).
Create your user and database, then run everything in /docker-entrypoint-initdb.d/.
Stop the temporary server.
Start the real server, listening on TCP.

pg_isready with no -h flag connects over the local Unix socket. During step 3, the temporary server is accepting socket connections, so pg_isready exits 0. Docker marks the container healthy. Compose releases your tests. Your tests connect over TCP to db:5432, where nothing is listening yet, because the real server hasn't started. ECONNREFUSED.

The window is small with an empty init directory. If you have a seed script with a few thousand rows in docker-entrypoint-initdb.d, the window grows with it, and so does your failure rate.

The fix is one flag. Force the healthcheck to use TCP, the same path your app uses:

    healthcheck:
      test: ["CMD-SHELL", "pg_isready -h 127.0.0.1 -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
      interval: 2s
      timeout: 3s
      retries: 30
      start_period: 10s

The temporary server doesn't listen on TCP, so this check can't pass until the real server is up. start_period gives init some grace time: failed checks during that period don't count against retries.

Loop again: 0 of 50 failed. I ran it another 50 to make sure I wasn't celebrating noise. Still zero.

The general lesson: a healthcheck should test the same path your client uses. Socket vs TCP, localhost vs service name, the right database name. A healthcheck that checks something adjacent will eventually pass at the wrong moment.

What about migrations that must finish before the app starts?

Use condition: service_completed_successfully. It waits until the dependency container exits with code 0, which is exactly what a one-shot migration job does.

  migrate:
    build: .
    command: npm run migrate
    depends_on:
      db:
        condition: service_healthy
  app:
    build: .
    depends_on:
      db:
        condition: service_healthy
      migrate:
        condition: service_completed_successfully

If migrate exits non-zero, Compose refuses to start app. That's the behavior you want: a failed migration should stop the deploy, not let the app boot against a half-migrated schema.

There are three conditions in total, and it helps to know all of them:

Condition	Waits until
`service_started`	Container is running (same as short form)
`service_healthy`	Healthcheck reports healthy
`service_completed_successfully`	Container exited with code 0

Is a healthcheck enough, or do I still need retries?

You still need retries in the app. Healthchecks only gate startup order inside Compose. They do nothing when Postgres restarts an hour later, when a network blip drops a connection, or when you deploy the same image to an orchestrator that has no depends_on at all.

My rule now: Compose healthchecks make local and CI boots deterministic. App-level connection retry with backoff makes production survivable. They solve different problems, and I stopped treating either as a replacement for the other.

One more CI tip: docker compose up -d --wait blocks until services are running and healthy, then returns. That's cleaner than a hand-rolled until pg_isready; do sleep 1; done loop in your workflow file.

The checklist I paste into every Compose file now

Long-form depends_on with an explicit condition, never the short list.
A healthcheck on every service someone depends on.
interval: 2s so readiness is noticed fast.
Healthcheck uses the same transport as the client (-h 127.0.0.1 for Postgres).
$$ for container-side variables, CMD-SHELL when you need a shell.
service_completed_successfully for migrations and seed jobs.
Retry logic in the app regardless.
Test with down -v, because a warm volume hides everything.

So does Docker Compose depends_on wait for the database?

No. Docker Compose depends_on, in its short form, only waits for the database container to start, not for Postgres to accept connections, which is why stacks with fresh volumes fail intermittently in CI. To wait for readiness, use depends_on with condition: service_healthy and add a healthcheck to the database service. For the official Postgres image, run pg_isready -h 127.0.0.1 so the check goes over TCP. Without -h, it can pass against the temporary socket-only server the image runs during first-boot initialization. In my case that one flag took a 50-run loop from 2 failures to 0.

Written by the developer behind Preterview, an interview prep platform.

Ollama num_ctx Truncated 287 of 400 Prompts and Never Told Me

jidonglab — Sat, 10 Oct 2026 16:52:09 +0000

My local RAG bot answered 46% of my test questions correctly. Llama 3.1 8B on Ollama, a 128K context window on the model card, retrieved chunks that I had checked by hand. The right paragraph was in the prompt every single time.

The model just never saw it. Ollama's num_ctx was set to 2048 tokens, and Ollama quietly chopped the front off every prompt longer than that. No error, no field in the response, no warning in my Python logs. 287 of my 400 prompts got cut.

This is the autopsy.

TL;DR

Ollama num_ctx is the context window Ollama actually allocates, not the one on the model card. In the version I ran, the default was 2048 tokens even for a 128K model.
Overflowing prompts are truncated silently from the front. The tail survives, so the question stays and the instructions and top-ranked chunks vanish.
Spot it with prompt_eval_count in the response. If it sits at or just under your num_ctx no matter how long your prompt is, you're being truncated.
Fix it with options: {"num_ctx": 8192} on the native API, or PARAMETER num_ctx 8192 in a Modelfile. The OpenAI-compatible endpoint ignored my per-request setting.
After the fix: accuracy went from 46% to 81% on the same 400 questions. Cost: about 1 GB more VRAM and a slower median response.

What was I building?

A notes assistant. Around 3,000 markdown files from five years of work logs, chunked at roughly 400 tokens, embedded, top 10 chunks retrieved per question. Everything runs locally on a 12 GB GPU because I don't want my work notes leaving the machine.

The prompt went to /api/generate as one string, in this order:

Instructions ("answer only from the context, reply in JSON with answer and source")
The 10 retrieved chunks, best match first
The question

That ordering is the textbook layout. It's also exactly the wrong layout for what Ollama was doing to me.

Why did the model ignore my instructions?

It ignored them because it never received them. When a prompt exceeds num_ctx, Ollama keeps the end of the input and discards tokens from the beginning to make it fit. My instructions lived at the beginning. So did chunk #1, the best match.

The symptoms looked like a dumb model, not a broken pipeline:

Answers came back as plain prose instead of JSON, about a third of the time.
The source field, when present, pointed at low-ranked chunks.
Questions with obvious answers got confident, wrong replies built from chunks 7 through 10.

I spent an evening rewriting the system prompt. I added "IMPORTANT" in caps. I moved the JSON instruction to two places. Nothing changed, which in hindsight was the clue: the edits were landing in a region the model never read.

How do you detect Ollama prompt truncation?

Check prompt_eval_count in the response. It's the number of prompt tokens the model actually processed. If it's pinned at your context limit while your input keeps growing, Ollama is truncating.

I logged it next to my own token estimate for every request:

import requests

def ask(prompt: str, num_ctx: int | None = None) -> dict:
    body = {"model": "llama3.1:8b", "prompt": prompt, "stream": False}
    if num_ctx:
        body["options"] = {"num_ctx": num_ctx}
    r = requests.post("http://localhost:11434/api/generate", json=body, timeout=120)
    r.raise_for_status()
    return r.json()

resp = ask(prompt)
print(len(prompt) // 4, resp["prompt_eval_count"])

The output was embarrassing in its regularity:

Every long prompt was capped at the same number. The short ones passed through intact. The server log had the confession all along, a line containing truncating input prompt with the limit and the original length. I just never read the server log, because the API returned HTTP 200 with a perfectly normal-looking body.

How many of my prompts were affected?

287 of 400, or 72%. With 10 chunks of about 400 tokens plus instructions, most prompts landed between 4,000 and 6,000 tokens. Only the questions whose retrieval returned short chunks fit under 2048.

Splitting the eval by that line made the problem obvious:

Prompt length	Questions	Correct
Under 2048 tokens	113	96 (85%)
Over 2048 tokens	287	88 (31%)
Total	400	184 (46%)

The model wasn't bad at my task. The model was great at my task whenever it could see the task.

How do you set num_ctx in Ollama?

Pass it per request in options on the native API, or bake it into a model with a Modelfile. Both worked for me. One thing that did not work is the obvious one.

Option 1: per request on the native API

resp = ask(prompt, num_ctx=8192)

Option 2: a Modelfile

FROM llama3.1:8b
PARAMETER num_ctx 8192

ollama create notes-llama -f Modelfile

Then call notes-llama instead of llama3.1:8b. This is the version I kept, because it makes the setting impossible to forget in some other script.

What failed: the OpenAI-compatible endpoint

My first fix was through /v1/chat/completions with the OpenAI Python client, sending num_ctx via extra_body. prompt_eval_count (exposed as usage.prompt_tokens) stayed at 2047. That endpoint didn't honor the option in my setup. If you use Ollama behind an OpenAI-style client, set the context in a Modelfile or at the server level. Newer Ollama releases added an OLLAMA_CONTEXT_LENGTH environment variable for the server-wide default, and raised the default itself, so check what your version does instead of trusting either my number or the model card.

What did the bigger context window cost?

About 1 GB of VRAM and roughly 1.5 seconds of median latency. The KV cache scales linearly with num_ctx, so a 4x window costs 4x the cache.

For Llama 3.1 8B at fp16 the math is short: 32 layers × 8 KV heads × 128 dims × 2 (K and V) × 2 bytes = 128 KB per token.

2048 tokens: 256 MB of KV cache
8192 tokens: 1 GB of KV cache

ollama ps moved from about 5.9 GB to 6.8 GB on my machine, which still fits the 12 GB card with room for the embedding model. Median end-to-end time per question went from 1.9 s to 3.4 s, because the model now actually evaluates 5,000 prompt tokens instead of 2,000. That's not a regression. That's the price of reading the whole prompt, which I thought I was already paying.

What were the results after the fix?

81% correct, 324 of 400, same questions, same retrieval, same prompt text. JSON format compliance went from about two thirds to every response but 3. The source field started pointing at chunk #1 or #2 most of the time, which is what you'd hope for when your retriever is any good.

Not everything was fixed. The remaining 76 misses were real retrieval failures and a few questions my notes genuinely can't answer. Those are honest problems. The 140 extra correct answers were never a model problem.

What should you change in your own Ollama setup?

Three habits, all cheap:

Assert on prompt_eval_count. If it's within a few tokens of num_ctx, raise or at least log loudly. One if statement would have saved me an evening of prompt engineering against a wall.
Put the question and the critical instructions at the end of single-string prompts. If something gets cut, it should be the lowest-ranked chunk, not your output format.
Set num_ctx explicitly everywhere. Model card context length is a ceiling, not a default. Pick a number your VRAM can carry and write it down in a Modelfile.

def ask_checked(prompt: str, num_ctx: int = 8192) -> dict:
    resp = ask(prompt, num_ctx=num_ctx)
    if resp["prompt_eval_count"] >= num_ctx - 8:
        raise RuntimeError(f"prompt truncated at {resp['prompt_eval_count']} tokens")
    return resp

So why did Ollama truncate my prompts without telling me?

Ollama truncated 287 of my 400 prompts because num_ctx, the context window it actually allocates, defaulted to 2048 tokens in my version, regardless of the model's advertised 128K. When a prompt exceeds num_ctx, Ollama drops tokens from the front, keeps the tail, and returns a normal HTTP 200; the only traces are a server log line and a prompt_eval_count stuck near the limit. Setting num_ctx to 8192 via request options or a Modelfile PARAMETER (not the OpenAI-compatible endpoint, which ignored it for me) raised my RAG accuracy from 46% to 81% for about 1 GB of extra VRAM.

Written by the developer behind Preterview, an interview prep platform.

bge-reranker Truncates at 512 Tokens: 31% of My Chunks Got Cut

jidonglab — Sat, 10 Oct 2026 04:50:26 +0000

My RAG pipeline had a question it kept getting wrong. "How do I fix the replication lag alert on the orders DB?" The answer was in my runbooks. Retrieval found the right chunk at rank 3. Then the reranker moved it to rank 14, my top-5 cutoff threw it away, and the LLM confidently answered from a chunk about a different database.

The reranker wasn't dumb. It was blind. My bge-reranker truncates at 512 tokens, and the fix commands lived at the bottom of a chunk it never finished reading. When I counted, 31% of my chunks were longer than what the reranker could see.

This post is about that one mechanism: why a cross-encoder reranker truncates at 512 tokens, which part of your text it throws away, and what to do about it.

TL;DR

Cross-encoder rerankers like BAAI/bge-reranker-base read the query and passage as one sequence capped at 512 tokens, including 4 special tokens.
When the pair is too long, the Hugging Face tokenizer uses longest_first truncation, which trims the end of the passage. No error, no warning.
Your chunker, your embedding model and your reranker usually measure length with three different rulers. A chunk that fits one can overflow another.
Detect it by tokenizing (query, passage) with the reranker's own tokenizer and logging how often the length exceeds max_length.
Fix it by chunking to the reranker's budget, or by scoring overlapping windows of the passage and taking the max.

How did my reranker bury the right chunk?

Here's the setup, because the details are where this bug hides:

Corpus: about 2,400 chunks from internal runbooks and incident notes.
Chunker: LangChain's RecursiveCharacterTextSplitter, chunk_size=2000. Characters, not tokens.
Retrieval: text-embedding-3-small, top 20 by cosine similarity.
Reranking: CrossEncoder("BAAI/bge-reranker-base") from sentence-transformers, keep top 5.

I had 60 hand-labeled questions. Retrieval put the correct chunk somewhere in the top 20 for 54 of them. After reranking, the correct chunk made the top 5 for only 41.

So the reranker was losing answers that retrieval had already found. That's backwards. The whole point of a reranker is to be the smarter, slower second pass.

I went through the 13 losses by hand. In 9 of them, the sentence that actually answered the question sat in the last third of the chunk. Runbooks are written that way: title, symptoms, context, and then, at the very bottom, the ## Resolution section with the commands you need.

Why does a cross-encoder reranker truncate at 512 tokens?

A cross-encoder reranker truncates at 512 tokens because it is a BERT-style encoder with a fixed number of position embeddings, and it reads the query and passage together as a single input. bge-reranker-base is built on XLM-RoBERTa, which was trained with 512 positions. Anything past that has no position to sit in, so the tokenizer cuts it before the model runs.

This is different from an embedding model (a bi-encoder). A bi-encoder embeds the query and the passage separately. A cross-encoder concatenates them:

<s> query tokens </s></s> passage tokens </s>

That's 4 special tokens for XLM-RoBERTa pairs. So your real passage budget is:

512 - 4 - len(query_tokens)

A 30-token question leaves 478 tokens for the passage. That joint attention over query and passage is exactly why cross-encoders rank better than cosine similarity. It's also why they have this hard ceiling.

Which part of the passage gets cut?

The end of the passage gets cut. sentence-transformers' CrossEncoder calls the tokenizer with truncation=True, which in Hugging Face means the longest_first strategy. It removes tokens one at a time from whichever sequence is currently longer. Your query is 30 tokens and your passage is 700, so the passage loses its tail every single time.

You can watch it happen:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("BAAI/bge-reranker-base")

enc = tok(query, passage, truncation=True, max_length=512)
seen = tok.decode(enc["input_ids"], skip_special_tokens=True)

print(seen[-300:])  # the last thing the reranker actually read

When I ran this on my replication-lag chunk, the last thing the reranker read was the middle of a paragraph explaining what replication lag is. The pg_stat_replication query and the fix were gone. The model scored "a chunk that explains the concept" against "a short chunk about a different DB that mentions the alert name in full," and the short chunk won. That's a reasonable judgment on the text it was given.

The silent part is what makes this nasty. With truncation=True, the tokenizer does not warn you. Your scores look like normal floats. Nothing in your logs says "I only read 60% of this."

Why didn't retrieval catch the problem?

Retrieval didn't catch it because every stage measured the chunk with a different ruler:

Stage	Unit	Limit
Chunker	characters	2,000
`text-embedding-3-small`	OpenAI tokens	8,191
`bge-reranker-base`	XLM-RoBERTa tokens, query included	512

The embedding model saw the whole chunk, including the Resolution section, and ranked it well. The reranker saw a truncated version of the same chunk and ranked it badly. The stage with the smallest window had the final say.

The character limit made it worse. 2,000 characters of English prose fits under 512 tokens comfortably. 2,000 characters of runbook does not. Shell commands, hostnames, UUIDs, stack traces and YAML break into many more subword tokens than ordinary words do. My chunks that overflowed were almost all the operational ones, which are exactly the ones people ask about.

How do you check if your reranker is truncating?

Tokenize every (query, chunk) pair with the reranker's tokenizer, without truncation, and count how many exceed the limit. It takes a few minutes on a few thousand chunks.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("BAAI/bge-reranker-base")
MAX_LEN = 512
TYPICAL_QUERY = "how do I fix the replication lag alert on the orders db"

def overflow(passage: str) -> int:
    ids = tok(TYPICAL_QUERY, passage, truncation=False)["input_ids"]
    return len(ids) - MAX_LEN

over = [c for c in chunks if overflow(c) > 0]
print(f"{len(over)}/{len(chunks)} chunks truncated "
      f"({len(over) / len(chunks):.0%})")

Mine printed 31%. Two more checks worth adding:

Log it in production. Record the untruncated pair length for every reranked candidate. A truncation rate is a metric you can alert on.
Check the max_length you actually pass. Wrappers and frameworks set their own defaults. If you swap in a reranker trained on longer inputs but your wrapper still passes 512, you're paying for a model you're not using.

How do you fix reranker truncation?

There are three fixes, and I ended up using the first two together.

1. Chunk to the reranker's budget, measured with the reranker's tokenizer. Reserve room for the query and the special tokens, then cap passages at what's left.

QUERY_RESERVE = 64      # longest query you expect, in reranker tokens
PASSAGE_BUDGET = 512 - 4 - QUERY_RESERVE   # 444

def token_len(text: str) -> int:
    return len(tok(text, add_special_tokens=False)["input_ids"])

splitter = RecursiveCharacterTextSplitter(
    chunk_size=PASSAGE_BUDGET,
    chunk_overlap=60,
    length_function=token_len,
)

The one-line change that matters is length_function=token_len. Now the chunker and the reranker use the same ruler.

2. Score overlapping windows and take the max. Some documents shouldn't be split, or you can't re-index right now. Slide a window over the passage, score each window, and keep the highest. In IR research this is known as MaxP scoring.

def windowed_score(model, query, passage, window=400, stride=200):
    ids = tok(passage, add_special_tokens=False)["input_ids"]
    if len(ids) <= window:
        return float(model.predict([(query, passage)])[0])

    starts = list(range(0, len(ids) - window + 1, stride))
    if starts[-1] + window < len(ids):
        starts.append(len(ids) - window)   # always cover the tail

    windows = [tok.decode(ids[s:s + window]) for s in starts]
    return float(max(model.predict([(query, w) for w in windows])))

Watch the starts.append line. Without it, a passage whose length isn't a neat multiple of the stride loses its last few dozen tokens. That would be the same bug you're fixing, just smaller.

3. Use a reranker trained on longer inputs. This works, but read the cost. Self-attention scales with the square of sequence length, and you run the reranker once per candidate. Going from 512 to 2,048 tokens on 20 candidates per query is a real latency change, so measure it on your own hardware before shipping.

After re-chunking with the reranker's tokenizer and using windowed scoring for the few long documents I kept whole, the correct chunk made the top 5 for 51 of my 60 questions, up from 41. Same embedding model, same reranker, same LLM. The only difference was that the reranker could now read the whole chunk.

What should you take away from this?

Every model in a RAG pipeline has a context window, and the smallest one controls quality. People check the LLM's window and maybe the embedding model's. Almost nobody checks the reranker's, because it returns a clean float either way.

If your documents put the important part at the end, like runbooks, API docs with examples at the bottom, or support tickets with the resolution last, you're in the worst case for longest_first truncation.

So does bge-reranker really truncate at 512 tokens?

Yes. bge-reranker-base and similar BERT- and XLM-RoBERTa-based cross-encoders read the query and passage as one sequence capped at 512 tokens, including special tokens. sentence-transformers truncates with longest_first, which silently drops the end of the passage. If your chunks are sized in characters, or in another model's tokens, a meaningful share of them may be partly invisible to your reranker, and any answer in that tail can't help the score. Measure overflow with the reranker's own tokenizer, size chunks to its budget, and use overlapping-window max scoring for passages you can't split.

Written by the developer behind Preterview, an interview prep platform.

LLM-as-a-Judge: Claude Gave 68% of Answers a 7 or 8 Out of 10

jidonglab — Fri, 09 Oct 2026 16:48:15 +0000

A user emailed me a screenshot of two of their reports side by side. One session they'd nailed. One they'd bombed so badly they quit halfway. Both said "7/10."

That's when I learned my LLM-as-a-judge wasn't judging anything. It was printing a seven with extra steps.

I pulled every score the system had ever produced: 1,140 scored answers. 41% were a 7. 27% were an 8. Anything below 5 showed up 3% of the time. My carefully written grading prompt had become a very expensive random number generator with a strong opinion about the number 7.

Here's what I measured, the four fixes I tried, and the one that actually worked.

TL;DR

LLM-as-a-judge on an unanchored 1-10 scale collapses into 7s and 8s. Mine put 68% of 1,140 answers there.
Telling the model to "be strict" doesn't fix it. It moves the whole pile down one point. Ranking quality barely changed (Spearman 0.41 → 0.44).
A 1-5 scale with a written description for every level jumped agreement with my hand grades to 0.68.
Binary checklist items, summed in code, hit 0.79, close to the 0.83 agreement between two human graders. Rerun noise dropped from 0.9 to 0.31 points.
The cost: scoring got ~75% more expensive per session, and the checklist still mis-scores short, unusual answers.

What was I actually scoring?

The system is Preterview, the interview practice platform I run (full disclosure: I built it). It runs a voice interview, transcribes each answer, and an LLM scores it on a few dimensions before writing a report. If the scores are mush, the report is mush, and the whole product is a chatbot that says "nice job."

The original judge prompt was what most people write the first time:

Score the candidate's answer from 1 to 10 on:
- clarity
- specificity
- relevance
Return JSON: {"clarity": int, "specificity": int, "relevance": int, "feedback": str}

Clean. Readable. Useless.

Why does LLM-as-a-judge cluster scores at 7 out of 10?

Because a number with no definition defaults to the model's prior for "fine," and that prior is about 7. Without anchors, "6" and "8" don't mean anything specific, so the model reaches for the safe middle-high answer that sounds supportive without sounding fake.

A few things stack up:

No anchors. What's the difference between a 6 and a 7 for "specificity"? I never said. The model can't either.
Leniency. Instruction-tuned models are trained to be helpful and pleasant. Handing a struggling person a 3 feels rude. A 7 never offends anyone.
Too many levels. Ten buckets is more resolution than either of us can use consistently. The extra buckets just become noise around the mode.

Here's what my production distribution looked like before any fixes:

How do you test whether an LLM judge is accurate?

Build a small hand-graded calibration set and measure rank agreement against it. You don't need thousands of examples. You need a set where you already know the right ordering.

I took 60 real answers (with permission, anonymized) and graded them myself on a 1-5 scale, deliberately picking 12 per level so the set was spread out. A friend who has run a lot of hiring loops graded them blind. Our agreement (Spearman ρ) was 0.83. That's the ceiling. No judge prompt should be expected to beat two humans agreeing with each other.

Then I ran every prompt variant over all 60 answers, five times each, and tracked three numbers:

Spearman ρ against my grades (does it rank answers correctly?)
Share of scores at the mode (is it collapsing?)
Rerun standard deviation (does the same answer get the same score?)

The baseline prompt on the calibration set: ρ = 0.41, and 71% of answers got a 7 or 8 even though I'd picked the set to be evenly spread. Rerun std dev: 0.9 points. One answer scored 5, 7, 8, 7, 6 across five runs. Same text, same prompt.

Fix 1: "Be strict." Did it work?

No. It shifted every score down about one point and left the ranking almost unchanged. ρ went from 0.41 to 0.44, which is inside the noise.

I added "You are a demanding senior interviewer. Most answers deserve below 6." The mode moved from 7 to 6. That's it. The pile slid left as one block. Bad answers and good answers were still crammed together, just at a lower altitude.

This is the most common fix I see recommended and it's basically a cosmetic filter. You get scores that look harsher, which feels like rigor. It isn't.

Fix 2: A 1-5 scale with anchored levels

This was the first real improvement: ρ jumped to 0.68. I cut the scale to five levels and wrote a concrete, observable description for each one.

specificity:
1 = No concrete example. Speaks in generalities ("I'm a team player").
2 = Mentions a situation but no actions or outcome.
3 = Situation and actions, but the outcome is vague ("it went well").
4 = Situation, own actions, and a concrete outcome.
5 = All of 4, plus a measurable result and what they'd do differently.

The trick is that each level describes something the model can check in the transcript, not a vibe like "good" or "excellent." A 3 now means a specific thing is missing.

Fix 3: Anchors plus reference answers

Adding two graded example answers per dimension pushed ρ to 0.74. I included one real answer I'd graded a 2 and one I'd graded a 4, with a one-line explanation each.

The downside: the prompt grew from about 1,100 tokens to about 3,400. Since that block is identical on every call, it caches well, so the real cost increase was small. The bigger issue was that the model started pattern-matching to the examples: answers that looked like my "4" example got 4s even when the content was weaker.

Fix 4: Binary checklists, scored in code

The winner: ρ = 0.79, rerun std dev 0.31, and no pile-up at any single level. Instead of asking for a number, I asked the model a set of yes/no questions and computed the score myself.

from pydantic import BaseModel

class Check(BaseModel):
    passed: bool
    evidence: str  # exact quote from the transcript, or "" if none

class SpecificityChecks(BaseModel):
    names_a_specific_situation: Check
    describes_own_actions: Check     # "I", not "we"
    states_an_outcome: Check
    outcome_is_measurable: Check
    includes_a_reflection: Check

WEIGHTS = {
    "names_a_specific_situation": 1,
    "describes_own_actions": 1,
    "states_an_outcome": 1,
    "outcome_is_measurable": 1,
    "includes_a_reflection": 1,
}

def score(checks: SpecificityChecks) -> int:
    total = sum(
        WEIGHTS[name] for name, c in checks
        if c.passed and c.evidence.strip()
    )
    return max(1, total)  # 0-5 checks → 1-5

Two details mattered more than I expected:

Require evidence. A check only counts if the model quotes the transcript. Without that rule, the model happily marked "states an outcome" as true for answers that didn't have one. Forcing a quote killed most of the generous passes.
Yes/no is easier than 1-10. "Did they state a measurable result?" has a right answer. "How specific was this, out of 10?" doesn't. Models are far more consistent on the first kind of question, which is exactly why rerun noise dropped by two thirds.

What did the checklist approach break?

Short, unusual, genuinely strong answers. On the calibration set, 5 of 60 answers came out 2 or more levels off my grade, and 4 of those 5 were concise answers from strong candidates.

One answer was two sentences: a precise problem, a precise fix, a number. I graded it a 5. The checklist gave it a 3 because there was no "reflection" and the actions were implied rather than spelled out with "I did X." A checklist rewards structure, and structure correlates with length. I haven't fully solved this. My current patch is a separate check for "complete despite being brief," and it helps on about half of those cases.

Other honest costs:

Price. Evidence quotes mean more output tokens. Scoring went from about $0.012 to $0.021 per session.
Users noticed. The average score dropped, and in the first week I got a handful of "this feels harsh" emails. Nobody had ever emailed to say a 7 felt too generous. That asymmetry is the whole problem in one sentence.
Checklists need maintenance. Every new dimension means writing and validating new checks against the calibration set, not just adding a word to a prompt.

So how do you stop an LLM-as-a-judge from giving everything a 7?

Stop asking it for a number. An LLM-as-a-judge on an unanchored 1-10 scale will cluster around 7 because "7" is the model's default for "acceptable," and telling it to be strict only slides the cluster down. Replace the scale with concrete yes/no checks that each require a quoted piece of evidence, compute the score in your own code, and validate the result against a small hand-graded set where you know the right ranking. In my case that took rank agreement with human grades from 0.41 to 0.79, against a human-vs-human ceiling of 0.83, at the cost of higher token spend and a known blind spot for brilliant short answers.

Written by the developer behind Preterview, an interview prep platform.

GitHub Actions GITHUB_TOKEN Won't Trigger Workflows: 0 of 11 Releases

jidonglab — Thu, 08 Oct 2026 04:46:24 +0000

Eleven releases. Eleven green "Create Release" runs. Eleven tags sitting on GitHub. And my npm package was still on the version from three weeks earlier.

The publish workflow had never run once. No failure, no red X, no email. It simply didn't exist in the Actions tab. My release workflow created tags and releases with the default GITHUB_TOKEN, and GITHUB_TOKEN won't trigger workflows. That's a deliberate rule, not a bug, and it took me an embarrassing amount of time to find it.

TL;DR

Events created with the built-in GITHUB_TOKEN (pushes, tags, releases, pull requests, comments) do not start new workflow runs. The two exceptions are workflow_dispatch and repository_dispatch.
GitHub does this on purpose so a workflow can't trigger itself in an infinite loop.
The symptom is silence: no failed run, just a missing one. Bot-opened PRs show required checks stuck on "Expected — Waiting for status to be reported".
Fixes: use a GitHub App token, a fine-grained PAT, an explicit workflow_dispatch call, or merge the two workflows into one job.
If you switch to a token that does trigger workflows, add a loop guard.

What exactly broke?

My setup looked like what half the internet runs:

# release.yml
on:
  push:
    branches: [main]

jobs:
  release:
    runs-on: ubuntu-latest
    permissions:
      contents: write
    steps:
      - uses: actions/checkout@v4
      - run: |
          VERSION=$(node -p "require('./package.json').version")
          git tag "v$VERSION"
          git push origin "v$VERSION"
          gh release create "v$VERSION" --generate-notes
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

# publish.yml
on:
  push:
    tags: ['v*']

jobs:
  publish:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm publish

The tag push is real. You can see the tag in the repo. But publish.yml never fires, because the push was authenticated with GITHUB_TOKEN. Same story if publish.yml listened on release: types: [published] instead. The release exists. The event is swallowed.

The kicker: when I pushed a tag by hand from my laptop to test it, publish ran perfectly. So my "test" proved the workflow was fine, and I went looking for bugs in the wrong file for an hour.

Why won't GITHUB_TOKEN trigger workflows?

GitHub blocks it to prevent recursive runs. If a workflow on push could push a commit with GITHUB_TOKEN and trigger itself, one formatting bot would spin forever and burn everyone's minutes.

So the rule is blunt: anything your workflow does with GITHUB_TOKEN is treated as "the workflow did it," and the workflow system ignores it as a trigger. The only events that still fire are workflow_dispatch and repository_dispatch, because those are explicit "please run this" requests, not side effects.

This applies no matter how the token reaches the API:

git push after actions/checkout (checkout stores GITHUB_TOKEN in the local git config by default)
gh CLI with GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
actions/github-script using the default token
Any marketplace action that defaults to github.token, which is most of them

That last point is the sneaky one. Plenty of release and PR-bot actions take a token input that silently defaults to GITHUB_TOKEN. You never typed it, so you never thought about it.

Which GitHub Actions workflows are affected?

Any workflow whose trigger is an event a bot produced. The common casualties I've hit or watched others hit:

Tag-triggered publish jobs. A release workflow pushes v1.2.3, the on: push: tags workflow never runs. My exact bug.
release: published deploys. A release-automation action creates the release, the deploy listening for it stays idle.
CI on bot-opened pull requests. A workflow opens a PR (version bumps, generated code, dependency syncs) using GITHUB_TOKEN. Your pull_request CI never runs on it. If those checks are required by branch protection, the PR sits forever on "Expected — Waiting for status to be reported" and nobody can merge it without admin override.
Commit-then-test chains. A formatter or codegen job commits back to the branch, and the follow-up test workflow on push doesn't run on the new commit. You end up merging an untested commit that looks tested because the previous commit was green.
Comment-driven bots. A workflow posts an issue comment hoping to kick an issue_comment workflow. Nothing happens.

The pattern: if a human did the same action, it would work. If your workflow did it with the default token, it won't.

How do I make a GitHub Actions push trigger another workflow?

You have four real options. Pick based on who should "own" the action.

Fix 1: Use a GitHub App installation token (my pick)

Create a small GitHub App, install it on the repo with the permissions you need (Contents: write, Pull requests: write), and store its ID and private key. Then mint a short-lived token inside the job:

steps:
  - uses: actions/create-github-app-token@v1
    id: app-token
    with:
      app-id: ${{ vars.RELEASE_APP_ID }}
      private-key: ${{ secrets.RELEASE_APP_PRIVATE_KEY }}

  - uses: actions/checkout@v4
    with:
      token: ${{ steps.app-token.outputs.token }}

  - run: |
      git tag "v$VERSION"
      git push origin "v$VERSION"
      gh release create "v$VERSION" --generate-notes
    env:
      GH_TOKEN: ${{ steps.app-token.outputs.token }}

Passing the token to actions/checkout matters. Checkout persists whatever token it was given into the git config, so the later git push uses the App identity instead of GITHUB_TOKEN. Forget that line and you've fixed gh but not git push.

Why I prefer this: the token expires in an hour, it isn't tied to a person who might leave, and the commits show up as your-app[bot], which makes loop guards easy.

Fix 2: Use a fine-grained personal access token

Same idea, faster to set up. Create a fine-grained PAT scoped to the one repo, store it as a secret, and use it in place of GITHUB_TOKEN in both checkout and gh.

The trade-off is ownership. Every action shows up as you. It expires on whatever schedule you set, and the day it does, your releases go silent again. When you leave the team, so does the token.

Fix 3: Call `workflow_dispatch` explicitly

Since workflow_dispatch is one of the two exceptions, you can keep GITHUB_TOKEN and trigger the downstream workflow by name:

# publish.yml
on:
  push:
    tags: ['v*']
  workflow_dispatch:
    inputs:
      tag:
        required: true

# end of release.yml
permissions:
  contents: write
  actions: write   # required to dispatch workflows

- run: gh workflow run publish.yml --ref "v$VERSION" -f tag="v$VERSION"
  env:
    GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

No extra secrets. The catch: the actions: write permission is easy to forget, and the coupling is now explicit, so renaming publish.yml breaks the release job.

This doesn't help with bot-opened PRs. You can dispatch CI, but branch protection waits for checks reported against the PR, so for that case use Fix 1 or 2.

Fix 4: Don't chain at all

Often the cleanest answer. If publish always follows release, make it a second job in the same workflow:

jobs:
  release:
    # ...creates the tag
    outputs:
      version: ${{ steps.v.outputs.version }}
  publish:
    needs: release
    # ...npm publish

No events, no tokens, no invisible walls. I ended up doing this for npm publish and keeping the App token only for the PR bot.

Won't a non-GITHUB_TOKEN push cause an infinite loop?

It can, which is exactly why GitHub put the wall there. Once you swap in an App token or PAT, a workflow that commits on push will trigger itself again. Add a guard:

jobs:
  format:
    if: github.actor != 'release-bot[bot]'

Or put [skip ci] in the bot's commit message. GitHub skips push and pull_request workflows for commits containing it. Or use paths-ignore so changes to CHANGELOG.md and package.json don't retrigger the release workflow. One of the three, every time.

How do I spot this bug quickly?

Look for a missing run, not a failed one. My checklist now:

The downstream workflow has zero runs for the event you expected, while the upstream run is green.
Pushing the same tag or opening the same PR by hand makes it work.
A bot PR shows required checks as "Expected" with no run attached.
Grep the upstream workflow for secrets.GITHUB_TOKEN, github.token, and any action with an optional token input. Assume the default is GITHUB_TOKEN until the README says otherwise.

So why doesn't GITHUB_TOKEN trigger workflows?

GITHUB_TOKEN won't trigger workflows because GitHub ignores every event created with it, apart from workflow_dispatch and repository_dispatch, to stop workflows from triggering themselves forever. Tags, releases, pushes and pull requests made with the default token are real, but they start no new runs. You won't see an error, just a missing run. To chain workflows, authenticate with a GitHub App token or fine-grained PAT (and pass it to actions/checkout too), dispatch the downstream workflow explicitly with actions: write, or merge both steps into one workflow with needs:. If you switch tokens, add a loop guard before the next push.

Written by the developer behind Preterview, an interview prep platform.

tiktoken Undercounts Claude Tokens: 37 of 418 Requests Blew the Limit

jidonglab — Wed, 07 Oct 2026 16:44:27 +0000

My nightly digest job died at 2:14 a.m. with a 400 and the message prompt is too long. My packer had measured the prompt at 181,874 tokens, comfortably under the 200K window. Claude said it was 207,431.

That's a 14% gap, and it came from one lazy decision I made months earlier: counting Claude tokens with tiktoken. It's fast and local, and it's already in every Python environment I own. On my code-heavy inputs it was also wrong in the same direction every single time.

This post covers the failure, the audit of 600 chunks, and the fix that took me from 37 failures to zero.

TL;DR

tiktoken is OpenAI's tokenizer. Using cl100k_base to count Claude tokens gives you a different vocabulary's answer, not an approximation of Claude's.
On my corpus it undercounted by a median of 14%: about 9% on prose, 16% on Python, 27% on JSON and lockfiles, and 38% on the worst single chunk.
The error only bites when you pack close to the limit. 61 of my 418 requests filled the budget, and 37 of those 61 failed.
It skews cost estimates too. My dashboard said $41.20 for the month. The invoice said $46.90.
Fix: do a cheap tiktoken pass with a multiplier, then verify the fully assembled request with Anthropic's free messages.count_tokens endpoint and trim until it fits.

What was I building?

I have a Python job that runs every night across my side-project repos. For each repo it collects the day's diffs, the touched files, and any related docs, then packs them into one Claude request that writes a digest: what changed, what looks risky, what I forgot to finish.

Packing was greedy. Sort the files by relevance and add them until the budget is full:

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
BUDGET = 180_000  # leave ~20K for system prompt + slack

def pack(files):
    chosen, used = [], 0
    for f in files:
        n = len(enc.encode(f.text))
        if used + n > BUDGET:
            break
        chosen.append(f)
        used += n
    return chosen, used

I'd picked cl100k_base because a blog post once said it was "close enough" for Claude. I never checked. For two months it looked fine, because most nights the repos were small and the packer never got near the budget.

Then I pointed it at a monorepo with a fat package-lock.json and a folder of JSON test fixtures.

Why does tiktoken undercount Claude tokens?

tiktoken undercounts Claude tokens because it implements OpenAI's BPE vocabularies, and Claude uses a different tokenizer that splits the same text into more pieces. Two tokenizers trained on different data with different merge rules will disagree, and the disagreement depends on the content. Anthropic doesn't ship a local tokenizer for current Claude models, so there's no offline library that gives you the exact number.

I wanted the real size of the gap on my data instead of a vibe, so I sampled 600 chunks from the job's inputs. For each one I counted with cl100k_base and with Anthropic's count_tokens endpoint, then took the ratio of Claude's count to tiktoken's.

Content type	Chunks	Median ratio (Claude / tiktoken)	Worst chunk
Markdown docs, commit messages	180	1.09	1.15
Python source	210	1.16	1.24
TypeScript source	110	1.17	1.26
JSON, lockfiles, fixtures	100	1.27	1.38
All 600		1.14	1.38

The ratio is never below 1.0 on my corpus. tiktoken never once overcounted. That's the worst kind of error, because a symmetric error would at least cancel out sometimes.

Structured, whitespace-heavy, and symbol-dense text had the biggest gap. A minified JSON fixture full of UUIDs hit 1.38. Plain English prose stayed under 1.10.

One caveat matters more than the table: this ratio isn't a constant. It's specific to my content and to the model I counted against. Tokenizers change between model generations, so a ratio you calibrated last quarter can drift when you upgrade. Measure it on your own data, against the model you actually call.

Why did only some requests fail?

Only requests that packed close to the context limit were at risk, because a 14% undercount on a half-empty prompt never reaches 200K. Out of 418 requests over 19 nights, 357 never filled the budget and none of them failed. The other 61 filled it, and 37 of those blew up.

The 24 that survived were mostly docs-heavy repos. At a 1.09 ratio, 180K tiktoken tokens comes out to about 196K real tokens. That's tight but it fits.

Code and JSON pushed the ratio to 1.16 or higher, which puts the same 180K over 208K. The monorepo failed every night it ran.

Then there was the overhead I hadn't counted at all. My system prompt was counted with tiktoken too, so it had the same error. My three tool definitions (a file lookup, a git log reader, and a "flag this" tool) added 1,180 tokens that never went through any counter. Tool schemas are part of the input. I'd just never thought of them as text.

What did it do to my cost estimates?

My cost estimate was low by about the same 14%, because I priced every request off the tiktoken count. My little dashboard multiplied tiktoken counts by the per-token input price and reported $41.20 for the month. The real bill for that job was $46.90.

$5.70 isn't going to bankrupt anyone. But the dashboard was the number I used to decide whether to add more repos to the job, and it was quietly flattering me. If you're quoting AI features to a client off tiktoken math, you're under-quoting every one of them.

How do you count Claude tokens accurately?

Use Anthropic's messages.count_tokens endpoint with the exact model, system prompt, messages, and tools you're about to send. It's free to call (rate-limited separately from message creation) and returns the input token count for the whole assembled request, tool definitions included. That last part is the one I needed.

I didn't want a network round trip for every file during packing, though. So the fix has two passes: a fast pessimistic estimate to fill the prompt, then one real count on the final request, trimming if needed.

import anthropic
import tiktoken

client = anthropic.Anthropic()
enc = tiktoken.get_encoding("cl100k_base")

MODEL = "claude-sonnet-5-5"
TARGET = 194_000   # real tokens; headroom under the 200K window
FAST_RATIO = 1.2   # pessimistic tiktoken multiplier for the first pass

def real_count(system, messages, tools):
    resp = client.messages.count_tokens(
        model=MODEL, system=system, messages=messages, tools=tools,
    )
    return resp.input_tokens

def pack(files, system, tools):
    # Pass 1: cheap local estimate, deliberately inflated
    chosen, est = [], 0
    for f in files:
        n = int(len(enc.encode(f.text)) * FAST_RATIO)
        if est + n > TARGET:
            break
        chosen.append(f)
        est += n

    # Pass 2: ask Claude's own tokenizer, drop the least relevant file until it fits
    while chosen:
        messages = build_messages(chosen)
        n = real_count(system, messages, tools)
        if n <= TARGET:
            return messages, n
        chosen.pop()
    raise ValueError("system prompt + tools alone exceed TARGET")

A few details that took me an iteration to get right:

Count the assembled request, not the pieces. Message wrappers, file headers, and tool schemas all add tokens. Summing per-file counts misses them.
Keep headroom. Anthropic's docs describe the count as an estimate that can differ slightly from what you're billed. I target 194K, not 199,999.
Pass the real model ID. The count belongs to the tokenizer of the model you send to. Counting against one model and sending to another brings back the original bug.
Trim by relevance. chosen is sorted most-relevant first, so pop() drops the least useful file.

Did the fix actually work?

Yes: zero overflow failures across the next 21 nights and 463 requests. The second pass needed one count_tokens call on 79% of the requests that filled the budget and two on the rest. Nothing needed a third.

The costs are small:

Latency: the count call added a median of 170 ms to each request. The digest takes a minute to generate, so nobody will ever notice.
Packing efficiency: budget-filling prompts now average 191K real tokens against the 194K target. Before the fix they averaged "whatever tiktoken thought, plus a coin flip."
Cost tracking: the dashboard now logs input_tokens from the API response's usage field instead of my estimate. This month's figure matched the invoice to the cent.

That last change deserves its own line. You never need to estimate tokens after the fact. Every response tells you exactly what you were billed for, so log the actual number.

What would I tell past me?

Don't use a tokenizer from a different model family as a ruler, even if it's "close enough." It's close enough until you pack to the limit, and a packer exists to pack to the limit.

If you just need a rough guess for a UI ("about 40K tokens"), tiktoken times a calibrated ratio is fine. If a request fails, or a bill comes out wrong, when the number is off, ask the API.

So, can you count Claude tokens with tiktoken?

You can get a rough estimate, but you can't get an accurate count. tiktoken implements OpenAI's tokenizers, and Claude splits text differently. On my corpus cl100k_base undercounted Claude by a median of 14%, and by up to 38% on JSON. That broke 37 of the 61 requests I packed near the 200K window and understated my bill by $5.70 a month. For anything that has to fit a limit or match an invoice, call Anthropic's free messages.count_tokens endpoint on the fully assembled request (system prompt, messages, and tools) for the exact model you're calling, keep a few thousand tokens of headroom, and log real usage from the response's usage field.

Written by the developer behind Preterview, an interview prep platform.

DEV Community: jidonglab

A Local-First Plan, Then 4 Days of Scope Creep

The plan was good

Then the plan grew

The race I reproduced on a real database

What I'd change

GitHub Actions fetch-depth: 1 Made My CI Skip Tests for 41 PRs

TL;DR

What does fetch-depth: 1 actually fetch in actions/checkout?

Why did my changed-files check return nothing instead of failing?

Which git commands break in a shallow GitHub Actions checkout?

How do I fix fetch-depth: 1 for git diff in GitHub Actions?

Option 1: fetch-depth: 0 (full history)

Option 2: fetch-depth: 2 with the merge commit (PRs only)

Option 3: fetch the base branch yourself

How do I make git failures fail the job instead of passing?

Should "skip tests" ever be the default path?

So why does GitHub Actions fetch-depth: 1 break git diff?

I Built an Agent Team, Then Couldn't See What It Did

The name belonged to a dead project

The team I couldn't see

Where it broke

What the office looks like today

Where it stands today

Sync Anthropic Client Blocked My asyncio Loop: Voice p95 Hit 11.4s

TL;DR

What was I building?

Why did single-user testing look perfect?

What did the bug look like in production?

How do you detect a blocked asyncio event loop?

Why does a sync client inside async def block everything?

How bad was it under load?

Is asyncio.to_thread enough to fix a blocking SDK call?

What's the correct fix?

How do I stop this from coming back?

So why did a sync Anthropic client block my asyncio event loop?

Python json.dumps ensure_ascii: 24% More Claude Tokens, Wrong Kanji

TL;DR

What does json.dumps ensure_ascii actually do?

Why does ensure_ascii only break prompts and not API calls?

How many extra tokens does ensure_ascii cost?

Does escaping non-ASCII hurt LLM accuracy?

Where else does ensure_ascii hide in a Python LLM pipeline?

How do you fix it and keep it fixed?

The lesson I'm keeping

So does json.dumps ensure_ascii waste LLM tokens?

Cosine Similarity Threshold 0.8: Embedding Anisotropy Broke My RAG

TL;DR

What happened when I switched embedding models?

Why is cosine similarity between unrelated texts so high?

Why does the score scale change between embedding models?

Is a cosine similarity threshold ever safe to use?

How do I calibrate a cosine similarity threshold?

Does mean-centering embeddings fix anisotropy?

What should I gate on instead of a raw score?

The checklist I run now on every embedding change

So, does a cosine similarity of 0.8 mean anything?

Docker Compose depends_on Won't Wait for Postgres: 11/50 Runs Failed

TL;DR

What does Docker Compose depends_on actually wait for?

How do I reproduce the depends_on race?

How do I make Docker Compose wait until Postgres is ready?

Why does pg_isready say ready before Postgres accepts connections?

What about migrations that must finish before the app starts?

Is a healthcheck enough, or do I still need retries?

The checklist I paste into every Compose file now

So does Docker Compose depends_on wait for the database?

Ollama num_ctx Truncated 287 of 400 Prompts and Never Told Me

TL;DR

What was I building?

Why did the model ignore my instructions?

How do you detect Ollama prompt truncation?

How many of my prompts were affected?

How do you set num_ctx in Ollama?

What did the bigger context window cost?

What were the results after the fix?

What should you change in your own Ollama setup?

So why did Ollama truncate my prompts without telling me?

bge-reranker Truncates at 512 Tokens: 31% of My Chunks Got Cut

TL;DR

Option 1: `fetch-depth: 0` (full history)

Option 2: `fetch-depth: 2` with the merge commit (PRs only)

Fix 3: Call `workflow_dispatch` explicitly