DEV Community: frank chu

Every checker I wrote this month had the same bug: it could not tell using a word from talking about one

frank chu — Tue, 29 Sep 2026 18:03:36 +0000

I changed the title of a draft last week. Nothing else. The body was byte-identical, which I checked rather than assumed. My writing checker went from a score of 0 to a score of 42 and flipped from pass to rewrite.

The title was deliberately terrible, so the flag was fair. What was not fair is that the title is not the article. It renders as the headline and the social card; it is not prose a reader wades through. My checker had no idea there was a difference, because it read the file as one undifferentiated blob of text.

Then I went back through the other tools I have written for this repo, and found I had shipped the same mistake five times.

A word-level checker counted flagged words inside the HTML comment at the top of my drafts, the one where I write notes to myself about the writing. Notes about tell-words, scored as tell-words. A gate that requires a post to contain real code counted the fenced blocks inside that same comment, so a post could pass by having code in its review notes and none in its body. A check that makes sure private notes never leak into a published file fired on a post that discussed the leak pattern in prose, because the prose contained the pattern. A script that wires a cover image into front matter skipped a post whose body happened to contain the words cover_image: in a code sample.

Five tools, five unrelated jobs. Same defect. Each one was handed a file and treated every byte of it as the thing it was checking, when in fact the file has regions with different meanings: notes that are not content, metadata that is not prose, quoted examples that are not claims.

Philosophers have a name for this, the use-mention distinction, and it is the difference between cat has three letters and a cat has four legs. It sounds like a word game until you write a tool, and then it turns out to be the entire problem. A linter that cannot distinguish a word you are using from a word you are discussing will flag every article about bad writing, including this one.

What makes it hard to notice is that the failure is always plausible in isolation. The tool ran. It produced a number. The number was even defensible if you squinted, since the title really does contain the word it flagged. Nothing crashes. You only catch it by constructing the case where the two readings diverge, and you only think to construct that case if you already suspect the bug exists.

Every fix turned out to be the same move: decide explicitly which slice of the file the tool is about, extract that slice, check only it. Three lines each time. The thinking was the expensive part, and it was the same thinking five times without me recognising it on the second, third, or fourth occasion.

I am not sure what the generalisation is. Part of me says the tools should share a parser, so that "the body of a post" is defined once instead of re-derived by each script from its own regex. That is clearly right for this repo. But I do not think it explains the pattern, because the scripts were written weeks apart for different reasons, and at no point while writing any of them did the question "what counts as the content here" feel like an open question. It felt like the file was obviously the content. That is the part I would want to fix, and it lives upstream of the code.

A postscript I did not expect to be writing. Since drafting this I collapsed those five scripts into one, specifically to fix this class of bug in one place, and the consolidated tool committed the same error twice more on its first two runs. Seven instances now. Knowing the failure mode, while actively writing about the failure mode, protected me from nothing.

Two things I would like to know from people who build this kind of tooling:

Does your linter, formatter, or scanner know which parts of a file are the subject and which are scaffolding? I am specifically curious about documentation and content tooling, where front matter and quoted examples are the norm rather than the exception. My suspicion is that this is near-universal and mostly invisible because nobody writes the adversarial case.
When you find the same bug class in the fifth place, what do you actually do about it? Writing one shared helper is the obvious answer, and I will probably do it, but the honest problem was that I did not recognise instances two through four as the same thing at the time. Is there a way to get better at that, or is noticing it on the fifth try just what the process looks like?

A server sent me Retry-After: 0 and my retry loop lost its brakes

frank chu — Tue, 29 Sep 2026 18:02:56 +0000

While tracking down an unrelated 403 last week I printed the full response headers, and one of them stopped me:

status      : 403
server      : Varnish
retry-after : '0'
body length : 0

Retry-After: 0. I had a helper in the same repo whose entire job is to read that header and sleep for that long, and I could see immediately what it would do with a zero. What I could not do was guess the magnitude, so I measured it.

The loop, and what it actually does

This is close to what I had, and close to what turns up if you search for how to handle a 429:

import time, urllib.request, urllib.error

def naive_retry(url, attempts=5):
    for i in range(attempts):
        try:
            return urllib.request.urlopen(url, timeout=5)
        except urllib.error.HTTPError as e:
            if e.code in (403, 429, 503):
                wait = int(e.headers.get("Retry-After", 2 ** i))
                time.sleep(wait)
                continue
            raise
    raise RuntimeError("out of attempts")

The 2 ** i fallback is the part that makes it look safe. Exponential backoff is right there in the code, so the function reads as if it backs off. But the fallback only runs when the header is missing. When the header is present, whatever it says wins, and the carefully written backoff never executes at all.

I did not want to test this by hammering a real service, so I stood up a stub that replays the exact response I got: 403, Retry-After: 0, empty body. That technique is not mine. A reader named @pm25coder found a defect in an earlier version of this same helper by running it against a stubbed clock, which was the first time it occurred to me that you can test a retry policy without waiting for it.

import http.server

class H(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        HITS[self.path] += 1
        self.send_response(403)
        self.send_header("Retry-After", "0")
        self.send_header("Content-Length", "0")
        self.end_headers()

Five attempts against it, ten repetitions, on a 12-core M2 Pro at load average 7.6:

naive  (trusts Retry-After)   5 requests in 1.3ms - 15.0ms   (332 - 3896 req/s, median 3634)
guarded (floor 1s, cap 30s)   5 requests in 5.025s           (1.0 req/s)

The loop I would have described to you as "it backs off between attempts" completes its entire retry schedule inside a single frame of video, and every one of those requests goes to a machine that has just told me to go away.

I want to be careful about that range rather than quote you one number, because I ran the same code five days ago and got 187 req/s. Not a different result. A different afternoon: my laptop was under a load average of 70 at the time from an unrelated experiment. The spread within today's ten runs alone is twelve-fold.

So the throughput figure is a property of the machine, and the only part that transfers is the shape. Five attempts is nothing; what matters is that the count scales with whatever you set attempts to, scales again with however many workers run the same helper, and does all of it at the exact moment the far side is least able to absorb it. The right mental model is not "187 requests a second." It is "as fast as your process can issue them, with the brakes disconnected."

Where a zero comes from

I do not think anyone typed Retry-After: 0 as advice. Reading the RFC, the header takes either a delay in seconds or an HTTP date, and nothing forbids zero. My best guess is that this is a template with an unset variable rendering to its zero value, on a response path that was never meant to be retried at all, since the body is empty and the rejection is permanent for that client.

Which is the general case, not a quirk of one site. Retry-After is a number chosen by a machine that does not know anything about your workload, sent on a path that may not have been thought about carefully, and your client obeys it without question. The other failure mode is the same bug with the sign flipped: I have seen Retry-After: 86400 on a transient error, which turns a five-second blip into a worker that sleeps for a day.

So treat it as input, not instruction:

import time, urllib.request, urllib.error

def guarded_retry(url, attempts=5, floor=1.0, cap=30.0):
    for i in range(attempts):
        try:
            return urllib.request.urlopen(url, timeout=5)
        except urllib.error.HTTPError as e:
            if e.code in (403, 429, 503):
                hdr = e.headers.get("Retry-After")
                try:
                    wait = float(hdr) if hdr is not None else 2 ** i
                except ValueError:          # it can also be an HTTP date
                    wait = 2 ** i
                wait = min(max(wait, floor), cap)
                time.sleep(wait)
                continue
            raise
    raise RuntimeError("out of attempts")

Three things changed, and each one fixes a separate failure.

float() instead of int(), inside a try. The spec allows an HTTP-date, and int("Wed, 21 Oct 2026 07:28:00 GMT") raises ValueError from inside your error handler, which is a fun way to lose the original exception.

max(wait, floor) is the one that fixes today's bug. A server's idea of an acceptable request rate is not binding on your client's idea of a sane one.

min(wait, cap) fixes the opposite bug, the one that has probably cost me more hours in aggregate: a process that looks hung but is politely asleep.

The wider version

The honest lesson is not about this header. It is that my code had a defensive-looking default sitting next to an unconditional trust of remote input, and the default is what I saw when I read the function back. 2 ** i is visible. int(header) is also visible. What is invisible is that the second one makes the first one dead code in exactly the situation you wrote it for.

I went looking for the same pattern elsewhere in the repo afterwards and found two more: a page size from a response used directly as a loop bound, and a timeout read from a config file with no upper limit. Both had a reasonable-looking fallback that a remote value silently outranked.

The grep that finds these is not clever. Look for anywhere a number crosses the boundary into your process and reaches sleep, range, timeout, or a buffer size without passing through a min or a max on the way.

What is the worst value a remote server has ever handed your client that your client simply believed? I would like to collect a few of these, because I suspect the zero is not even the funny one.

The Server header told me which layer rejected my request, and it wasn't the one I was debugging

frank chu — Tue, 29 Sep 2026 18:02:16 +0000

I had a valid API key, the right Accept header, and a 403. So I did what you do: regenerated the key, checked the account, re-read the auth docs. None of it was the problem, and I had ignored the one header that would have told me so in the first minute.

--- request with the default Python UA -> 403
    server: Varnish
    content-length: 0
    retry-after: 0

--- request with User-Agent: WhatTechPost/1.0 -> 200
    server: Heroku
    content-type: application/json; charset=utf-8
    x-request-id: 57878af6-2c5f-2391-8efc-8984dd2b59ea

Two different values in Server means two different machines answered me. The 403 came from Varnish, a cache sitting in front of the app. The 200 came from Heroku, where the application actually lives. Whatever rejected me was not the thing that knows about API keys, because my request never got far enough for anyone to look at one.

Proving the ordering

If the gate really sits in front of auth, then the key should not matter on one side and should matter on the other. That is a two-by-two you can just run:

import urllib.request, urllib.error

def probe(ua, key):
    h = {"Accept": "application/vnd.forem.api-v1+json"}
    if ua:  h["User-Agent"] = ua
    if key: h["api-key"] = key
    req = urllib.request.Request(
        "https://dev.to/api/articles/me/published?per_page=1", headers=h)
    try:
        r = urllib.request.urlopen(req, timeout=20)
        return r.status, r.headers.get("server"), r.read()[:40]
    except urllib.error.HTTPError as e:
        return e.code, e.headers.get("server"), e.read()[:40]

default UA  + valid key    -> 403 Varnish  b''
default UA  + no key       -> 403 Varnish  b''
default UA  + garbage key  -> 403 Varnish  b''
custom UA   + valid key    -> 200 Heroku   b'[{"type_of":"article","id":4704524,...'
custom UA   + no key       -> 401 Heroku   b'{"error":"unauthorized","status":401}'
custom UA   + garbage key  -> 401 Heroku   b'{"error":"unauthorized","status":401}'

The left column is completely insensitive to the key. The right column is completely sensitive to it. That is the ordering, and it also explains the empty body: an edge rejection has nothing to say, because the code that writes helpful JSON errors lives downstream of it. A zero-length body on a 4xx is itself a signal. Applications explain themselves. Proxies hang up.

What is actually on the blocklist

At this point I assumed it was a general "looks like a bot" heuristic, so I threw a spread of client strings at it to find the shape.

'python-urllib'           -> 403 Varnish
'urllib'                  -> 403 Varnish
'urllib3'                 -> 403 Varnish
'my-python-urllib-client' -> 403 Varnish
''                        -> 403 Varnish
'Java/17.0.1'             -> 403 Varnish

'python-requests'         -> 200 Heroku
'curl/8.4.0'              -> 200 Heroku
'Wget/1.21'               -> 200 Heroku
'okhttp/4.12.0'           -> 200 Heroku
'Go-http-client/1.1'      -> 200 Heroku
'scrapy/2.11'             -> 200 Heroku

It is not a bot heuristic. scrapy is a scraping framework and it sails through. Wget sails through. What gets stopped is a hand-picked list of substrings, and urllib is on it.

Then I checked the casing, mostly out of habit, and got the result that made me want to write this down:

urllib   -> 403        URLLIB   -> 200
Urllib   -> 200        urllib3  -> 403
uRllib   -> 200        URLlib3  -> 200

The match is case-sensitive. urllib is blocked and Urllib is not. One capital letter is the whole difference between a 403 and a 200.

Why this lands on Python specifically

The practical damage is narrow and unlucky. Here are the real default User-Agent strings of the clients people actually use:

urllib (stdlib)   Python-urllib/3.14        -> 403
urllib3 direct    python-urllib3/2.2.1      -> 403
requests          python-requests/2.32.3    -> 200
httpx             python-httpx/0.27.0       -> 200
aiohttp           Python/3.14 aiohttp/3.9.5 -> 200

Two of five, and they are the two that ship lowest in the stack. If you reach for requests you never see this. If you write a script with no dependencies, which is exactly what you do when you are poking at an API for the first time, you get a bare 403 that says nothing and points at your credentials.

The fix is one line, and I want to be clear that it is a fix for my client, not a bypass of anything. The site publishes this API and issues me a key for it; what it declines to serve is an anonymous default string.

HEADERS = {
    "api-key": KEY,
    "Accept": "application/vnd.forem.api-v1+json",
    "User-Agent": "MyApp/1.0 (+https://example.com/contact)",
}

Name your client. It costs nothing, it is what the header is for, and it means that when someone on the other end wonders who is hammering an endpoint, there is an answer.

The part I want to keep

I spent about forty minutes on a credentials theory that the response had already ruled out. What I should have done first, and now do first, is read Server, Content-Length, and whether the body is machine-readable, before reading the status code's usual meaning.

A 403 does not mean "you lack permission." It means something returned 403. Which something is a different question, and the headers usually answer it before you have to guess.

def where_did_this_come_from(e):
    return {"status": e.code,
            "server": e.headers.get("server"),
            "request_id": e.headers.get("x-request-id"),   # app layer sets this
            "body_len": len(e.read())}                     # 0 == edge, probably

x-request-id is the tell in the other direction. It is present on every 200 and absent on the 403, because the app assigns it and the app never saw me.

What is the longest you have spent debugging the wrong layer because the status code sounded like it was describing your problem? I want to hear the one where the header was sitting right there.

Five things I read in an HTTP response before I read the status code

frank chu — Sun, 27 Sep 2026 01:01:41 +0000

I lost about forty minutes this week to a 403 that had nothing to do with permissions, on an API I have been using daily for three months. The response had told me the answer immediately. I just was not reading that part.

Since then I have been going through responses in a fixed order before I let myself form a theory, and it has caught three things it would previously have taken me a while to find. The order is roughly "who answered, what did they say, and do they want me to come back."

1. `Server`, because it tells you which machine is talking

Every example here comes from the same endpoint, minutes apart:

403 response          200 response
  server: Varnish       server: Heroku

Two values means two different machines answered. Varnish is a cache in front of the application; Heroku is where the application runs. A rejection from the first one never reached the second, which immediately rules out every theory that involves application logic, including credentials, permissions, quotas, and your code.

This is the check with the best ratio of effort to information and it is the one I had never bothered with. Not every deployment exposes it, and plenty of stacks return the same value from every layer, but when it differs between a working and a failing request you have located the problem before you have started looking.

2. Whether the body is empty

403 -> content-length: 0     body: b''
401 -> content-type: application/json
       body: b'{"error":"unauthorized","status":401}'

Both of these are authentication-adjacent failures and only one of them explains itself. That is not an accident of who wrote the error handler. Applications know why they refused you and have a serialiser handy, so they tend to say. Edges and proxies are matching a pattern and dropping the connection, so they have nothing to say and often no JSON serialiser in the path at all.

A zero-length 4xx body is therefore weak evidence that you are talking to infrastructure, not to the service. Combined with point 1 it is usually conclusive.

3. Whether a correlation ID came back

200 -> x-request-id: 57878af6-2c5f-2391-8efc-8984dd2b59ea
403 -> (absent)

The application assigns the request ID. If it is missing, the application did not see your request. If it is present, it did, and you now have the one string that makes a support conversation productive instead of a description of your afternoon.

I have started logging it on every response, not just failures, because by the time you want it you cannot reproduce the request that had it. The header name varies by stack: x-request-id, x-correlation-id, x-amzn-requestid, cf-ray, x-ms-request-id. Grab whichever one exists.

4. `Retry-After`, and then don't trust it

The response I got carried this:

retry-after: 0

Zero. My retry helper read that header and slept for that long, which means five attempts completed in 27 milliseconds, all of them aimed at the machine that had just blocked me. Measured, at 187 requests per second.

The header is remote input, and nothing stops it being zero, or a date, or a number so large your worker naps for a day. Clamp it at both ends and keep your own backoff as the fallback:

hdr = e.headers.get("Retry-After")
try:
    wait = float(hdr) if hdr is not None else 2 ** attempt
except ValueError:          # the spec also allows an HTTP-date
    wait = 2 ** attempt
wait = min(max(wait, 1.0), 30.0)

The float inside a try is not paranoia. Retry-After is legally an HTTP-date, and int("Wed, 21 Oct 2026 07:28:00 GMT") raises from inside your exception handler.

5. The length of a list, compared to what you asked for

This one is not a header, and it is the one that cost me an actual published mistake.

per_page=5     ->   5 returned
per_page=60    ->  60 returned
per_page=100   ->  95 returned
per_page=1000  ->  95 returned

The true total is 95. Every request below that came back exactly full, and the response body carries no total, no count, no next-page link and no has-more flag. From inside a single call there is no way to tell a complete result from a first page.

So the rule I now apply everywhere: if a collection comes back with a length exactly equal to the page size you requested, treat it as truncated. Real populations do not land on round numbers. A cheap version, when you do not want to write the pagination loop yet:

rows = get(f"{url}?per_page={n}")
assert len(rows) < n, f"got exactly {n} rows; this is a page, not a result set"

It converts a silent wrong answer into a loud one, which is the trade I want on anything feeding a number I am going to act on.

The actual habit

None of these are clever and I suspect most people reading this already knew four of them. The change that mattered was doing them in a fixed order, before forming a hypothesis, because the failure mode was never ignorance. It was that "403" has a meaning I have internalised, so reading it started a search for a permissions bug, and the search felt productive enough that I did not go back and look at the rest of the response for half an hour.

A status code is a three-digit summary written by whoever gave up on your request. The headers say who that was.

Which response detail do you check that I have not listed? I am building this into a helper and I would rather steal a sixth than discover it the way I discovered these.

77% of the Python I published references names it never defines

frank chu — Sun, 27 Sep 2026 01:01:01 +0000

I wanted a cheap number for a different post, so I pulled every fenced code block out of my archive and ran each Python one through ast.parse. My expectation was that a few would be broken snippets and I would have a tidy confession to write.

posts: 94   fenced blocks: 118
python blocks: 48  — compiling each with ast.parse
  parse OK : 48
  SyntaxErr: 0  (0%)

Nothing. Forty-eight for forty-eight. I sat with that for a minute and then realised I had measured the wrong thing, because syntactic validity is a bar that a snippet clears by being typed carefully, and nobody's actual complaint about a code sample is that it fails to tokenise.

The complaint is that you paste it and it does not run. So I wrote the check for that instead.

Free names

The question "would this run if you pasted it into an empty file" has a decent static approximation: walk the AST, collect every name the code binds, collect every name it reads, and subtract. What is left over is names the block expects to already exist. Anything not in builtins is something the reader has to supply.

import ast

class Scope(ast.NodeVisitor):
    def __init__(s): s.bound, s.used = set(), []
    def _alias(s, a): s.bound.add((a.asname or a.name).split(".")[0])
    def visit_Name(s, n):
        s.bound.add(n.id) if isinstance(n.ctx, ast.Store) else s.used.append(n.id)
    def visit_FunctionDef(s, n):
        s.bound.add(n.name); A = n.args
        for a in A.args + A.kwonlyargs + A.posonlyargs + [x for x in (A.vararg, A.kwarg) if x]:
            s.bound.add(a.arg)
        s.generic_visit(n)
    visit_AsyncFunctionDef = visit_FunctionDef
    def visit_ClassDef(s, n): s.bound.add(n.name); s.generic_visit(n)
    def visit_ExceptHandler(s, n):
        if n.name: s.bound.add(n.name)
        s.generic_visit(n)
    def visit_comprehension(s, n):
        for t in ast.walk(n.target):
            if isinstance(t, ast.Name): s.bound.add(t.id)
        s.generic_visit(n)
    def visit_Import(s, n): [s._alias(a) for a in n.names]
    visit_ImportFrom = visit_Import

It is deliberately crude. It flattens all scopes into one, so it will not catch a name used before assignment, and it over-approximates what is defined. That means every name it reports is a real miss, and the true number is at least this bad.

python blocks that parse            : 48
  self-contained (all names defined): 11  (22%)
  reference undefined names         : 37  (77%)

Better than three quarters of them. Eleven blocks out of forty-eight can be pasted into a file and run.

Not all misses are equal

A raw count gives you nothing to act on, because plenty of blocks are illustrative by design and should not carry twelve lines of setup. So I split the 61 distinct undefined names into three buckets, and the buckets turned out to be different problems with different answers.

stdlib modules never imported (7) : html json re subprocess tempfile time urllib
ALL-CAPS config constants (5)     : API_KEY ARTICLE_ID CHROME HEADERS TEMPLATE
post-local helpers and types (49) : get, api, client, spec, Question, Result, RetryPolicy, ...

The middle bucket is fine, and arguably good. API_KEY undefined is a signal to the reader that they supply their own, and writing API_KEY = "sk-..." in a post is worse than omitting it.

The third bucket is mostly fine too. If a block calls a helper I defined three paragraphs earlier, that is normal prose-with-code and re-declaring it in every snippet would be noise. Some of those 49 are type names from a library the post is about, where the import line is pure ceremony.

The first bucket has no defence at all. Seven stdlib modules, appearing across 7 of 48 blocks, fourteen percent of everything I have published. These are pure friction: someone pastes the snippet, gets NameError: name 'subprocess' is not defined, and now has to reconstruct which import I left out. It is the single easiest thing in this entire exercise to fix and it is the one I got wrong most often, because an import line at the top of a snippet feels like clutter while you are writing and feels essential the moment you are reading.

The other 41

While I had the data out:

by language: python 48, (none) 41, bash 11, js 6, yaml 5, json 2, markdown 2, ts 1, html 1, css 1

Forty-one blocks with no language tag. Those are mostly terminal output and console logs, where untagged is a reasonable choice. But I spot-checked and some are real code that just never got a tag, which means no highlighting and no chance of a static check like this one ever seeing them.

What goes in the pipeline

I already have a gate that fails a post claiming to be technical without enough code in it. It was counting blocks. Counting blocks is how you get 118 fences and 77% of them unrunnable.

import ast, sys

def block_report(md):
    for lang, code in FENCE.findall(md):
        if lang.lower() not in ("python", "py"):
            yield lang, None
            continue
        try:
            tree = ast.parse(code)
        except SyntaxError as e:
            yield lang, f"SyntaxError: {e.msg}"
            continue
        free = free_names(tree)
        missing = [n for n in free if n in sys.stdlib_module_names]
        yield lang, f"missing import: {missing}" if missing else None

I am only making the stdlib-import case a hard failure. The other two buckets get printed as a warning and I decide per block, because "this snippet needs context from the post" is a legitimate authoring choice and a gate that forbids it would make every post worse.

That split matters more than the check. My first instinct was to require every block to be standalone, and I am fairly sure that instinct produces posts full of ceremonial imports and redeclared helpers that bury the four lines the post is actually about. The goal is not that every block runs. The goal is that a block which looks like it should run, does.

If you write technical posts, what is your rule for imports in snippets? I have been treating them as clutter for three months and the data says I was optimising for the wrong reader.

I have the same boolean in 94 files. It is wrong in 90 of them.

frank chu — Sun, 27 Sep 2026 01:00:22 +0000

I was reconciling my blog archive against the API and printed a cross-tab I expected to be boring:

files in published/           : 94
  front matter published:true : 0
  front matter published:false: 94
  actually live on dev.to     : 90

Zero. Not a few stragglers. Every file in the directory named published/ declares that it is not published, and 90 of them have been live for weeks. The flag matches reality in 4 files out of 94, and it does that by being wrong in the same direction as four posts that never shipped, which is not agreement so much as a broken clock.

The line

# scripts/publish.py, inside prepare_body(). `import re` is at the top of the module.
if publish:
    text = re.sub(r"^published:\s*false\s*$", "published: true",
                  text, count=1, flags=re.MULTILINE)
return text

That is from the function that prepares a post before sending it. The regex is correct. The flag is correct. The payload that goes to the API is correct, which is exactly why every post actually published fine and nothing ever alerted me.

text is a local string holding a copy of the file. The substitution rewrites the copy. The copy becomes the request body. The file on disk is never opened for writing, in this function or anywhere else in the script. Everything about the line reads like a state change, including the verb in re.sub, and it is a pure transformation of a value that is about to be thrown away.

So the on-disk flag does not record whether a post is published. It records what the file said before a publish attempt, forever, regardless of outcome.

Four sources of truth, in descending order of authority

What made this worth more than a one-line fix is how many things in my repo were confidently claiming to know the answer.

published/2026-08-21-anthropic-model-2-withheld.md
    first commit : 632076f "Publish: Anthropic Model 2 post + cover + CoBench chart"
    directory    : published/
    front matter : published: false
    live on API  : NO

The commit message says Publish. The directory says published. The flag says false. The API, which is the only one of the four that actually knows, says the post does not exist. Three of my four signals are derived from my intent at the moment I ran a command, and none of them from the result.

That post is not live, by the way. I do not know why. The commit from that day shows the file moved and the images added, so the most likely story is that the API call failed or was never made after the move, and every subsequent tool I wrote believed the directory.

The check that should have existed on day one

import glob

def reconcile():
    live = {a["title"] for a in all_published_from_api()}
    for path in glob.glob("published/**/*.md", recursive=True):
        fm = front_matter(path)
        on_disk = fm.get("published") == "true"
        on_api  = fm.get("title") in live
        if on_disk != on_api:
            yield path, on_disk, on_api

Eleven lines, and running it today is what produced every number in this post. I did not write it for two months because the archive felt like it was self-describing. The file is in published/, so obviously it is published.

There is a decent argument that the flag should not exist on disk at all. It is a request parameter for the API, not a property of my file, and duplicating remote state into a local file is how you end up with a field nobody updates. The version I am moving to keeps the flag out of the archive entirely and derives live status from the API when anything needs it, with a small cache.

But I want to name the failure more precisely than "don't duplicate state", because I would have agreed with that advice and still written this bug. The specific trap is that re.sub on a variable you are about to send is indistinguishable, at a glance, from re.sub on a variable you are about to save. Same function, same shape, same line. The difference lives entirely in what happens to the result three lines later, and when I read that function back during review, I read the intent rather than the data flow.

The two greps I ran on the rest of the repo afterwards:

rg 're\.sub|\.replace\(' --type py -A3 | rg -v 'open\(|write\(|Path\('

Any in-place-looking string transform with no write nearby. And the one that found the second instance:

rg 'def \w+\(.*path' -A20 --type py | rg 'return (text|body|content|data)'

A function that takes a path, transforms, and returns instead of persisting. Two of those were fine and one was the same bug in a cover-image script.

What I actually changed

The fix is not writing the flag back. If I did that, I would have a boolean that is correct at publish time and starts rotting immediately, because a post can be unpublished or deleted on the platform without my repo hearing about it. Local mirrors of remote state have exactly one correct value, which is the value at the moment you last synced, and they never say when that was.

So the archive stops claiming. The reconcile function runs as a check, it prints disagreements, and the API stays the only thing that gets to answer the question.

If you have a field in your repo that mirrors a remote system's state, when did you last verify it? Mine went 94 files deep before I thought to ask, and I only asked because I was looking for something else.

My linters have never found a real bug in my posts. Readers have found two.

frank chu — Fri, 25 Sep 2026 04:18:03 +0000

I have three automated checks standing between a draft and the publish button. One scores the writing for machine-sounding patterns. One fails anything that claims to be technical without real code in it. One makes sure my private review notes never leak into a live post.

In two weeks they have caught: some filler words, a few em-dashes, and a post that was prose pretending to be a tutorial. Useful, all of it.

In the same two weeks, readers found two actual defects in code I had published. One person took a retry helper out of a post, ran it against a stubbed clock, and showed that my 45-second budget would run for 120 seconds when the server sent a Retry-After header, because I checked the budget before sleeping instead of comparing it to the sleep. Another re-ran a query from a different post and found I had compared a 68-item dataset against a 60-item one, because a page size had truncated silently.

Both were real. Both were in the part of the post a reader would copy. My three gates had nothing to say about either, and I want to be precise about why, because "add more linting" is the wrong lesson.

The checks I built all answer questions about form. Does this text have the shape of machine writing. Does this file contain fenced code. Does this body contain a private comment. Those are decidable by looking at the artifact, which is exactly why I could automate them.

What the readers checked was whether the thing is true. Does this function respect its own stated budget. Do these two numbers describe the same set of things. You cannot answer either by inspecting the text. You have to execute it, or reproduce the query, and then compare the result against what the prose claims. That is not a linting problem with a harder linter at the end of it; it is a different category of question.

The uncomfortable part is that I reviewed both posts carefully and approved both defects. I am not a reliable checker of my own claims at the moment I finish making them, and the tooling I built to compensate was built to catch the mistakes I already knew I made. It could never have caught the ones I did not know about, because I designed it from my own model of my failures.

There is a version of this that ends in "so write tests for the code in your blog posts," and I do not think that is quite it either. The retry helper was illustrative, twelve lines, the kind of snippet nobody tests. The point is not the test. The point is that someone with a different mental model ran it and found out, and that is a resource I cannot build in a script.

So I am starting to think of the comment section as the only part of my verification pipeline that can check semantics, which reframes what a correction is. It is not embarrassment to be minimised. It is the most expensive kind of review I get, and I do not pay for it.

Two things I want to know:

Has an automated check in your own setup ever caught something that would have shipped as a bug, or do yours also mostly catch style? I am curious whether my split is typical or whether I built the wrong gates.
When someone corrected something you shipped publicly, did you fix it visibly or quietly? I have started leaving the correction in the post with the person's name on it, and I am not fully sure that is right, only that a silent edit reaches none of the people who already copied the broken version.

I published a benchmark. Two weeks later the same code ran 2.5x faster.

frank chu — Fri, 25 Sep 2026 04:17:24 +0000

Two weeks ago I published a timing measurement and called it a fixed cost. Five runs, 2.24 to 2.36 seconds each, variance of 0.12 seconds. I wrote that the tightness was "the signature of a fixed cost rather than work that scales with the input," and I built a whole argument on it.

Today, same machine, same script, same Chrome flags:

12 sequential runs: min 0.78s  max 1.28s  mean 0.93s  stdev 0.164s
first run 1.28s vs rest mean 0.90s  (cold-start effect: +0.37s)

0.93 seconds. Two and a half times faster than the number I published, from code I have not touched.

What actually differed

Not the code. Not the Chrome version. The machine:

machine: Apple M2 Pro | cores 12
load:  70.59  35.02  17.73

A one-minute load average of 70 on a 12-core box. I had spent the previous ten minutes firing two dozen concurrent Chrome processes at it for a different experiment, and I took this measurement in the wake of that.

Which means neither number is the truth. The 2.3s I published was measured under whatever my machine happened to be doing that afternoon, and the 0.93s today was measured under a load spike I created myself. I reported the first one as a property of the software. It was a property of the afternoon.

The tightness fooled me, and that is the interesting part

Here is what I got wrong conceptually, and I think it is a common error.

Five runs clustered inside 0.12 seconds looks like strong evidence. Low variance reads as a signal that you have isolated a real constant, and I said so explicitly in the post. But low variance within a sample tells you the conditions were stable during that sample. It says nothing about whether those conditions are the normal ones.

A tight cluster measured under a consistent-but-unrepresentative load is exactly as tight as a tight cluster measured under normal load. The spread cannot distinguish them. I treated precision as accuracy, which is the oldest measurement mistake there is, and I did it while writing a post about measuring things properly.

What I should have recorded

The reading is not useless, it is just incomplete. A timing number without its conditions is not reproducible by anyone including me. The minimum I should have captured alongside it:

import os, subprocess, time, statistics

def context():
    load1, load5, load15 = os.getloadavg()
    chrome = subprocess.run([CHROME, "--version"], capture_output=True, text=True).stdout.strip()
    return {"load_1m": round(load1, 2), "load_15m": round(load15, 2),
            "cores": os.cpu_count(), "chrome": chrome}

def timed(fn, n=12):
    xs = []
    for _ in range(n):
        t0 = time.time(); fn(); xs.append(time.time() - t0)
    return {"n": n, "min": min(xs), "max": max(xs),
            "mean": statistics.mean(xs), "stdev": statistics.pstdev(xs),
            "first": xs[0], "rest_mean": statistics.mean(xs[1:]),
            **context()}

Two details in there I would not have bothered with before. Reporting first separately from rest_mean surfaces cold start, which today was +0.37s and would have been invisible in a mean. And capturing load_1m at measurement time is the single line that would have caught this entire error, because a load average of 70 in the output would have stopped me publishing.

The concurrency numbers, with the same caveat

The same session, measuring batch rendering:

unbounded parallel n=6    7.75s   1.29s/img
unbounded parallel n=12   7.53s   0.63s/img
unbounded parallel n=24  14.00s   0.58s/img
bounded pool(4)    n=12   6.45s   0.54s/img
bounded pool(4)    n=24  13.46s   0.56s/img

A reader who runs a screenshot service told me that spraying unbounded processes stops working past a few dozen renders, because memory contention makes that per-process constant unpredictable and something eventually gets OOM-killed. I did not reproduce an OOM kill on this box: all 24 renders succeeded in every configuration. But the direction supports him. At n=12 a bounded pool of four beat unbounded parallelism outright, 6.45s against 7.53s, which is the opposite of what "more concurrency is faster" predicts and exactly what contention looks like when it starts to bite.

I am not publishing those as constants either. They are one machine, one afternoon, one load profile.

What I am changing

The post with the wrong number now carries a correction. Beyond that, I am not going to publish a timing figure again without the conditions attached, because a benchmark without its environment is not a measurement, it is an anecdote with decimal places.

And the general version, which cost me two public numbers to learn: a tight cluster proves your conditions were stable, not that they were typical. If you cannot say what the machine was doing while you measured, you do not know what you measured.

What is the most embarrassing benchmark you have had to retract? I would like company.

If a list endpoint returns exactly as many rows as you asked for, you have a bug

frank chu — Fri, 25 Sep 2026 04:16:44 +0000

A reader ran the numbers in one of my posts against the live API and told me they did not add up. He was right, and the tell had been sitting in my own published output for a week.

60 posts | 690 views | 3 reactions

Sixty. I had asked for sixty.

mine = get("https://dev.to/api/articles/me/published?per_page=60", auth=True)

per_page is a page size, not a limit. My call took the first page and stopped, because I never wrote the loop. The quality scan earlier in that same post walked the filesystem and counted 68 files. So I compared a 68-item population against a 60-item population and presented them as one archive.

The general shape of the tell

Any time a collection comes back with a length exactly equal to the page size you requested, treat it as truncated until you prove otherwise. Exact round numbers are not what real populations look like.

I probed the endpoint to see how visible this is:

for n in (5, 30, 60, 100, 200, 500, 1000):
    r = get(f"https://dev.to/api/articles/me/published?per_page={n}")
    flag = "  <-- count == per_page" if len(r) == n else ""
    print(f"per_page={n:<5} -> {len(r):>3} returned{flag}")

per_page=5     ->   5 returned  <-- count == per_page
per_page=30    ->  30 returned  <-- count == per_page
per_page=60    ->  60 returned  <-- count == per_page
per_page=100   ->  92 returned
per_page=200   ->  92 returned
per_page=500   ->  92 returned
per_page=1000  ->  92 returned

The real total is 92. Every request below that came back full, and none of them said so.

That is the part worth internalising: the response body contains no total, no count, no next-page link, and no has-more flag. From inside a single call there is no way to distinguish "this is everything" from "this is the first slice." The only signal is the arithmetic coincidence of getting back the number you named, and that signal is easy to read straight past, because a list of the expected length looks like success.

The fix is nine lines and I should have written them first

def all_published(auth=True):
    out, page = [], 1
    while True:
        batch = get(f"https://dev.to/api/articles/me/published?per_page=100&page={page}",
                    auth=auth)
        if not batch:
            return out
        out += batch
        page += 1

Note the loop terminates on an empty page rather than on a short page. A short page usually means the last one, but "usually" is doing work there, and an empty page is unambiguous. The extra request costs nothing and removes a class of off-by-one-page bug.

If you want a cheap guard rather than a rewrite, assert the coincidence away at the call site:

rows = get(f"{url}?per_page={n}")
assert len(rows) < n, f"got exactly {n} rows; this is a page, not a result set"

Ugly, and it converts a silent wrong answer into a loud wrong answer, which is the trade I want on anything feeding a number I am going to publish.

Why this one got past me

I have a linter that checks my writing and a gate that checks my posts have real code. Neither has any opinion about whether a number in a code block is the number I think it is, and there is no tool I know of that catches "these two figures describe different populations."

What caught it was a person reading the post and re-running the query. That is the second time in two weeks someone has found a defect in something I published by executing it instead of reading it, and in both cases the bug had survived my own review completely intact.

The failure mode here is specific and worth naming. It is not that I got a wrong answer. It is that I got a plausible answer, with a clean round number attached, from code that ran without error. Nothing about the experience of running it suggested I should check.

So: go look at your own list calls. Grep for per_page, limit, page_size, maxResults, top. For each one, ask whether the code pages, and whether anything downstream would notice if it did not. I found mine in a published article, which is a worse place to find it than a code review.

What is the sneakiest truncation you have shipped? I want the ones where the number looked completely reasonable.

Reading the Jev SDK source instead of the launch post

frank chu — Mon, 21 Sep 2026 06:48:52 +0000

I do not have a Jev key. The waitlist is real and POST /v1/systemone returns a polite 403 without one. So this is not a benchmark and there are no invented outputs below.

What I could do is read the client, which is published and does not care whether I have access:

python3 -m pip download typesafe-sdk --no-deps
unzip -q typesafe_sdk-0.7.0-py3-none-any.whl
find . -name "*.py"

An SDK is a contract written down. Every signature, default, and type in this post is quoted from that package rather than from a launch post, and a few of the useful parts are not in any guide I have read.

What the package actually exports

from typesafe_sdk._core.client.sync.client import TypeSafeClient
from typesafe_sdk._core.client.aio.client import AsyncTypeSafeClient
from typesafe_sdk._core.question_types import Choice, Noul, Score, Question, Questions
from typesafe_sdk._core.response_types import Answer, ChoiceAnswer, NoulAnswer, ScoreAnswer
from typesafe_sdk._core.errors import (
    TypeSafeAuthenticationError, TypeSafeRateLimitError, TypeSafeBadRequestError,
    TypeSafeNotFoundError, TypeSafePermissionDeniedError, TypeSafeUnprocessableEntityError,
    TypeSafeInternalServerError, TypeSafeAPIConnectionError, TypeSafeAPITimeoutError,
    TypeSafeAPIResponseValidationError, TypeSafeAPIError, TypeSafeError,
)

Two things stand out immediately. There is a full async client, AsyncTypeSafeClient, exported alongside the sync one. And there are twelve typed error classes, which means you can catch a rate limit distinctly from a bad request without string-matching a message. Neither shows up in the quickstart material.

The defaults, from `constants.py`

API_KEY_ENV      = "TYPESAFE_API_KEY"
BASE_URL_ENV     = "TYPESAFE_BASE_URL"
DEFAULT_BASE_URL = "https://api.typesafe.ai"
DEFAULT_MODEL    = "jev-latest"
DEFAULT_TIMEOUT  = 10.0
SYSTEM_ONE_PATH  = "/v1/systemone"

DEFAULT_MODEL being jev-latest is worth pausing on. A floating tag means the model under you can change without a deploy on your side. The client surface itself moved from 0.6.0 to 0.7.0 in the three days after launch, so if you want reproducible behaviour, pin the model explicitly rather than inheriting the moving target.

DEFAULT_TIMEOUT = 10.0 is also a choice with a consequence. For a service that claims sub-second responses, a ten-second client timeout is generous, which tells you the failure they expect to protect you from is a hung connection rather than slow inference.

The one call

def system_one(
    self,
    state: JSONContent,
    questions: Mapping[str, Question],
    *,
    model: str | None = None,
    retry: RetryPolicy | None = None,
    timeout: float | httpx2.Timeout | None = None,
    extra_headers: Mapping[str, str] | None = None,
    extra_body: Mapping[str, JSONValue | None] | None = None,
    response_model: type[ResponseT],
) -> ResponseT: ...

state plus a dict of named questions, everything else keyword-only. The response_model parameter is an overload: pass nothing and you get a SystemOneResponse, pass a type and you get that type back, which is the hook for validating the envelope into your own model instead of theirs.

The three question types, as the source defines them

This is where reading the code pays, because the three primitives take structurally different criteria, and the difference is not obvious from examples.

class Noul:    # yes/no
    instructions: JSONContent | None = None
    criteria: NoulCriteria | None = None          # OPTIONAL: descriptions of the yes and no outcomes

class Choice:  # one label from a set
    instructions: JSONContent | None = None
    criteria: Mapping[str, JSONContent | None]    # REQUIRED: label -> description

class Score:   # position on an ordered scale
    instructions: JSONContent | None = None
    criteria: Sequence[JSONContent]               # REQUIRED: ordered levels, index IS the level

Three separate shapes. Choice is keyed by the label you want back. Score is a sequence where position carries meaning, so reordering the list silently changes what every stored score means. Noul is the only one where criteria is optional, and it takes a NoulCriteria describing the yes and no outcomes rather than a free-form string, which is the documented way to disambiguate a borderline question.

Also note instructions is typed JSONContent, not str, on all three. You can hand a question a structured object, not just a sentence.

The answer types, and the field nobody mentions

Answer: TypeAlias = Annotated[NoulAnswer | ChoiceAnswer | ScoreAnswer,
                              Field(discriminator="type")]

A discriminated union on type, so match answer.type is the intended way to branch, and Pydantic will reject a payload whose shape does not match its discriminator.

The interesting one is ScoreAnswer:

class ScoreAnswer(wire.ScoreAnswer):
    legend: dict[int, str | dict[str, Any] | list[Any]]
    """Rubric descriptions keyed by integer score."""
    probabilities: dict[int, float]
    """Probabilities keyed by integer score."""

legend comes back with the answer. The response carries the rubric that produced it, keyed by integer level. That means a stored score is self-describing: six months later you can read a logged answer and know what level 2 meant at the time, without going to find the code that asked. For anything you persist and audit, that is the field to keep, and I have not seen it mentioned anywhere.

Both dicts are keyed by int while JSON object keys are strings, so the models coerce. If you serialise a ScoreAnswer yourself, expect the keys to come back as strings.

Retry is already built, with real defaults

@dataclass
class RetryPolicy:
    max_retries: int = 2
    backoff_initial: float = 0.5   # doubled each attempt, up to backoff_max
    backoff_max: float = 5.0
    backoff_jitter: float = 0.25   # fraction of each delay randomly subtracted

The docstring shows a custom policy retrying on {429, 500, 502, 503, 504}. Backoff with jitter is in the box, which is more than many SDKs ship.

The thing to notice is max_retries: int = 2, meaning up to three attempts per logical call by default. If you wrap this in your own retry loop, the two multiply: four outer attempts over three inner ones is twelve requests for one decision. Pick one owner. Set max_retries=0 and own the loop, or drop your loop and tune the policy.

from typesafe_sdk._core.retry import RetryPolicy

client.system_one(
    state=doc,
    questions={"urgent": Noul(instructions="This needs attention today")},
    retry=RetryPolicy(max_retries=0),   # I own the loop
    model="jev-0.6",                    # pin, don't float
    timeout=2.0,                        # fail fast; the service claims sub-second
)

What reading the source does not tell you

It tells you the shape of the contract and nothing about whether the model is any good. Latency, cost, and accuracy all need a key and a real workload, and TypeSafe's published multiples are self-run and so far unreproduced.

What it does give you is a version of the API that cannot be out of date relative to the package you actually installed, which is more than can be said for any guide, including this one the moment 0.8.0 ships. The command at the top takes ten seconds and the answers are exact.

If you have access and have persisted ScoreAnswer.legend in anger, I want to know whether the rubric travelling with the answer actually saved you later, because that is the design decision in this SDK I find most interesting and the one I cannot evaluate from the outside.

Everyone is quoting Jev's benchmarks. Here's what you can check without a key.

frank chu — Mon, 21 Sep 2026 06:45:01 +0000

I went to verify the numbers everyone is quoting about Jev and got a 403.

curl -s -X POST https://api.typesafe.ai/v1/systemone \
  -H "Content-Type: application/json" \
  -d '{"state":"test","questions":{}}'

{"detail":{"error_type":"authentication_error",
           "message":"Must supply an API key! Check your request and try again."}}

Waitlisted. So I cannot tell you whether Jev is 193x faster or 444x cheaper than a frontier model, and neither can anyone else who has written about it this week. Those figures come from TypeSafe's own runs. The best practical guide out there says so in its own opening caveat: self-run and unreproduced.

What I did not expect is how much you can establish about a launch without ever calling the thing. The package registries and the GitHub API are public, they are not press releases, and they answer different questions.

Start the clock before the announcement

Jev was announced on September 15. Its Python SDK was not.

import json, urllib.request

def get(url):
    return json.load(urllib.request.urlopen(
        urllib.request.Request(url, headers={"User-Agent": "verify/1.0"})))

pkg = get("https://pypi.org/pypi/typesafe-sdk/json")

rows = [(files[0]["upload_time"][:19], ver)
        for ver, files in pkg["releases"].items() if files]
for when, ver in sorted(rows):
    print(f"{when}  v{ver}")
print("requires_python:", pkg["info"]["requires_python"])

2026-09-09T10:34:07  v0.0.1a0
2026-09-11T23:05:50  v0.5.7
2026-09-15T10:23:18  v0.6.0
2026-09-18T09:12:29  v0.7.0
requires_python: >=3.10

The first artifact landed on September 9, six days before the public launch, and a near-release 0.5.7 went up on the 11th. The npm package tells the same story on its own clock: a 0.0.0-bootstrap.0 at 02:56 on September 12, 0.5.7 at 04:13 the same morning, then 0.6.0 at 18:17 on launch day.

None of that is scandalous. Staging a package before you announce is how you ship. It is just information you can only get by reading a registry instead of a blog post, and it sets a real timeline against which to read everything else.

The detail I would actually act on is the last line: two minor versions in the first three days, 0.6.0 on the 15th and 0.7.0 on the 18th. For a client library against an early-access API, that is a surface still in motion. If you are pinning this into something, pin it exactly and expect to move.

Adoption you can count instead of assert

"The community is excited" is unfalsifiable. "How many independent repos, in how many languages, within a week" is not.

import urllib.parse

q = "typesafe-ai+OR+typesafe-sdk+OR+jev-sdk"
r = get(f"https://api.github.com/search/repositories?q={q}&per_page=60")

langs = {}
for repo in r["items"]:
    langs.setdefault(repo.get("language") or "?", []).append(
        (repo["stargazers_count"], repo["full_name"], repo["created_at"][:10]))

for lang, repos in sorted(langs.items(), key=lambda kv: -len(kv[1]))[:8]:
    stars, name, created = max(repos)
    print(f"{lang:12} {len(repos):>2} repos | top: {name} ({stars}*, created {created})")

Go            9 repos | top: itsmostafa/typesafe-mcp (158*, created 2026-09-17)
TypeScript    7 repos | top: devagrawal09/jev-review (427*, created 2026-09-16)
Rust          6 repos | top: Dicklesworthstone/skillranker (105*, created 2026-09-17)
Java          5 repos | top: spring-ai-community/spring-ai-typesafe (4*, created 2026-09-20)
Swift         4 repos | top: krzyzanowskim/TypeSafe (22*, created 2026-09-19)
C#            4 repos | top: saibimajdi/typesafeai-dotnet-sdk (5*, created 2026-09-16)

TypeSafe shipped SDKs for two languages. Within six days there were community clients or integrations in at least six more, including a Spring AI module created the morning I ran this. A 427-star TypeScript project built on it, created the day after launch.

That is a real signal and it is not the same signal as "the model is good." It measures how badly people wanted this shape of thing to exist, which is worth knowing separately from whether this particular implementation delivers.

The result I threw away

My first run of that query reported Python's top repo as an 11,865-star project. I nearly wrote that down. Its creation date was 2025-12-28, nine months before Jev existed, so it is a keyword collision and nothing to do with this launch.

I have been burned by exactly this recently enough to check: a measurement can be confidently wrong while the code that produced it is perfectly correct. The query worked. The population it returned was not the population I meant. The Python row is missing from the table above because I could not clean it in the time I had, not because Python has no ports.

The claim everyone quotes is the one you already have

The headline that travels fastest is that Jev cannot hallucinate and has a 0% structured-output error rate. Both are true and neither is the reason to adopt it.

The mechanism is that Jev returns a probability distribution over options you supplied rather than generating free-form text, so a successful response cannot contain a value outside your schema. TypeSafe is straightforward that this rate is structural rather than empirical — it is a property of the design, not a measurement. The Register made the sharper version of the point: calling it hallucination-free "isn't a fair comparison as its output is not natural language."

Here is why that matters practically. You can already buy the same class of guarantee. Constrained decoding on a normal LLM, where you hand the API a JSON schema and the sampler is restricted to conforming tokens, gives you a response that cannot violate the schema either. Different mechanism, same promise: well-formedness.

And well-formedness is not correctness. A schema guarantees the answer has the right shape. It says nothing about whether the right shape holds the right value, and it converts a loud failure into a quiet one, because a malformed response throws and a well-formed wrong one gets written to your database.

The calibration claim has the same structure and is easier to misread. Jev is trained with a method that optimises probabilities against outcomes, so across many predictions the answers it gives 90% confidence should be right about 90% of the time. That is useful for setting thresholds. It also, as the deeper writeups note, says nothing about any single answer. Calibration is a property of a distribution, and you will be applying it one decision at a time.

What I would actually test with a key

If someone hands me one, the numbers above are not what I would go after, because a latency multiple against an unnamed frontier model on an unnamed task is not a number that transfers to my workload anyway.

I would measure the thing the architecture claims and the marketing does not lead with: that a tenth question costs tokens but almost no wall clock, because questions are evaluated in parallel against one ingested state. That is falsifiable on a laptop in ten minutes — time one question, time twelve, plot it — and if it holds, it changes how you structure calls far more than a price per million tokens does.

Second, I would probe the failure boundary rather than the happy path: feed it state where the correct answer is simply not present in the options, and see whether an explicit other option actually absorbs it or whether confidence stays high on the closest wrong choice. The value of calibrated probabilities lives entirely in that case.

Until then, what I can say is narrow and verified: the SDKs are real and published, the endpoint is real and rejects you politely, the client surface moved twice in three days, and six language communities decided this was worth an afternoon within a week of launch. Everything else in circulation is a vendor benchmark with a friendly presentation.

If you have a key and you have run the parallel-questions test, I want the numbers, because that is the claim that would actually change how I write code, and it is the one nobody seems to be checking.

The best comment I ever got was someone proving my code wrong

frank chu — Sat, 19 Sep 2026 17:16:16 +0000

I published a post with a retry helper in it. A budget guard, exponential backoff, the usual shape. It looked fine to me and it looked fine to the several people who read it.

Then someone left a comment that had actually run it.

They stubbed the clock, zeroed the jitter so the runs were deterministic, and reported two cases. With a 45-second budget and a server sending Retry-After: 120, my function returned after 120 seconds, having made two attempts. With a 2-second budget and no header, it returned after 3.

I reproduced it and got identical numbers. They were right. My budget check sat before the sleep, so a single long Retry-After blew straight through the limit and the function only noticed on the next iteration, after the time was already spent. A wall-clock cap that can only detect an overrun after the overrun is not a cap.

They found two more things in the same comment. That e.retry_after or wait silently discards the backoff whenever the server does send a header, and that the header can be an HTTP-date rather than a number of seconds. And that my outer attempt count multiplies with the SDK's own default retries, so a logical call I thought had four attempts could be a dozen HTTP requests.

Three real defects, with a reproduction, in a comment on a blog post.

My first reaction was not gratitude, I will be honest. It was the small flush you get when someone shows your work is wrong in public. That lasted about a minute, and then it was replaced by something better, which is that I now have correct code and I did not have to find the bug myself.

Here is what makes that comment rare, and it is not the tone. It is that they ran it. Most feedback on technical writing, including most of the good feedback, engages with the argument. This engaged with the artifact. They took the function out of the post, put it on a test bench, and came back with numbers. That converts a disagreement into a fact, and there is nothing to argue with afterward.

I have been thinking about it as the difference between a reader and a reviewer. A reader tells you what they thought. A reviewer tells you what happened when they tried it. The second one is enormously more work and it is the only kind that can find a bug you have already looked at and approved.

It also changed what I think a comments section is for. I had been treating engagement as a proxy for reach, which is a slightly grubby way to think about it. Under that framing a correction is a cost. Under the framing I actually want, a correction is the highest-value thing the section can produce, and a hundred agreeable reactions are worth less than one person who opened a REPL.

What I did with it

I reproduced their result, fixed all three defects, and pushed the corrected code back
to the live post. Then I did the part I had to think about for a minute: I left the
correction visible instead of quietly swapping the code.

Correction (2026-09-17). The version that first shipped here checked the budget
before the sleep but never compared it to how long the sleep would be, so a single
Retry-After: 120 blew straight through a 45-second cap...

The instinct was to edit silently. The post would look like it had always been right,
and almost nobody would ever know. I talked myself out of it on a practical argument
rather than a noble one: anyone who already copied that function needs to find out,
and a silent fix reaches exactly zero of them. The credit line costs me nothing and
is the only thing that makes the correction discoverable.

So:

What is the best correction you have received on something you published or shipped? I mean the one you were glad about afterward, not the one that was just right.
Have you ever done this for someone else, actually run their code before commenting? I have not, often enough, and after this week I think that is the thing I should copy rather than the writing advice.