DEV Community: Andrew Avery

OpenAI's Codex plugin told me my session transfer failed. It hadn't.

Andrew Avery — Tue, 25 Aug 2026 22:38:28 +0000

I was two hours into a refactor when Claude Code warned me I was running low on tokens. Annoying, but solvable: OpenAI ships an official plugin, codex-plugin-cc, with /codex:transfer for exactly this. Hand the live session to Codex, keep the conversation, keep working.

I ran it. It said:

Codex reported that the Claude import completed, but did not record
an imported thread.

So I opened Codex to start fresh — and the thread was already there. Full turn history. Continuable.

The transfer had worked. The plugin just didn't know it. This is the story of finding out why, and the three other things I broke my head on along the way.

The symptom, and the shape of it

Once I knew the message was wrong, the question was whether it was wrong sometimes or always. On my machine: always. Every transfer, every session size, every project — the thread appeared in Codex, and the plugin reported failure anyway.

Three issues described the same thing: #417, #514, and #513 — since closed, on 2026-08-11, as a duplicate of a newer report, #618. At the time I found them, none had a reply. I've since answered all three myself, with the fix below.

A consistent failure is a gift. It means the bug is structural, not a race — and structural bugs are findable by reading.

Following the state

Codex keeps its own records, and both of them turned out to be readable.

The first is a SQLite database at ~/.codex/state_5.sqlite, with a threads table carrying id, created_at, title, and cwd. That's the same store the Codex UI reads — ground truth for whether a thread got created. I opened it with a read-only URI so I could never corrupt live state while Codex was mid-write:

sqlite3.connect(f"file:{DB}?mode=ro", uri=True)

SELECT MAX(created_at) FROM threads before the import, the same query after, and the answer was unambiguous. A new thread, with the right title, every time the plugin claimed nothing had been recorded.

The second record is a JSON import ledger at ~/.codex/external_agent_session_imports.json. Each entry pairs the Claude transcript it came from with the Codex thread it produced: source_path, content_sha256, imported_thread_id, imported_at. Mine was in there too, correctly linked to the transcript I'd transferred.

So Codex had done everything right and written it all down. The failure was in the reading.

The root cause: four characters, and the one hiding behind them

The plugin's success check lives in scripts/lib/codex.mjs. Here it is as shipped in 1.0.6, unmodified:

function importedThreadIdForSource(sourcePath) {
  const ledgerPath = path.join(resolveCodexHome(), "external_agent_session_imports.json");
  if (!fs.existsSync(ledgerPath)) {
    return null;
  }
  const ledger = readJsonFile(ledgerPath);
  const canonicalSource = fs.realpathSync(sourcePath);
  const contentSha256 = sourceContentSha256(canonicalSource);
  const records = Array.isArray(ledger?.records) ? ledger.records : [];
  const match = records
    .filter(
      (record) =>
        record?.source_path === canonicalSource &&
        record?.content_sha256 === contentSha256 &&
        typeof record?.imported_thread_id === "string"
    )
    .at(-1);
  return match?.imported_thread_id ?? null;
}

The caller treats null as a hard error and throws the message I saw.

Now look at the two strings record?.source_path === canonicalSource has to reconcile.

Codex writes the ledger from the Rust side, storing the path in Windows extended-length form:

\\?\C:\Users\<you>\.claude\projects\...\session.jsonl

Node's fs.realpathSync returns the plain form:

C:\Users\<you>\.claude\projects\...\session.jsonl

Those are the same file. They are not the same string. === says no, .filter returns empty, .at(-1) gives undefined, the function returns null, and the plugin reports a failure that never happened. Four characters of prefix.

The \\?\ prefix is the win32 opt-out from MAX_PATH and normal path parsing. Rust's std::fs::canonicalize emits it; Node's realpathSync doesn't. Anything comparing the two needs to normalize first — nothing in codex.mjs does. I grepped the whole file for a normalizer before writing the patch.

Now the part I got wrong at first — the more interesting half.

Look at the filter again: two conditions, not one — the path has to match and content_sha256 has to match. My first read was that because the path never matches on Windows, the hash clause is simply unreachable — dead code, harmless, ignore it.

True about the evaluation. Exactly the wrong conclusion to draw from it. The path bug isn't hiding dead code — it's masking a second live bug, and the mask is the only reason you can't see it.

contentSha256 here is computed by reading the transcript at lookup time. The ledger's content_sha256 was computed by Codex when it imported. Between those two moments, the transcript keeps growing: the transfer is itself a turn in the live session, so Claude Code is still appending to the very file Codex just read. The hash you compute afterwards trails the content that was imported.

So there are two independent reasons the lookup returns null on Windows, and every transfer trips at least the first. On my machine, all 58 ledger records carried the \\?\ prefix, and 12 of 22 surviving source transcripts had already diverged in content from what was recorded. Fix the path and you don't fix the bug — you promote the second failure from theoretical to routine.

I didn't understand that when I wrote my patch. That's why the patch was only half a fix.

The path half is small and platform-honest:

function normalizeLedgerPath(value) {
  if (typeof value !== "string" || value === "") {
    return "";
  }
  let out = value;
  // \\?\UNC\server\share -> \\server\share
  if (out.startsWith("\\\\?\\UNC\\")) {
    out = `\\\\${out.slice(8)}`;
  } else if (out.startsWith("\\\\?\\")) {
    out = out.slice(4);
  }
  out = path.normalize(out);
  return process.platform === "win32" ? out.toLowerCase() : out;
}

And at the comparison, keeping the un-normalized real path for the file read so the hash still works on case-sensitive filesystems:

  const canonicalSource = fs.realpathSync(sourcePath);
  const canonicalKey = normalizeLedgerPath(canonicalSource);
  const contentSha256 = sourceContentSha256(canonicalSource);
  const records = Array.isArray(ledger?.records) ? ledger.records : [];
  const match = records
    .filter(
      (record) =>
        normalizeLedgerPath(record?.source_path) === canonicalKey &&
        record?.content_sha256 === contentSha256 &&
        typeof record?.imported_thread_id === "string"
    )
    .at(-1);

The UNC branch matters because \\?\UNC\server\share is the extended form of \\server\share, so stripping a flat four characters there would produce UNC\server\share — a path that resolves to nothing. Lowercasing is gated on win32 because case-folding paths on Linux is a correctness bug, not a convenience.

The fix that wasn't mine

I submitted the patch above upstream as openai/codex-plugin-cc#551. It didn't land: I deleted the fork a couple of days later, GitHub auto-closed the PR with it, and a PR whose head repo is gone can't be reopened. Two weeks of a maintainer's queue, gone with one housekeeping click.

But the deletion isn't the real problem with #551. It normalized the path and left the hash comparison sitting untouched right next to it. Merged as written, it would have turned a bug that fails every time into one that fails most of the time — harder to report, and arguably worse than what it replaced.

The patch worth watching is #469, and it isn't mine. @ayobamiseun opened it two weeks before mine, and it's the better design for a reason I'd missed entirely: instead of making the two comparisons succeed, it stops depending on them. It snapshots the ledger before the import and falls back to whatever Codex appended during it — sidestepping the path prefix and the hash divergence at once, and defusing trap two below as a bonus.

Worth sitting with for a second: I found the bug, described it correctly, explained a stranger's three-week-old issue — and still wrote the wrong fix, because I'd reasoned carefully about one condition in a two-condition filter. Diagnosis and remedy are different skills; being right about the first doesn't earn you the second.

As of writing, #469 is open, thinly reviewed, and main still compares those paths with ===.

Trap two: a re-transfer that legitimately creates nothing

No longer trusting the plugin's message, I built my own check: snapshot MAX(created_at), run the import, poll for a thread newer than the snapshot. Clean, evidence-based, and wrong in one case.

Transfer the same session twice with nothing new said in between, and Codex creates no new thread. It hashes the transcript content — that's what content_sha256 in the ledger is for — recognizes the re-import, and reuses the existing thread. Correct behavior. My newest-thread-after-snapshot detector polls until it times out and calls that a failure.

So I added a second lookup for the timeout case: find the source path in the ledger (normalizing the prefix, same as above) and reuse the thread it already points at, saying so out loud. Note what that lookup keys on — the path alone, never the hash. That was luck, not foresight, at the time — but it's why my wrapper survives the divergence problem that sank my own upstream patch. Unchanged transcript gives you the same thread; changed transcript gives you a new one. Both are success, and they read differently to the user.

The general shape of this mistake is worth keeping: I had built a detector for state change when the thing I actually wanted to know was state.

Trap three: the app-container PATH trap

This one produced the most misleading error message of the batch.

I had installed the Codex CLI with npm i -g @openai/codex from a shell inside a packaged (MSIX) desktop app. Inside that container, codex resolved fine. Every check I ran passed.

Then the transfer printed a codex resume command, a terminal opened to run it, and:

codex : The term 'codex' is not recognized as the name of a cmdlet,
function, script file, or operable program.

MSIX packages get a virtualized view of the file system. A global npm install from inside the container lands in %LOCALAPPDATA%\Packages\<app>\LocalCache\Roaming\npm\..., and a shell spawned for the user runs outside the container. For that shell, the PATH entry isn't there and, depending on virtualization, neither are the files. The CLI existed and did not exist, depending on who was asking.

The fix is to stop emitting a bare command name that another process has to resolve. My resolver prefers a genuinely global install when there is one, and otherwise embeds an absolute path to the standalone vendored codex.exe in the container's LocalCache backing store — a real directory on disk, readable by any process, with no Node on PATH required.

Trap four: two skill roots, every skill listed twice

Smaller, but it cost me a confused afternoon. Codex discovers skills from both ~/.codex/skills and ~/.agents/skills, the cross-tool Agent Skills root. Put a skill in both — easy to do accidentally, for instance by symlinking one into the other — and it appears twice in the picker. Three copies, three listings.

The useful discovery was the verification command rather than the bug: codex debug prompt-input renders the prompt the model actually sees. Skill list, instructions, all of it. That turns "I think Codex can see my skills" into something checkable. The skill index also caches until the surface restarts, which is its own source of phantom results.

A bonus: the deep-link routes

Once I reliably have a thread id, printing a resume command stops being the best ending. So I went looking for whether Codex could open directly on a thread.

Two routes, both found by inspection rather than documentation. The OpenAI.Codex package manifest declares a codex URL protocol, and the app bundle contains codex://threads/ route strings — so codex://threads/<thread-id> opens the desktop app on that conversation. Separately, the VS Code extension's handleUri forwards the URI path into its webview router, and that router serves local threads at /local/<id> — so vscode://openai.chatgpt/local/<thread-id> opens the panel on it.

Both work. Neither is documented, which is exactly the caveat you should attach to them.

One honest limitation: a URL can't carry a model choice. The -m flag applies to the terminal resume path; the desktop app and the VS Code panel each use their own in-UI model selector.

Where this ended up

The workarounds became a Claude Code plugin: claude-codex-bridge, MIT. It wraps the official plugin's importer rather than replacing it, verifies the outcome against Codex's own state instead of the plugin's message, and deep-links you into the thread.

claude plugin marketplace add AndrewAvery7/claude-codex-bridge
claude plugin install codex-bridge@claude-codex-bridge

Then /to-codex in any session.

The limitations, plainly:

Cross-platform, with one honest asterisk. The engine is a single Python file and CI runs its test suite on Windows, macOS and Linux. The complete flow is verified end-to-end on Windows; the macOS and Linux paths (the protocol handler and terminal launch) are implemented and CI-checked but not yet confirmed against a real Codex install on those platforms. Reports either way are welcome.
It depends on undocumented Codex internals: the state_5.sqlite threads schema, the import ledger format, and both deep-link routes. A Codex update could move any of them. That is why every transfer also prints the plain codex resume command, which is the fallback that survives.
Verified against Codex CLI 0.145, the VS Code extension 26.721, and codex-plugin-cc 1.0.6, on Windows 11 with PowerShell 5.1. Other combinations are untested by me.

The thing I'd take away from this one isn't the prefix. It's that a tool telling you it failed is a claim, not evidence — and when the tool is manipulating state you can read yourself, checking the state is usually cheaper than believing the message.

My AI said the PDF was empty. The PDF was not empty.

Andrew Avery — Wed, 19 Aug 2026 14:51:30 +0000

I asked Claude Code to pull the key dates out of a PDF I had saved from a
webpage. It came back immediately:

The document appears to be empty — it contains no extractable text.

Which was strange, because I had the PDF open on my other monitor and it was
four and a half megabytes of perfectly legible text.

The interesting part is not that it was wrong. The interesting part is that
nothing had failed. Every component in that chain did exactly what it was
designed to do, reported success, and stacking those successes together
produced a lie I believed.

Reproducing it

I was preprocessing documents with markitdown,
Microsoft's file-to-markdown converter, so I ran it by hand:

$ markitdown screenshot.pdf -o out.md
$ echo $?
0
$ wc -c out.md
0 out.md

Exit code 0. Zero-byte output file. No warning, no stderr, nothing on the
console at all.

My integration did what integrations do: checked the return code, saw success,
cached the result, and handed the model a path to a file with nothing in it. The
model read the file, found nothing in it, and told me the document was empty.
From its position that was a reasonable conclusion. It had been given an empty
file and told the conversion worked.

This is not a bug

My first instinct was to file an issue. I am glad I did not, because markitdown
is behaving correctly and I would have been publicly wrong.

The PDF was a full-page browser screenshot exported to PDF. It contains raster
images and no text layer whatsoever. pdfminer reports 0 characters, and so
does PyMuPDF when you ask it. markitdown's PDF backend extracts embedded text
and does not OCR — that is a documented design decision, not an oversight.

So there was genuinely nothing to find. And finding nothing is not an error.
A converter that exited non-zero every time a document happened to be empty
would be wrong in a much more annoying way.

The bug is somewhere else entirely, and it is worth naming precisely:

The bug is in every integration that treats exit code as evidence of yield.

Exit code answers "did the process complete?" I was reading it as an answer to
"did we get the text?" Those are different questions, and for a document
converter meeting a scanned page they have different answers. Mine was one of
those integrations. Probably yours is too — pdftotext, pandoc and most
extraction tooling have the same shape, because they should.

That reframing is the whole story. Everything below is consequences.

The rule

Never measure success by exit code. Measure it by yield.

Simple to say. The trouble starts immediately, because "measure the yield" needs
a threshold, and thresholds are where honest engineering goes to become
arbitrary. Anyone can write if len(text) == 0: fail. That catches the
screenshot. It does not catch the case that actually cost me time.

The case that is actually hard

A course completion certificate. One page, a decorative graphic, and a title
line rendered as real text. It converts to this:

Certificate of Completion

Thirty-nine characters. Not zero. It sails through an emptiness check, gets
cached as a successful conversion, and the model dutifully reports that your
certificate says "Certificate of Completion" and nothing else — which is, again,
technically what it was given.

That is the shape of the real problem. The fully-empty case is easy and any
check catches it. The expensive failures are the near-misses: a certificate,
a slide deck exported as page images with a footer on every slide, a contract
scanned at an angle with a header that happened to OCR at some point. They all
return some characters.

So the question becomes: how do you tell "extraction failed" from "this document
is legitimately short"?

Byte count is the wrong instrument

The obvious move is a minimum size — reject anything under, say, 500 bytes. It
does not work, and the reason it does not work is worth being precise about:
raw byte count conflates document length with extraction quality.

800 bytes is a complete and correct conversion of a one-page memo. 800 bytes
from a 200-page report is a catastrophic extraction failure. The same number
means opposite things and the measure cannot distinguish them.

What you want is density, not volume. Characters per page normalises
document length away and leaves only the question you actually care about: on
each page, did we recover a page's worth of text?

The measurement

I ran four real documents — a deliberately mixed set, including the near-miss
that started this:

Document	Pages	Chars	Chars/page
Webpage screenshot saved as PDF	1	0	0
Course certificate (graphic + title)	1	39	39
Two-page text document	2	1,864	932
Twenty-page slide deck	20	13,289	664

Two populations. No overlap. An order of magnitude between them.

That gap is the finding. It is not a subtle statistical separation requiring a
tuned classifier — a text extractor meeting a page of text produces hundreds of
characters per page, and a text extractor meeting a picture produces tens or
zero. There is nothing in between, because there is no such thing as a document
that is 40% made of text.

A threshold of 100 characters per page sits in that gap with roughly a 6×
margin on both sides. The certificate is 2.5× below it; the sparsest real
document is 6.6× above it. Small variations in document style — bigger fonts,
more whitespace, a title slide — cannot cross a gap that wide.

That margin is the entire justification for the number. I would not defend 100
as optimal. I would defend it as comfortably inside a gap where nothing lives,
which is a much better property for a threshold than being finely tuned.

Which direction to fail

Every threshold gets some cases wrong. What you get to choose is which way.

A false positive — a genuinely sparse PDF gets flagged as image-based — means
the model reads the original document with vision instead. That costs more
tokens. It loses nothing. The user gets a correct answer at a higher price.

A false negative — an empty conversion is presented as real — means the model
confidently reports an empty document. The content is lost entirely and the
user has no signal anything went wrong. That is the failure I started with.

Those costs are not remotely symmetric, so the threshold is deliberately set to
prefer the first. When it is wrong, it is wrong in the direction that costs
money instead of the direction that costs truth.

One more asymmetry, discovered by getting it wrong: the density test has to be
PDF-only. Applying it to every format looks consistent and is a mistake. A
one-line email, a ten-second voice memo, a spreadsheet with four cells — all
convert correctly to very little text. Rejecting those would discard good
conversions to guard against a failure mode they cannot have. Only PDFs carry
the specific hazard of a text extractor meeting a picture. Everything else is
rejected only at literally zero.

Three more traps in the same family

Once you start looking for "success that isn't", it turns out to be a genre.

Caching a bad result is worse than producing one. My first version cached by
modification time. A zero-byte conversion was therefore served for every future
reference to that document — permanently, with no retry, even after I had fixed
the underlying cause. A transient failure had been promoted to a permanent one
by the cache. Conversions are now re-graded before reuse, unusable results are
deleted rather than stored, and zero-byte artifacts from older versions get
swept on the next run so an upgrade heals the cache without anyone intervening.

Silence is a failure mode. The hook must never block a prompt, so unexpected
errors exit 0 quietly. That is correct for almost everything and catastrophic
for one case: if markitdown is not installed at all, a silent no-op is
indistinguishable from "this document is empty" — the exact failure the whole
project exists to prevent. Missing dependencies are now the one error reported
loudly, with the install command.

The bug that produced no output at all. On Windows, PowerShell 5.1 prepends
a UTF-8 BOM when piping to a native command. json.load raises on the BOM. That
exception hit the never-block-a-prompt handler and was swallowed, so the hook
did nothing, silently, on every prompt, on an entire platform. Two invisible
failure modes composing into a third. The input parsing is BOM-tolerant now, but
the lesson is the one above: an error handler that guarantees silence will
eventually guarantee it for something you needed to hear about.

A licence is a dependency decision. PyMuPDF reads some PDFs pdfminer cannot,
so it is used when present — but it is never required. It is dual-licensed
AGPL-3.0/commercial, which does not belong in the dependency set of an MIT
project. Page counting, which the grading needs, uses pdfminer instead, which
markitdown already depends on. The core path adds no dependency and no copyleft.

What it looks like when it works

When a conversion recovers real content, the model gets a pointer rather than
the text:

[markitdown] /path/report.pdf was converted to markdown at
~/.claude/markitdown-cache/report-a1b2c3d4.md (14973 bytes, 304 lines,
14472 chars, 20 page(s)). If this document's content is needed, Read or
Grep the .md file (not the original).

The pointer matters more than it first looks. Prompts mention documents
speculatively — "compare these three reports" may only genuinely need one of
them. Loading all three into context costs those tokens on every subsequent turn
of the conversation, used or not. A pointer costs about 400 characters and is
paid once. The model reads or greps the file, with offset and limit for big
ones, only if the content turns out to matter.

And when extraction recovers nothing, no file is written, nothing is cached, and
the model is told the truth:

[markitdown] NO USABLE TEXT extracted from /path/screenshot.pdf (no text
extracted). This is an image-based/scanned PDF -- text extraction cannot
see into it and no .md was written. Read the ORIGINAL file natively with
the Read tool (use the `pages` parameter for long PDFs); Claude's vision
can read it. Do NOT report the document as empty.

That last line is there because without it the model does exactly that.

The part that generalises

None of this is really about PDFs.

Any pipeline that hands one tool's output to another has this hazard the moment
the first tool can succeed at doing nothing. Exit codes are a claim about
process completion. They were never a claim about yield, and we have all been
reading them as one because for most tools, most of the time, the two happen to
coincide.

The habit worth taking from it is small and cheap: after any extraction step,
measure what came back and decide whether it is plausible for the input. Not
"did it error" — errors are the easy case, they announce themselves. The
dangerous outcome is the one that returns 0, writes a file, and is empty.

And when it is empty, say so. A downstream model handed an empty file will not
wonder whether something went wrong. It will tell your user their document is
blank, in the same assured voice it uses when it is right.

The hook I built out of this is MIT and runs on Windows, macOS and Linux:
claude-markitdown-hook.
The measurements behind the threshold, including the fixtures used to calibrate
it, are in docs/DESIGN.md.

What LLM Cost Calculators Get Wrong

Andrew Avery — Wed, 12 Aug 2026 21:04:19 +0000

Is the number these calculators show you actually right? Not whether the
model is in the catalog — whether the arithmetic behind that number matches
what a provider will actually bill. Below are nine specific ways it usually
doesn't, one story about how a wrong-but-plausible number hid from the
people who built the tool, and what it takes to catch either failure before
a reader does.

Every figure below carries a source and a date, the same way the underlying
catalog does. Don't take our word for it — the same provenance is one
request away:

curl -s https://promptspend.dev/v1/models/gpt-5

The response carries provenance.source, provenance.lastVerified, and
provenance.verifiedUrl on every model. If a number in this article and the
number that endpoint returns ever disagree, the endpoint is right — this
article just goes stale first.

1. Output is priced separately

Every major provider charges more for what a model writes than for what you
send it. Generation is the expensive part, not comprehension. Claude Sonnet
4.6 bills input at $3 and output at $15 per million tokens (Anthropic,
verified 2026-08-03) — output costs five times input, not some blended
average of the two. A calculator that reports one "per-token" figure for a
model is wrong before you've typed a prompt: it's picking one side of a 5x
gap and hoping your workload matches it.

2. Chat history compounds

A multi-turn conversation doesn't cost turn-count times one-turn-cost.
Every provider requires the full conversation to be resent as input on each
turn, because none of them hold state for you between requests. Take a
support exchange running 200 tokens per turn: a flat estimate for ten turns
says 2,000 tokens. What actually gets billed — because turn N resends
turns 1 through N-1 in full — is 9,000 tokens of resent history before a
single new word of turn ten is counted. That's more than 4x the flat
estimate, and it's before adding what each turn actually contributes. The
shape is quadratic in turn count. A ten-message support chat and a
ninety-message one don't scale the same way, and a calculator that
multiplies instead of summing gets more wrong the longer the conversation
runs.

3. Tokenizers differ per family

"How many tokens is this prompt" doesn't have one answer across vendors,
because it isn't one algorithm. OpenAI-family models use a published,
runnable tokenizer (o200k_base), so an exact count is possible
client-side. Anthropic doesn't ship a public tokenizer at all — counting
happens server-side — so anything outside OpenAI's family is necessarily a
calibrated estimate: roughly 3.6 characters per token for English prose, 1.5
for CJK text, tuned against real samples rather than the model's own code.
A calculator that applies one ratio to every model it lists is applying an
OpenAI-shaped assumption to providers that never agreed to it, and it won't
say which of its numbers are exact and which are guesses, because it isn't
tracking the difference itself.

4. Caching is not free

"Prompt caching" sounds like a discount with no downside. The read side is:
cached input on GPT-5.6 Terra runs $0.20 per million against a $2 standard
rate, a 10x saving. The write side is where that framing falls apart. The
first time a prefix enters the cache, both OpenAI and Anthropic charge
more than the standard input rate to store it — 1.25x across every model
we've checked, from Claude Haiku 4.5 ($1.25 cache-write against a $1 input
rate, Anthropic, verified 2026-08-01) to GPT-5.6 Terra ($2.50 against $2,
OpenAI, verified 2026-08-01). A calculator that only ever shows you the 10x
read discount is showing you the number that makes the feature look best,
not the number your first request of the day will actually cost.

5. Long context costs more

Cross a provider's context threshold and the entire request reprices, not
just the tokens past the line. OpenAI's GPT-5.4 bills $2.50 input / $15
output per million tokens under 272,000 input tokens, and $5 / $22.50 — 2x
input, 1.5x output — the instant a request crosses that line (OpenAI,
verified 2026-08-03). That threshold applies per request, which means a
conversation can start under the tier and cross it mid-thread as history
accumulates: the same chat that priced cheaply on turn three can silently
double its input rate by turn twelve, and a calculator that quotes one flat
rate per model has no way to tell you when that happens.

6. Reasoning tokens are billable

What you see as "the answer" and what you're billed for are not the same
text. Reasoning models like OpenAI's o3 (billed at $2 input / $8 output per
million tokens, OpenAI, verified 2026-08-03) generate hidden thinking tokens
before the visible response, and those tokens are billed as output whether
or not you ever see them. A calculator that counts only the words on screen
is pricing a request that doesn't exist — the invoice includes tokens the
interface never showed you, and the gap between "visible answer" and
"billed output" is exactly the part a screenshot can't reveal.

7. Promotional pricing expires

Introductory rates are real, and they're also temporary. A calculator that
hard-codes them stops being honest the day the window closes. As of this
writing, Claude Sonnet 5 is priced at $2 input / $10 output through
2026-08-31 (Anthropic, verified 2026-08-01) — the standard rate that takes
over the day after is $3 / $15. An estimate built today and still quoted in
September is quoting a rate that no longer exists, unless the tool doing the
estimating knows the window has a date and checks it against the date it's
actually being asked. Most don't carry a date at all; they carry the number
that was true when someone last edited the page.

8. Assumptions are visible

Every calculator makes assumptions the provider hasn't published: what share
of a conversation is a cache hit, how large the hidden reasoning share
usually runs, whether the batch API is in play. Those assumptions decide the
number on screen as much as the published rates do. The difference is
whether they're printed next to the answer or buried in a methodology page
nobody reads — a promotional rate in effect, a reasoning multiplier applied,
a batch discount assumed, each stated in plain language under the total
rather than folded silently into it. A number with its assumptions attached
can be argued with. A number without them can only be believed or not.

9. Impossible scenarios are named

GPT-5 has a 272,000-token context window and a 128,000-token output ceiling
(OpenAI, verified 2026-08-03). Paste a 300,000-token document in and the
request can't be sent at all — it's 28,000 tokens over the line before
generation even starts. A calculator that just multiplies token count by
rate will return a confident dollar figure for a request the API would
reject outright, because multiplication doesn't know the difference between
an expensive request and an impossible one. The honest answer to "what would
this cost" is sometimes "nothing, because it can't happen" — and that answer
requires checking the ceiling, not just the rate.

The proof nobody asked for

Everything above is a claim about pricing engines in the abstract. Here's
what it looked like to get one wrong in public, because we did, twice, on
our own product.

Building an editor extension that prices the model on the line of code that
calls it meant running the same cost engine outside a browser, inside VS
Code, for the first time. Six defects came out of that work. Every one of
them passed 735 tests and six CI gates cleanly — verified, all green — and
two of them reached the Marketplace before a person looked at the screen and
noticed something was off.

A call with no token cap of its own silently inherited the cap from the
call above it, and priced it at that neighbor's rate. The extension scans
a file for model IDs and their nearby max_tokens values; the backward
search for "which cap belongs to this call" stopped at the previous model
ID. That sounds like the right boundary. It isn't — a cap written between
two model IDs actually belongs to the earlier call's forward-looking window.
So a call that never specified an output ceiling displayed one anyway: a
real number, correctly multiplied, describing a request nobody had written.

Exact token counting quietly degraded to an estimate, and reported a
number either way, so nothing on screen said it had happened. The tokenizer
ships as four external files rather than bundled into the extension — bundle
it, and the download goes from 19.7 KB to 3.4 MB, because you'd be shipping
a full rank table to every editor window whether or not anyone ran the
estimate command. The packaging step that decides which files actually ship
diverged from the module resolution that looks for them at runtime, so a
release could pass every test and still leave one required file behind.
When that happens, the code catches the failed import and falls back to the
same calibrated-ratio estimate section 3 describes above — silently,
correctly formatted, indistinguishable on screen from an exact count. Not an
error. Not a crash. A plausible number, standing in for a real one, with
nothing to tell you which you were looking at.

Both defects share that shape, and so does every row above: the failure
mode of a pricing tool is not crashing. A crash is loud, and someone fixes
it that afternoon. The failure mode is returning a real-looking figure —
clean formatting, correct arithmetic, plausible magnitude — for a scenario
that isn't the one you asked about. Nothing about a wrong-but-plausible
number announces itself. The only defense is being able to check where it
came from, which is the same defense this article has been asking for from
every calculator in the category, including — it turns out — our own.

Monitoring the promise, not the machinery

Fixing those two defects made the extension correct on the day it shipped.
It says nothing about whether the published catalog stays correct on every
day after. Those are different failure modes, and the second one has its own
watcher: a daily job that fetches promptspend.com/data/pricing.json — the
live site, not the repository — four hours after the sync runs, and opens an
alert if the catalog it finds is more than two days old.

The live site, deliberately, not the repository. Between a price moving at
a vendor and a reader seeing the update, there are four places the chain can
break: the sync can error, the sync can raise a review flag that sits
unmerged, the deploy can fail after a clean sync, or the CDN can serve a
stale build. Checking the file in git catches exactly one of those four.
Fetching what the site actually serves catches all of them, because it's
asking the question from where the answer matters — the reader's browser,
not a folder on a build server.

Monitor the promise you made to the reader, not the machinery you built to
keep it.

Every calculator in this category, including this one, is a pile of
assumptions wearing a clean interface. The nine on this page are the ones
that quietly favor a low number. The tenth is what happens when the people
who wrote the code stop trusting their own tests to catch the eleventh —
and start checking the thing they actually promised, instead of the thing
they built to promise it.