DEV Community: Andrew Avery

My AI said the PDF was empty. The PDF was not empty.

Andrew Avery — Wed, 19 Aug 2026 14:51:30 +0000

I asked Claude Code to pull the key dates out of a PDF I had saved from a
webpage. It came back immediately:

The document appears to be empty — it contains no extractable text.

Which was strange, because I had the PDF open on my other monitor and it was
four and a half megabytes of perfectly legible text.

The interesting part is not that it was wrong. The interesting part is that
nothing had failed. Every component in that chain did exactly what it was
designed to do, reported success, and stacking those successes together
produced a lie I believed.

Reproducing it

I was preprocessing documents with markitdown,
Microsoft's file-to-markdown converter, so I ran it by hand:

$ markitdown screenshot.pdf -o out.md
$ echo $?
0
$ wc -c out.md
0 out.md

Exit code 0. Zero-byte output file. No warning, no stderr, nothing on the
console at all.

My integration did what integrations do: checked the return code, saw success,
cached the result, and handed the model a path to a file with nothing in it. The
model read the file, found nothing in it, and told me the document was empty.
From its position that was a reasonable conclusion. It had been given an empty
file and told the conversion worked.

This is not a bug

My first instinct was to file an issue. I am glad I did not, because markitdown
is behaving correctly and I would have been publicly wrong.

The PDF was a full-page browser screenshot exported to PDF. It contains raster
images and no text layer whatsoever. pdfminer reports 0 characters, and so
does PyMuPDF when you ask it. markitdown's PDF backend extracts embedded text
and does not OCR — that is a documented design decision, not an oversight.

So there was genuinely nothing to find. And finding nothing is not an error.
A converter that exited non-zero every time a document happened to be empty
would be wrong in a much more annoying way.

The bug is somewhere else entirely, and it is worth naming precisely:

The bug is in every integration that treats exit code as evidence of yield.

Exit code answers "did the process complete?" I was reading it as an answer to
"did we get the text?" Those are different questions, and for a document
converter meeting a scanned page they have different answers. Mine was one of
those integrations. Probably yours is too — pdftotext, pandoc and most
extraction tooling have the same shape, because they should.

That reframing is the whole story. Everything below is consequences.

The rule

Never measure success by exit code. Measure it by yield.

Simple to say. The trouble starts immediately, because "measure the yield" needs
a threshold, and thresholds are where honest engineering goes to become
arbitrary. Anyone can write if len(text) == 0: fail. That catches the
screenshot. It does not catch the case that actually cost me time.

The case that is actually hard

A course completion certificate. One page, a decorative graphic, and a title
line rendered as real text. It converts to this:

Certificate of Completion

Thirty-nine characters. Not zero. It sails through an emptiness check, gets
cached as a successful conversion, and the model dutifully reports that your
certificate says "Certificate of Completion" and nothing else — which is, again,
technically what it was given.

That is the shape of the real problem. The fully-empty case is easy and any
check catches it. The expensive failures are the near-misses: a certificate,
a slide deck exported as page images with a footer on every slide, a contract
scanned at an angle with a header that happened to OCR at some point. They all
return some characters.

So the question becomes: how do you tell "extraction failed" from "this document
is legitimately short"?

Byte count is the wrong instrument

The obvious move is a minimum size — reject anything under, say, 500 bytes. It
does not work, and the reason it does not work is worth being precise about:
raw byte count conflates document length with extraction quality.

800 bytes is a complete and correct conversion of a one-page memo. 800 bytes
from a 200-page report is a catastrophic extraction failure. The same number
means opposite things and the measure cannot distinguish them.

What you want is density, not volume. Characters per page normalises
document length away and leaves only the question you actually care about: on
each page, did we recover a page's worth of text?

The measurement

I ran four real documents — a deliberately mixed set, including the near-miss
that started this:

Document	Pages	Chars	Chars/page
Webpage screenshot saved as PDF	1	0	0
Course certificate (graphic + title)	1	39	39
Two-page text document	2	1,864	932
Twenty-page slide deck	20	13,289	664

Two populations. No overlap. An order of magnitude between them.

That gap is the finding. It is not a subtle statistical separation requiring a
tuned classifier — a text extractor meeting a page of text produces hundreds of
characters per page, and a text extractor meeting a picture produces tens or
zero. There is nothing in between, because there is no such thing as a document
that is 40% made of text.

A threshold of 100 characters per page sits in that gap with roughly a 6×
margin on both sides. The certificate is 2.5× below it; the sparsest real
document is 6.6× above it. Small variations in document style — bigger fonts,
more whitespace, a title slide — cannot cross a gap that wide.

That margin is the entire justification for the number. I would not defend 100
as optimal. I would defend it as comfortably inside a gap where nothing lives,
which is a much better property for a threshold than being finely tuned.

Which direction to fail

Every threshold gets some cases wrong. What you get to choose is which way.

A false positive — a genuinely sparse PDF gets flagged as image-based — means
the model reads the original document with vision instead. That costs more
tokens. It loses nothing. The user gets a correct answer at a higher price.

A false negative — an empty conversion is presented as real — means the model
confidently reports an empty document. The content is lost entirely and the
user has no signal anything went wrong. That is the failure I started with.

Those costs are not remotely symmetric, so the threshold is deliberately set to
prefer the first. When it is wrong, it is wrong in the direction that costs
money instead of the direction that costs truth.

One more asymmetry, discovered by getting it wrong: the density test has to be
PDF-only. Applying it to every format looks consistent and is a mistake. A
one-line email, a ten-second voice memo, a spreadsheet with four cells — all
convert correctly to very little text. Rejecting those would discard good
conversions to guard against a failure mode they cannot have. Only PDFs carry
the specific hazard of a text extractor meeting a picture. Everything else is
rejected only at literally zero.

Three more traps in the same family

Once you start looking for "success that isn't", it turns out to be a genre.

Caching a bad result is worse than producing one. My first version cached by
modification time. A zero-byte conversion was therefore served for every future
reference to that document — permanently, with no retry, even after I had fixed
the underlying cause. A transient failure had been promoted to a permanent one
by the cache. Conversions are now re-graded before reuse, unusable results are
deleted rather than stored, and zero-byte artifacts from older versions get
swept on the next run so an upgrade heals the cache without anyone intervening.

Silence is a failure mode. The hook must never block a prompt, so unexpected
errors exit 0 quietly. That is correct for almost everything and catastrophic
for one case: if markitdown is not installed at all, a silent no-op is
indistinguishable from "this document is empty" — the exact failure the whole
project exists to prevent. Missing dependencies are now the one error reported
loudly, with the install command.

The bug that produced no output at all. On Windows, PowerShell 5.1 prepends
a UTF-8 BOM when piping to a native command. json.load raises on the BOM. That
exception hit the never-block-a-prompt handler and was swallowed, so the hook
did nothing, silently, on every prompt, on an entire platform. Two invisible
failure modes composing into a third. The input parsing is BOM-tolerant now, but
the lesson is the one above: an error handler that guarantees silence will
eventually guarantee it for something you needed to hear about.

A licence is a dependency decision. PyMuPDF reads some PDFs pdfminer cannot,
so it is used when present — but it is never required. It is dual-licensed
AGPL-3.0/commercial, which does not belong in the dependency set of an MIT
project. Page counting, which the grading needs, uses pdfminer instead, which
markitdown already depends on. The core path adds no dependency and no copyleft.

What it looks like when it works

When a conversion recovers real content, the model gets a pointer rather than
the text:

[markitdown] /path/report.pdf was converted to markdown at
~/.claude/markitdown-cache/report-a1b2c3d4.md (14973 bytes, 304 lines,
14472 chars, 20 page(s)). If this document's content is needed, Read or
Grep the .md file (not the original).

The pointer matters more than it first looks. Prompts mention documents
speculatively — "compare these three reports" may only genuinely need one of
them. Loading all three into context costs those tokens on every subsequent turn
of the conversation, used or not. A pointer costs about 400 characters and is
paid once. The model reads or greps the file, with offset and limit for big
ones, only if the content turns out to matter.

And when extraction recovers nothing, no file is written, nothing is cached, and
the model is told the truth:

[markitdown] NO USABLE TEXT extracted from /path/screenshot.pdf (no text
extracted). This is an image-based/scanned PDF -- text extraction cannot
see into it and no .md was written. Read the ORIGINAL file natively with
the Read tool (use the `pages` parameter for long PDFs); Claude's vision
can read it. Do NOT report the document as empty.

That last line is there because without it the model does exactly that.

The part that generalises

None of this is really about PDFs.

Any pipeline that hands one tool's output to another has this hazard the moment
the first tool can succeed at doing nothing. Exit codes are a claim about
process completion. They were never a claim about yield, and we have all been
reading them as one because for most tools, most of the time, the two happen to
coincide.

The habit worth taking from it is small and cheap: after any extraction step,
measure what came back and decide whether it is plausible for the input. Not
"did it error" — errors are the easy case, they announce themselves. The
dangerous outcome is the one that returns 0, writes a file, and is empty.

And when it is empty, say so. A downstream model handed an empty file will not
wonder whether something went wrong. It will tell your user their document is
blank, in the same assured voice it uses when it is right.

The hook I built out of this is MIT and runs on Windows, macOS and Linux:
claude-markitdown-hook.
The measurements behind the threshold, including the fixtures used to calibrate
it, are in docs/DESIGN.md.

What LLM Cost Calculators Get Wrong

Andrew Avery — Wed, 12 Aug 2026 21:04:19 +0000

Is the number these calculators show you actually right? Not whether the
model is in the catalog — whether the arithmetic behind that number matches
what a provider will actually bill. Below are nine specific ways it usually
doesn't, one story about how a wrong-but-plausible number hid from the
people who built the tool, and what it takes to catch either failure before
a reader does.

Every figure below carries a source and a date, the same way the underlying
catalog does. Don't take our word for it — the same provenance is one
request away:

curl -s https://promptspend.dev/v1/models/gpt-5

The response carries provenance.source, provenance.lastVerified, and
provenance.verifiedUrl on every model. If a number in this article and the
number that endpoint returns ever disagree, the endpoint is right — this
article just goes stale first.

1. Output is priced separately

Every major provider charges more for what a model writes than for what you
send it. Generation is the expensive part, not comprehension. Claude Sonnet
4.6 bills input at $3 and output at $15 per million tokens (Anthropic,
verified 2026-08-03) — output costs five times input, not some blended
average of the two. A calculator that reports one "per-token" figure for a
model is wrong before you've typed a prompt: it's picking one side of a 5x
gap and hoping your workload matches it.

2. Chat history compounds

A multi-turn conversation doesn't cost turn-count times one-turn-cost.
Every provider requires the full conversation to be resent as input on each
turn, because none of them hold state for you between requests. Take a
support exchange running 200 tokens per turn: a flat estimate for ten turns
says 2,000 tokens. What actually gets billed — because turn N resends
turns 1 through N-1 in full — is 9,000 tokens of resent history before a
single new word of turn ten is counted. That's more than 4x the flat
estimate, and it's before adding what each turn actually contributes. The
shape is quadratic in turn count. A ten-message support chat and a
ninety-message one don't scale the same way, and a calculator that
multiplies instead of summing gets more wrong the longer the conversation
runs.

3. Tokenizers differ per family

"How many tokens is this prompt" doesn't have one answer across vendors,
because it isn't one algorithm. OpenAI-family models use a published,
runnable tokenizer (o200k_base), so an exact count is possible
client-side. Anthropic doesn't ship a public tokenizer at all — counting
happens server-side — so anything outside OpenAI's family is necessarily a
calibrated estimate: roughly 3.6 characters per token for English prose, 1.5
for CJK text, tuned against real samples rather than the model's own code.
A calculator that applies one ratio to every model it lists is applying an
OpenAI-shaped assumption to providers that never agreed to it, and it won't
say which of its numbers are exact and which are guesses, because it isn't
tracking the difference itself.

4. Caching is not free

"Prompt caching" sounds like a discount with no downside. The read side is:
cached input on GPT-5.6 Terra runs $0.20 per million against a $2 standard
rate, a 10x saving. The write side is where that framing falls apart. The
first time a prefix enters the cache, both OpenAI and Anthropic charge
more than the standard input rate to store it — 1.25x across every model
we've checked, from Claude Haiku 4.5 ($1.25 cache-write against a $1 input
rate, Anthropic, verified 2026-08-01) to GPT-5.6 Terra ($2.50 against $2,
OpenAI, verified 2026-08-01). A calculator that only ever shows you the 10x
read discount is showing you the number that makes the feature look best,
not the number your first request of the day will actually cost.

5. Long context costs more

Cross a provider's context threshold and the entire request reprices, not
just the tokens past the line. OpenAI's GPT-5.4 bills $2.50 input / $15
output per million tokens under 272,000 input tokens, and $5 / $22.50 — 2x
input, 1.5x output — the instant a request crosses that line (OpenAI,
verified 2026-08-03). That threshold applies per request, which means a
conversation can start under the tier and cross it mid-thread as history
accumulates: the same chat that priced cheaply on turn three can silently
double its input rate by turn twelve, and a calculator that quotes one flat
rate per model has no way to tell you when that happens.

6. Reasoning tokens are billable

What you see as "the answer" and what you're billed for are not the same
text. Reasoning models like OpenAI's o3 (billed at $2 input / $8 output per
million tokens, OpenAI, verified 2026-08-03) generate hidden thinking tokens
before the visible response, and those tokens are billed as output whether
or not you ever see them. A calculator that counts only the words on screen
is pricing a request that doesn't exist — the invoice includes tokens the
interface never showed you, and the gap between "visible answer" and
"billed output" is exactly the part a screenshot can't reveal.

7. Promotional pricing expires

Introductory rates are real, and they're also temporary. A calculator that
hard-codes them stops being honest the day the window closes. As of this
writing, Claude Sonnet 5 is priced at $2 input / $10 output through
2026-08-31 (Anthropic, verified 2026-08-01) — the standard rate that takes
over the day after is $3 / $15. An estimate built today and still quoted in
September is quoting a rate that no longer exists, unless the tool doing the
estimating knows the window has a date and checks it against the date it's
actually being asked. Most don't carry a date at all; they carry the number
that was true when someone last edited the page.

8. Assumptions are visible

Every calculator makes assumptions the provider hasn't published: what share
of a conversation is a cache hit, how large the hidden reasoning share
usually runs, whether the batch API is in play. Those assumptions decide the
number on screen as much as the published rates do. The difference is
whether they're printed next to the answer or buried in a methodology page
nobody reads — a promotional rate in effect, a reasoning multiplier applied,
a batch discount assumed, each stated in plain language under the total
rather than folded silently into it. A number with its assumptions attached
can be argued with. A number without them can only be believed or not.

9. Impossible scenarios are named

GPT-5 has a 272,000-token context window and a 128,000-token output ceiling
(OpenAI, verified 2026-08-03). Paste a 300,000-token document in and the
request can't be sent at all — it's 28,000 tokens over the line before
generation even starts. A calculator that just multiplies token count by
rate will return a confident dollar figure for a request the API would
reject outright, because multiplication doesn't know the difference between
an expensive request and an impossible one. The honest answer to "what would
this cost" is sometimes "nothing, because it can't happen" — and that answer
requires checking the ceiling, not just the rate.

The proof nobody asked for

Everything above is a claim about pricing engines in the abstract. Here's
what it looked like to get one wrong in public, because we did, twice, on
our own product.

Building an editor extension that prices the model on the line of code that
calls it meant running the same cost engine outside a browser, inside VS
Code, for the first time. Six defects came out of that work. Every one of
them passed 735 tests and six CI gates cleanly — verified, all green — and
two of them reached the Marketplace before a person looked at the screen and
noticed something was off.

A call with no token cap of its own silently inherited the cap from the
call above it, and priced it at that neighbor's rate. The extension scans
a file for model IDs and their nearby max_tokens values; the backward
search for "which cap belongs to this call" stopped at the previous model
ID. That sounds like the right boundary. It isn't — a cap written between
two model IDs actually belongs to the earlier call's forward-looking window.
So a call that never specified an output ceiling displayed one anyway: a
real number, correctly multiplied, describing a request nobody had written.

Exact token counting quietly degraded to an estimate, and reported a
number either way, so nothing on screen said it had happened. The tokenizer
ships as four external files rather than bundled into the extension — bundle
it, and the download goes from 19.7 KB to 3.4 MB, because you'd be shipping
a full rank table to every editor window whether or not anyone ran the
estimate command. The packaging step that decides which files actually ship
diverged from the module resolution that looks for them at runtime, so a
release could pass every test and still leave one required file behind.
When that happens, the code catches the failed import and falls back to the
same calibrated-ratio estimate section 3 describes above — silently,
correctly formatted, indistinguishable on screen from an exact count. Not an
error. Not a crash. A plausible number, standing in for a real one, with
nothing to tell you which you were looking at.

Both defects share that shape, and so does every row above: the failure
mode of a pricing tool is not crashing. A crash is loud, and someone fixes
it that afternoon. The failure mode is returning a real-looking figure —
clean formatting, correct arithmetic, plausible magnitude — for a scenario
that isn't the one you asked about. Nothing about a wrong-but-plausible
number announces itself. The only defense is being able to check where it
came from, which is the same defense this article has been asking for from
every calculator in the category, including — it turns out — our own.

Monitoring the promise, not the machinery

Fixing those two defects made the extension correct on the day it shipped.
It says nothing about whether the published catalog stays correct on every
day after. Those are different failure modes, and the second one has its own
watcher: a daily job that fetches promptspend.com/data/pricing.json — the
live site, not the repository — four hours after the sync runs, and opens an
alert if the catalog it finds is more than two days old.

The live site, deliberately, not the repository. Between a price moving at
a vendor and a reader seeing the update, there are four places the chain can
break: the sync can error, the sync can raise a review flag that sits
unmerged, the deploy can fail after a clean sync, or the CDN can serve a
stale build. Checking the file in git catches exactly one of those four.
Fetching what the site actually serves catches all of them, because it's
asking the question from where the answer matters — the reader's browser,
not a folder on a build server.

Monitor the promise you made to the reader, not the machinery you built to
keep it.

Every calculator in this category, including this one, is a pile of
assumptions wearing a clean interface. The nine on this page are the ones
that quietly favor a low number. The tenth is what happens when the people
who wrote the code stop trusting their own tests to catch the eleventh —
and start checking the thing they actually promised, instead of the thing
they built to promise it.