This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.
A user uploaded a 106-page PDF and clicked "Ask this book".
The screen said "Preparing… 0/0".
It said that for an hour.
No errors. No failed requests. No exceptions in the logs. Just one graph I still think about: a CPU-only container, pinned at 390%. Almost exactly four cores, flat, like a ruler.
The strange part: I pay for GPT-4.1 exactly so this kind of work never runs on my CPU. And the OpenAI dashboard showed zero traffic. Something was doing the work. Just not the thing I configured.
What was supposed to happen
TextStack is a reading app with an "Ask this book" feature. It is RAG over the book you uploaded. For table-heavy PDFs, plain text extraction is useless, so each page goes through vision parsing: page image in, Markdown out. That job belongs to GPT-4.1.
The provider is picked by a routing config. Simplified:
"Ai": {
"DefaultProvider": "ollama",
"Routes": {
"pdf.parse": "openai-pdf"
}
}
If a task has a route, it goes to that provider. If it doesn't, it falls back to the default. The default is a local Ollama, which I use for small cheap tasks.
Keep that fallback in mind. It is the villain of this story.
Debugging by absence
Most debugging starts with an error. This bug gave me none. The strongest clue was a thing that was missing.
Here is what I could see:
- The indexing job was running. Not failed, not stuck in a queue. Running.
- Zero requests to OpenAI. Not slow requests, not errors. Zero.
- The Ollama container was burning four cores. On a box with no GPU.
So the vision parsing was happening. On the local model. On CPU. I measured it later: about 42 seconds per page. The book had 106 pages. That is roughly 74 minutes of parsing for one book, all of it awaited inline by the indexing worker. The whole RAG pipeline sat behind it. From the outside it looked frozen at "Preparing… 0/0". From the inside it was working very hard on the wrong hardware.
Nothing failed. That was the problem. A crash would have paged me in a minute. A fallback that "works" can burn for an hour.
The root cause hurt a little
A few days earlier I shipped a reliability fix. Indexing used to run chunking inside the HTTP request, and it could get permanently stuck if the request died. So I moved the whole thing into a background Worker. Good change. I would do it again.
But the API and the Worker are separate processes, each with its own appsettings.json. The API config had the route:
"Routes": { "pdf.parse": "openai-pdf" }
The Worker config did not.
So when the parsing moved from the API into the Worker, the router looked up pdf.parse, found nothing, and did what fallbacks do. It quietly handed a vision parsing job for a 106-page book to a local model on a CPU-only container.
The work moved. The config did not. My reliability fix caused the regression.
The fix
One config block in the Worker's appsettings.json, mirroring the API:
"Routes": { "pdf.parse": "openai-pdf" }
plus the matching Ai:Pdf settings block. No code change. The container dropped from 390% to idle, OpenAI traffic came back, the book indexed in minutes.
Five minutes to write. Much longer to find.
Why this bug hid so well
Three reasons, and I think they generalize:
A fallback doesn't fail. It succeeds with the wrong tool. Every health check was green. The job was making progress. Slowly, expensively, but progress. There was no moment where the system could say "this is wrong", because by its own rules nothing was.
Nobody alerts on silence. I had alerts for errors and for high latency on my API. I had nothing for "an expensive external provider we depend on received zero calls today". Absence of a signal is the hardest thing to notice, because there is nothing to point at.
Config lives per process, but my mental model didn't. In my head there was one routing table. In reality there were two files, and they agreed only by accident, until they didn't.
What I changed in how I work
Three rules I took from this:
- When work moves between processes, config moves with it. That is now a checklist item on any "move X to the Worker" change: diff the relevant config sections between the two processes before shipping.
-
Make routing loud. One log line per job that says which provider was resolved and why:
pdf.parse -> ollama (no route, DefaultProvider fallback). That line would have turned an hour of confusion into a 30-second fix. - For expensive work, prefer fail-fast over fallback. A missing route for a cheap task can fall back. A missing route for vision parsing of a whole book should throw. A silent fallback is a decision the system makes without telling you. If the choice of provider matters, it should not be made silently.
The uncomfortable lesson on top: the bug was introduced by a fix I was proud of. Reliability work changes where code runs, and "where" is exactly what config is about.
Where in your system could a fallback take over silently, and how long would it take you to notice? Mine took a 106-page PDF and a CPU graph. I would love to hear yours in the comments.
I build TextStack, an open-source reader for technical books, in .NET. This bug is from the RAG indexing pipeline behind its "Ask this book" feature. github.com/mrviduus/textstack
Top comments (5)
I nearly choked on my coffee at "Nothing failed. That was the problem." That sentence alone should be printed on a mug for every DevOps engineer out there.
What got me most about this story is how the fix that caused the bug was actually a good fix. You moved work out of the HTTP request—that's textbook reliability. But the config didn't follow, because in your head there was still one routing table, while in reality there were two files living separate lives. I have been there, and it always stings a little more when the regression came from something you were genuinely proud of shipping.
The thing that scares me is how easily this could happen again in a larger system with ten services instead of two. One thing I started doing after a similar incident is adding a cheap "readiness check" that resolves every configured route at startup and logs which provider it actually picks. That way, if a route falls back to the default unexpectedly, I catch it within seconds of deployment, not an hour into a 106-page PDF. It is like a smoke test for your configuration surface area, and it has saved me more than once.
To your question about silent fallbacks in my own system—I had one last year where a rate-limited third-party API started quietly falling back to a local cache that was six months stale. No errors, just increasingly wrong recommendations that nobody complained about because they assumed the product was just supposed to be that mediocre. We only noticed when a customer finally asked "why does this keep suggesting the same thing?" and that question hurt more than any alert ever could.
Anyway, I am absolutely stealing your "prefer fail-fast over fallback for expensive work" rule. And I just cloned your TextStack repo—curious to poke around that indexing pipeline now.
The startup readiness check is better than my per-job logging, it catches the bad fallback right after deploy. Stealing it. And your stale cache story is the same bug with a worse ending. Enjoy the repo!
The 42 seconds per page on CPU with zero requests to OpenAI is the classic silent-fallback signature, and the fix that generalizes is to emit the resolved provider name as a trace attribute on every routed call. Then absence-of-signal at OpenAI becomes presence-of-signal at Ollama on the same span, and the pdf.parse route mismatch shows up in a single query instead of an hour of graphs. Do you also log when the default kicks in vs when a route matched, or just the provider it ended up on?
Both, the reason is the important part. "Route matched" is boring, "fell back to default" is the alarm. And you are right, a span attribute beats a log line, same data but you can query it. Adding that.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.