DEV Community: Vasyl

My Uptime Monitor Was Set to Every 5 Minutes. In July It Ran 15 Times a Day.

Vasyl — Tue, 18 Aug 2026 13:30:20 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

🔨 #bugsmash, week by week: a 390% CPU hour nobody noticed, a state machine with no exit, a backup that leaked 156 GB, a monitor that caught its own lies. Week five: the monitor that was quietly not running at all.

I had not opened the TextStack repo properly in a while. I have been working through Claude Code for weeks, task by task, and each task looked small. Yesterday I opened the whole thing and looked at it as one project. It has grown into something I cannot hold in my head anymore. That is a separate problem and I will get to it at the end.

While I was looking around I opened .github/workflows/health-check.yml. It is the file that tells me my site is alive.

on:
  schedule:
    - cron: '*/5 * * * *'

Every 5 minutes. It curls /health on textstack.app and textstack.dev, then does two smoke checks, book listing and search. If any of them fails, the run goes red and I get a notification.

288 runs a day. That is what the file says.

Then I counted the actual runs.

The number

The workflow was created on 6 January 2026. Here is what it did, month by month.

Month	Runs per day	Percent of 288
January (from the 6th)	107	37%
February	74	26%
March	73	25%
April	53	18%
May	35	12%
June	26	9%
July	15	5%
August (first 11 days)	20	7%

It went down every single month from January to July.

In July my health check ran 15 times a day. I had asked for 288. That is one check every 96 minutes instead of every 5.

And it never started at 100 percent either. Even in the first month it was already at about a third.

I did not notice any of this for seven months.

What made me look

On 6 August GitHub Actions went down. The incident opened at 15:22 UTC and was fully resolved at 02:04 UTC the next day, so almost 11 hours. At the peak, 71 percent of workflow runs were failing with infrastructure errors. My own work stopped for about four hours that afternoon, which is why I remembered the date.

So I checked what my health check did in that window. Four runs. Two of them failed. The largest gap that day was 5 hours 25 minutes, from 18:33 to 23:59, and that gap sits inside the incident.

At first I thought I had a clean story. GitHub went down, my monitor went down with it, blind for five hours.

Then I looked at the same day before the incident started. There was already a gap of 3 hours 22 minutes, from 00:08 to 03:30, hours before anything broke. And on 3, 4 and 5 August the workflow ran 12, 13 and 13 times.

So no. The outage did not blind me. The outage was just the day I happened to look. It had been like this since spring.

Why

This is documented. It is in the GitHub docs, in the part about the schedule event.

Scheduled events can be delayed during periods of high load of GitHub Actions workflow runs, and if the load is high enough, some queued jobs may be dropped.

Dropped. Not delayed. Dropped.

I knew that sentence existed. I read it as "sometimes it runs a bit late." It does not mean that. It means the run never happens and nothing tells you.

There is a second half to this, and I want to be careful about how strongly I put it. GitHub itself has been under a lot of load this year. Commits went from about 1 billion in all of 2025 to 275 million per week by April 2026. Actions compute hit 2.1 billion minutes in a single week. Pull requests opened by AI agents went from about 4 million in September 2025 to more than 17 million in March 2026.

My curve goes down while their curve goes up. That fits, and the mechanism is documented. But I only have my one repo, so I cannot prove the two curves are connected. I can only say the shape matches and GitHub says dropping is what happens under load.

The part I did not expect

Same repository, same seven months, same schedule trigger. My daily database backup:

on:
  schedule:
    - cron: '0 3 * * *'
jobs:
  backup:
    runs-on: self-hosted

224 runs since January. Zero missing days. Not one. Every month it ran on exactly as many days as the month has, and in May and June it ran a couple of extra times because I triggered it by hand.

So in the same repo one scheduled workflow degraded to 5 percent and the other stayed at 100 percent.

I want to say the difference is runs-on: self-hosted, because that is the obvious difference and it is the one I like. But the two workflows also differ in how often they ask for something. One asks 288 times a day, the other asks once. High frequency schedules are exactly what a queue sheds first. I cannot separate those two causes with the data I have, so I am not going to pretend I can.

I did get one thing wrong before checking, and it is worth saying. I assumed self-hosted would not protect me on 6 August, because the root cause that day was invalid job assignments to runners, and job assignment is GitHub's side, not mine. That sounded right. The backup ran normally on 6 and 7 August anyway. My reasoning was fine and the answer was still wrong.

Three layers, all built by me, none of them checking anything

Once I started pulling on this I found it was not one problem.

My README says the project has uptime monitoring with UptimeRobot probes. There is a whole file for it, docs/03-ops/uptime-monitoring.md. It is a good document. It has a table of three monitors with URLs, expected responses and intervals. It has numbered setup steps. It has an alert-response runbook with the exact commands to run for each kind of alert, down to which containers to check logs for.

I do not have an UptimeRobot account. I never signed up. I wrote the whole document for the setup I was going to build, and then did not build it.

The last section of that same file is the part that got me. It describes the GitHub Actions health check as internal belt-and-braces, useful when UptimeRobot itself has an outage, but redundant otherwise.

Redundant. My documentation called it the backup to the real monitoring. It was the only monitoring. And it was running at 5 percent.

Second layer: the cron says every 5 minutes. It runs every 96 minutes.

Third layer: the Actions tab is green. A green tab means every run that happened passed. It looks exactly the same when almost no runs happened at all. There is no colour for "did not run."

Three sources of confidence. All three mine. None of them lying on purpose, and none of them checking anything.

This is the same shape as a bug I wrote about a few weeks ago, where my backup verifier filled the disk of the machine it was verifying. Someone in the comments gave me the sentence for it: a verifier that shares a resource pool with the thing it verifies eventually becomes the largest consumer of it. This is one level up. A monitor that shares a platform with the thing it monitors inherits that platform's bad days, and stays silent about it.

What I am not going to do

The obvious fix is to sign up for an external monitoring service. Free tier, five minute checks, runs on somebody else's machines, done in twenty minutes.

I am not going to do that, and I want to explain why, because the reason is the thing I actually learned this week.

I opened this project yesterday and could not hold it in my head. It is one person's side project and it has a backend, a web app, an admin app, a mobile app, a worker, an MCP server, a browser extension, an SSG pipeline, and now eight workflows. Every one of those was a reasonable decision on the day I made it. Together they are more than I can carry.

Every new service is not just the service. It is another account, another set of credentials, another page in the docs, another thing that is quietly broken in six months while I believe it is running. I know that last one is real now, because that is exactly what my UptimeRobot documentation was: a doc for a thing that did not exist.

Adding a monitoring service to fix bad monitoring is how the pile got this big.

So the rule I am taking out of this is not "add a monitor." It is narrower. Do not add a second thing where the first thing was never checked. And when you do keep something, keep it where you can see it.

Why I am leaving it exactly where it is

The obvious fix is to move the health check to runs-on: self-hosted, next to the backup that has not missed a day in seven months. One line changed. 100 percent instead of 5 percent. I spent a day thinking that was the answer.

It is the wrong answer, and working out why is the part of this I am actually glad about.

My self-hosted runner is a machine in my flat. If that machine dies, a monitor running on it dies at the same moment and tells me nothing. I would find out on my next login, and I log in maybe once a week.

GitHub is the opposite. When GitHub breaks, somebody whose job it is fixes it, usually within hours, without me. The 6 August outage lasted 11 hours and I did nothing about it.

So the two options are not "5 percent reliable" versus "100 percent reliable." They are:

A check that runs every 96 minutes and is not on my machine.
A check that runs every 5 minutes and dies silently with my machine.

The second one has perfect uptime for every failure except the one failure I have no other way of seeing. My server dying is the thing I am most exposed to, because I am not looking at it. A monitor that shares its fate is not a monitor. It is a second thing that will be dead at the same time.

So the degraded one is the one worth keeping. Not because 15 runs a day is good, but because it runs somewhere I do not control and do not have to maintain, and that is the whole property I need from it.

What I give up is resolution. If the site goes down I might hear about it 96 minutes later instead of 5. For a project with my traffic that is fine, and I would rather know late than not know.

The mistake I made for seven months was not putting the check on GitHub. That was right. The mistake was reading */5 * * * * as a promise and never checking whether it was kept. I optimised the number in the file and never looked at the number in reality.

A monitor has two properties. How often it runs, and whether it can survive what it is watching. I spent all my attention on the first one. The second one is the one that decides whether it is a monitor at all.

The trade used to be easier

I want to be honest that I am less comfortable with this than I would have been a year ago.

Leaning on GitHub used to be a decision you made once and forgot. It was background. It was there. That is what made "just put it on Actions" the obvious call for a solo project.

It is not that obvious now, and the numbers say why. My own schedule went from 37 percent of what I asked for in January to 5 percent in July, and nothing about my repo changed in that time. GitHub had nine service-degrading incidents in May alone. On 6 August, Actions was broken for almost 11 hours with 71 percent of runs failing at the peak. In June, Microsoft started renting capacity from AWS to keep GitHub standing up, which is not a thing a company does when the situation is comfortable.

The reason underneath is not a mystery either. Commits went from roughly 1 billion in all of 2025 to 275 million per week. Actions compute hit 2.1 billion minutes in one week. Pull requests opened by AI agents went from about 4 million to over 17 million in six months. The platform is absorbing a change in how code gets written, and the parts that get squeezed first are the low priority background ones. A 5 minute cron on a small public repo is exactly that.

So I am keeping the trade, but I am not calling it free anymore. It has a cost that is going up, and the cost is paid in silence, which is the worst currency for a monitor.

What I will do is check the number again in three months. If 5 percent has become 2 percent, then the check has stopped being a check and I will have to pay for something. I would rather find that out by counting than by an outage.

The one thing I am changing is the documentation. docs/03-ops/uptime-monitoring.md still describes an UptimeRobot setup I never built, and calls the GitHub check redundant. It is not redundant. It is the whole thing. That file now says what actually exists, roughly how often it really runs, and what it does not catch.

That edit does not improve my uptime by one second. It just means that in six months I will not believe something that is not true.

What I keep from all this is the number. 15 runs a day out of 288, for a month, without noticing. And the thing that finally told me was not any of my monitoring. It was me opening the repo and counting by hand.

If you have a scheduled workflow you rely on, go count its runs. Not the failures. The runs. It takes two minutes and I would like to know whether my curve is mine alone.

I build TextStack, an open-source reader for technical books, in .NET. The full write-up lives on my blog. github.com/mrviduus/textstack

Nobody Alerts on Silence: Wiring Sentry Into an LLM Pipeline

Vasyl — Tue, 11 Aug 2026 12:00:00 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

🔨 #bugsmash, week by week: a 390% CPU hour nobody noticed, a state machine with no exit, a backup that leaked 156 GB. Week four is the finale: I wired monitoring into the pipeline that produced all three — and its best catch was itself.

Project Overview

TextStack is an open-source reader for technical books, built in .NET: an ASP.NET Core API, a background Worker, PostgreSQL + pgvector, React on top. The LLM pipeline does translation, word explanations, "Ask this book" RAG, and three production agents (Enrichment, Librarian, Tutor), routed between a local Ollama and OpenAI by a config-driven router. The code is public: github.com/mrviduus/textstack.

Bug Fix or Performance Improvement

Three weeks ago a user's PDF fell through my LLM router onto a CPU-only Ollama container instead of GPT-4.1, and my CPU sat at 390% for an hour. Zero exceptions. Zero error logs. Zero alerts. And when I went to see what my existing observability had recorded, the answer was nothing at all: the OTLP exporter pointed at an Aspire dashboard container that is profile-gated and doesn't run in production. Every span my services had ever produced in prod had been fired into a closed socket.

Observability you never read is indistinguishable from observability you never installed.

The one-line config fix was submission #1. This submission is the fix for the class of bug — a system that has no way to make a sound when it does the wrong thing successfully:

the router now records why it picked a provider, not just which one;
expensive tasks landing on the default provider fire a throttled Sentry alert;
provider failures the client deliberately swallows now report before returning their empty response;
the Worker probes provider reachability at startup and a circuit breaker stops a dead provider from eating 50 × 90 s of wall-clock per start;
and an environment-tag fix so a laptop can never masquerade as production again (that story is below — it earned its own PR).

Code

Four PRs, all merged to main; 1,363 unit tests, full CI green:

#445 — the Sentry integration: SDK for API + Worker, route-reason spans, agent transactions, allowlist scrubber
#446 — the first leak it found in itself: EF Core SQL riding in breadcrumb messages, past a green-tested scrubber
#447 — the reader-facing race it found in me: 23505 on reading-progress upserts, real users losing their place in books
#448 — the readiness probe, the circuit breaker, and the environment-tag fix

My Improvements

The router now says why. Route resolution was a ?? chain that produced a string — identical whether an operator deliberately routed a task or it fell off the end onto the default. That chain doesn't just fail to record intent; it destroys it. So it returns two things now:

private RouteDecision ResolveRoute(string? featureTag)
{
    var matched = RegistryKey(featureTag) ?? ConfigRouteKey(featureTag);
    return matched is not null
        ? new RouteDecision(matched, RouteReason.RouteMatched)
        : new RouteDecision(config["Ai:DefaultProvider"] ?? "openai",
                            RouteReason.DefaultFallback);
}

Every LLM call tags its span with ai.task, ai.provider.resolved, and ai.provider.reason = route_matched | default_fallback. "Which model answered this, and did anyone choose it on purpose?" is now a trace query instead of a CPU graph.

Alert arithmetic. pdf.parse resolves a route once per page with parallelism six — my first version would have turned the original incident into 106 identical Sentry events. Every alarm goes through a throttle keyed on (task, provider, reason): first hit fires immediately, then one event per hour per distinct problem. The unit test literally counts to 106 and asserts one claim.

No silent fallback, ever — in either direction. When the breaker finds Ollama dead, tasks are skipped and stay queued; nothing auto-switches to a paid provider, because that converts an outage into unbounded spend. Provider choice stays 100% config-driven.

And the first live run found a hole in my own fix. The startup probe opens the circuit on a one-minute backoff; the backfill worker wakes after a two-minute start delay — by then the circuit is legitimately half-open, and my single up-front gate waved the whole batch through. A per-book re-check turned 38 calls into one:

Metadata backfill: enriching 38 user books
Metadata backfill: aborting after 0 enriched / 1 failed — provider 'ollama'
  is unavailable; the remaining candidates stay queued

Tests check what you imagined; a live run checks what's there.

Best Use of Sentry

Error Monitoring — what the first 24 hours in production caught:

The OpenAI account was out of credits. HTTP 429 (insufficient_quota) on /translate and /explain — the entire paid surface had been failing for readers for twelve hours. No version of my logs would have surfaced that before a user complained.
Readers were losing their place in books. PUT /me/progress threw 23505: duplicate key value violates unique constraint ten times in four hours: a textbook read-then-insert race (session heartbeat + sendBeacon on unload + second device), milliseconds wide, invisible in tests. Fixed in #447.

Custom tags (ai.task, ai.provider, ai.failure, agent.name, agent.outcome) go through an allowlist scrubber — every tag not explicitly blessed dies at the edge, so a future SetTag("prompt", userText) can never leak. A Sentry issue answers "which feature, on which model, is broken?" without opening a trace.

Tracing covers agent runs and RAG indexing at 100% sampling (they're the reason I installed this), HTTP at 20%, health checks at 0%. I rejected the deprecated OTel bridge specifically because spans leaving through the OpenTelemetry SDK bypass BeforeSend — my OTel pipeline carries raw client IPs and full SQL text that must never leave the box. A tiny TraceScope dual-writes an Activity and a Sentry span instead, so everything Sentry receives passes my scrubber.

Breadcrumbs caught my scrubber lying — twice. A live event's breadcrumb trail contained SQL: EF Core interpolates the query into the breadcrumb message, not the structured data bag my scrubber nulled (and my unit tests were green the whole time, asserting exactly the wrong thing). Fixed by dropping EF command breadcrumbs outright — then production found the same leak in a second channel: EF logs a failed command at Error level and Sentry's ILogger integration promotes it to an event, SQL in the message again. A scrubber written against one egress path will be bypassed by the next one. Both doors are closed in #446, and dropping loses no signal — the exception middleware already reports the same failure with the SQLSTATE and constraint name, no SQL.

Release + environment tags as forensics. The most interesting issue of the first day showed a dead Ollama starving a metadata pipeline: thirty events, tagged environment: Production. I read it as an outage and started writing the fix. It was my laptop — a dev .env with ASPNETCORE_ENVIRONMENT=Production plus the production DSN I'd pasted in to verify the integration. What broke the spell was Sentry's own metadata: linux-arm64 runtime on an x86_64 prod, and a release tag pointing at a commit that had never been deployed. An environment tag is a claim a process makes about itself, not a fact. Now SENTRY_RELEASE comes from the GIT_SHA build arg — every CI-built image has one, no dotnet run ever does — and a Production claim without a release gets renamed production-unverified (#448).

And the meta-lesson that justified the whole exercise: I verified the integration by sending real events at the real DSN and reading the captured payloads in the UI — that's how the leaks, the inferred-geo surprise, and the middleware capture path all surfaced. A monitoring system whose first act is to indict itself is one you can start trusting.

What's the most embarrassing thing your monitoring has ever caught — and was it in the code, or in you?

I build TextStack, an open-source reader for technical books, in .NET. The full write-up lives on my blog. github.com/mrviduus/textstack

My Backup Leaked 156 GB, Filled the Disk, and Broke Itself. The Fix Was One Letter.

Vasyl — Tue, 04 Aug 2026 12:00:00 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

🔨 #bugsmash, week by week. Every week I take one real production bug from textstack.app — my public-domain e-reader side project — and write up the full detective story: symptom, wrong suspects, root cause, fix. This one is about a backup that ate its own disk. Originally published on vasyl.blog.

My nightly backup has a verify step. It restores the fresh dump into a throwaway postgres container and runs sanity queries — because a backup you never restored is a hope, not a backup.

One week it started flapping. postgres did not become ready. That was the whole error. The dump itself succeeded. The storage tarball succeeded. Only the gate failed — the step whose entire job is to tell me the backup is real.

Then one morning the box stopped answering SSH.

The innocent suspect

First check, obviously: disk.

$ df -h /
Filesystem      Size  Used Avail Use%
/dev/sda1        96G   56G   36G  61%

61%. Plenty of room. Disk is innocent. Move on.

Except df -h / answers a narrower question than the one I asked. On this box, Docker's data-root lives on a separate partition — /mnt/data. Everything Docker writes — images, containers, volumes — goes there, not to /. And /mnt/data looked like this:

$ df -h /mnt/data
Filesystem      Size  Used Avail Use%
/dev/sdb1       196G  183G     0 100%

Zero bytes free. And docker system df pointed at the culprit: 156 GB of dangling anonymous volumes. Fifty-seven of them. All identical. All pgdata.

One volume per run, every run, for 34 days

Here's the mechanism, and it's built from three facts that are each harmless on their own.

Fact one: the official postgres image declares VOLUME /var/lib/postgresql/data. If you don't mount something there yourself, Docker silently creates an anonymous volume for every container you start.

Fact two: --rm only fires when the container exits on its own. A run that gets killed or times out never reaches that point.

Fact three: my cleanup path for exactly those killed runs did docker rm -f. Without -v. That removes the container and leaves its anonymous volume behind — a full initialized pgdata directory, orphaned, every time.

One leaked volume per bad run. Daily backups. Thirty-four days. 156 GB.

The loop closes

This is the part I find beautiful, in the way you can only appreciate after the incident is over.

The docker root fills up. A fresh throwaway postgres can no longer initdb — nowhere to write its data dir. So the verify step fails. A failed, timed-out run is exactly the kind that skips --rm and goes through the leaky cleanup. Which leaks another volume. Which leaves the disk fuller than before.

The backup broke the very step that verified the backup. A self-reinforcing failure, powered entirely by its own cleanup code.

One thing that mattered a lot at 11pm: real data was never at risk. Prod postgres and file storage bind-mount to / — the partition sitting comfortably at 61%. The only thing bloating /mnt/data was fifty-seven copies of a database that existed for ninety seconds each, just to prove a dump restores.

The fix

One letter, in two places:

# -v removes the container's ANONYMOUS volume too. postgres declares an
# anonymous VOLUME at /var/lib/postgresql/data, so every run that reaches
# `docker rm -f` (a killed/timed-out run where --rm never fired) otherwise
# leaks a full pgdata volume.
cleanup() {
  docker rm -fv "$CONTAINER" >/dev/null 2>&1 || true
}
trap cleanup EXIT

docker rm -f became docker rm -fv — in the trap handler and in the reap of leaked verify containers from previous runs. Plus a catch-all for the runs even a trap can't cover (SIGKILL):

# Belt-and-suspenders: drop any dangling anonymous volumes orphaned before
# this fix (or by an OOM-killed `docker rm`). Named volumes are untouched.
docker volume prune -f >/dev/null 2>&1 || true

And because the original failure hid behind a bare postgres did not become ready, the verify script now diagnoses itself: on readiness failure it dumps the container's docker logs, df -h, and docker system df straight into the CI log, and bails early if the container dies during startup instead of waiting out the full window. The next time this class of bug shows up, the error message will contain its own root cause.

Live remediation was one command. docker volume prune reclaimed 163 GB. /mnt/data went from 100% to 17%.

What I keep

df -h / is not "the disk." If Docker's data-root lives on its own partition, the partition you check by habit can say 61% while the one that matters says 100%.

Any image with a VOLUME declaration is a leak waiting for a missing -v. You don't opt into anonymous volumes; they happen to you. Every docker rm without -v on such a container strands one.

--rm is happy-path cleanup. The trap handler is the real cleanup — and it must be at least as thorough as --rm would have been, which means it needs -v too.

A verify step needs its own observability. Mine guarded the backups for months and then failed with seven words and no evidence. Any gate that can fail should dump the state needed to diagnose the failure, in the failure itself.

And one for the road: a full disk still lets you SSH in. When the box stopped answering entirely, that was a clean manual reboot — not the disk. An SSH connect timeout is a host-offline signal. Knowing which symptom belongs to which failure saved me from chasing a second ghost that night.

Stuck at "Preparing… 0/0" Forever. My State Machine Had No Exit.

Vasyl — Tue, 28 Jul 2026 12:00:00 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

The project

TextStack is an open-source reader for technical books, built in .NET. It has an "Ask this book" feature: RAG over the book you uploaded. Before you can ask, the book must be indexed: split into chunks, and for PDFs each page goes through a paid vision parse that takes minutes. The code is public: github.com/mrviduus/textstack.

The bug

A user clicks "Ask this book". The screen shows "Preparing… 0/0". And for some books it stayed there forever. Not slow. Forever. The retry button did nothing. The only cure was me poking the database by hand.

Here is what the old flow did:

POST /me/books/{id}/index
  -> flip rag_status to Indexing        (in the endpoint)
  -> run the whole chunking inline      (minutes of vision parsing, still in the HTTP request)
  -> return 202 when done

Two problems living inside one line: the work ran inside the HTTP request, and the status flip happened before the work.

A vision parse of a big PDF runs for many minutes. Cloudflare cuts the request long before that. The API container also restarts on every deploy. In both cases the process doing the work dies quietly, and the row stays like this:

rag_status = Indexing, rag_chunk_count = 0

And here is the real bug. The retry path only re-claimed rows in status NotIndexed or Failed:

// old claim: an Indexing row can never be claimed again
if (book.RagStatus is RagIndexStatus.NotIndexed or RagIndexStatus.Failed) { ... }

Indexing with zero chunks was a state with no exit. Not failed, so no retry. Not ready, so no answers. A dead end that looks like progress. "Preparing… 0/0", forever.

The fix, in three pieces

The shape of the fix: the endpoint only claims and answers, a background worker does the work, and every path ends in a terminal state. (Full code: RagIndexingService.cs and RagIndexingWorker.cs.)

1. The endpoint returns 202 immediately. It flips the status and goes home. No paid work on the HTTP path, nothing for Cloudflare to kill.

2. A worker claims the row atomically. No locks, no queue infrastructure. One UPDATE where the WHERE clause is the whole concurrency story:

var claimed = await db.Database.ExecuteSqlInterpolatedAsync($"""
    UPDATE user_books
    SET rag_indexing_started_at = now()
    WHERE id = {bookId}
      AND rag_status = 1              -- Indexing
      AND rag_chunk_count = 0
      AND rag_indexing_started_at IS NULL;
    """, ct);
if (claimed == 0) return;             // someone else owns it, walk away

If two workers pick the same row, one gets rowcount 1 and proceeds, the other gets 0 and returns. Postgres is the referee.

3. A sweep recovers dead rows. Every 30 seconds the worker also checks: any row still Indexing with zero chunks whose rag_indexing_started_at is older than 15 minutes means the process that claimed it died mid-parse. That row flips to a terminal Failed with a human-readable reason:

UPDATE user_books
SET rag_status = 3, rag_error = 'indexing interrupted, retry'
WHERE rag_status = 1 AND rag_chunk_count = 0
  AND rag_indexing_started_at < {staleCutoff};

Note what it does NOT do: auto-requeue. A vision parse is real paid spend. The row fails loudly, the user re-triggers deliberately. Retry is a human decision when every retry costs money.

The detail that bit me anyway

The failure path had its own bug waiting. When indexing dies because the worker is shutting down, the cancellation token is already cancelled. If you write the Failed status using that same token, the error write gets cancelled too, and you are back to the forever-Indexing dead end you just fixed.

// the terminal write runs on a FRESH context with CancellationToken.None:
// the token that killed the work must not kill the record of its death
await using var db = await dbFactory.CreateDbContextAsync(CancellationToken.None);

The same cancellation also produces a second trap: the chunker returns 0 chunks on shutdown, which looks identical to a genuinely empty book. Without checking ct.IsCancellationRequested, a perfectly good book gets labeled "No chapters to index" and a user believes their upload is broken. Zero is the least trustworthy number in a distributed system.

Before and after

	Before	After
Work runs	inside the HTTP request	in a background worker
Request dies mid-parse	row stuck Indexing forever	sweep flips it to Failed in 15 min
Retry button	dead for stuck rows	works, Failed is claimable
Shutdown during parse	"No chapters to index" (wrong)	"indexing interrupted, retry"
Concurrency control	none needed (and none possible)	one atomic UPDATE, rowcount as the lock

What I took away

A status enum is not a state machine until every state has an exit. Indexing was a real status, rendered nicely in the UI, and it was also a trap: no transition out except success. The fix was not clever code. It was drawing the diagram and asking one question about every state: how does a row leave here if the process dies right now?

And the operational rule on top: never let an HTTP request own work that outlives an HTTP request. The request is a messenger, not a worker.

What is the longest a "temporary" status has survived in your database? I found rows that had been "Preparing" for weeks.

I build TextStack, an open-source reader for technical books, in .NET. This fix is from the indexing pipeline behind its "Ask this book" feature. github.com/mrviduus/textstack

Nothing Crashed. Nothing Errored. My CPU Sat at 390% for an Hour.

Vasyl — Tue, 21 Jul 2026 12:00:00 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

A user uploaded a 106-page PDF and clicked "Ask this book".

The screen said "Preparing… 0/0".

It said that for an hour.

No errors. No failed requests. No exceptions in the logs. Just one graph I still think about: a CPU-only container, pinned at 390%. Almost exactly four cores, flat, like a ruler.

The strange part: I pay for GPT-4.1 exactly so this kind of work never runs on my CPU. And the OpenAI dashboard showed zero traffic. Something was doing the work. Just not the thing I configured.

What was supposed to happen

TextStack is a reading app with an "Ask this book" feature. It is RAG over the book you uploaded. For table-heavy PDFs, plain text extraction is useless, so each page goes through vision parsing: page image in, Markdown out. That job belongs to GPT-4.1.

The provider is picked by a routing config. Simplified:

"Ai": {
  "DefaultProvider": "ollama",
  "Routes": {
    "pdf.parse": "openai-pdf"
  }
}

If a task has a route, it goes to that provider. If it doesn't, it falls back to the default. The default is a local Ollama, which I use for small cheap tasks.

Keep that fallback in mind. It is the villain of this story.

Debugging by absence

Most debugging starts with an error. This bug gave me none. The strongest clue was a thing that was missing.

Here is what I could see:

The indexing job was running. Not failed, not stuck in a queue. Running.
Zero requests to OpenAI. Not slow requests, not errors. Zero.
The Ollama container was burning four cores. On a box with no GPU.

So the vision parsing was happening. On the local model. On CPU. I measured it later: about 42 seconds per page. The book had 106 pages. That is roughly 74 minutes of parsing for one book, all of it awaited inline by the indexing worker. The whole RAG pipeline sat behind it. From the outside it looked frozen at "Preparing… 0/0". From the inside it was working very hard on the wrong hardware.

Nothing failed. That was the problem. A crash would have paged me in a minute. A fallback that "works" can burn for an hour.

The root cause hurt a little

A few days earlier I shipped a reliability fix. Indexing used to run chunking inside the HTTP request, and it could get permanently stuck if the request died. So I moved the whole thing into a background Worker. Good change. I would do it again.

But the API and the Worker are separate processes, each with its own appsettings.json. The API config had the route:

"Routes": { "pdf.parse": "openai-pdf" }

The Worker config did not.

So when the parsing moved from the API into the Worker, the router looked up pdf.parse, found nothing, and did what fallbacks do. It quietly handed a vision parsing job for a 106-page book to a local model on a CPU-only container.

The work moved. The config did not. My reliability fix caused the regression.

The fix

One config block in the Worker's appsettings.json, mirroring the API:

"Routes": { "pdf.parse": "openai-pdf" }

plus the matching Ai:Pdf settings block. No code change. The container dropped from 390% to idle, OpenAI traffic came back, the book indexed in minutes.

Five minutes to write. Much longer to find.

Why this bug hid so well

Three reasons, and I think they generalize:

A fallback doesn't fail. It succeeds with the wrong tool. Every health check was green. The job was making progress. Slowly, expensively, but progress. There was no moment where the system could say "this is wrong", because by its own rules nothing was.

Nobody alerts on silence. I had alerts for errors and for high latency on my API. I had nothing for "an expensive external provider we depend on received zero calls today". Absence of a signal is the hardest thing to notice, because there is nothing to point at.

Config lives per process, but my mental model didn't. In my head there was one routing table. In reality there were two files, and they agreed only by accident, until they didn't.

What I changed in how I work

Three rules I took from this:

When work moves between processes, config moves with it. That is now a checklist item on any "move X to the Worker" change: diff the relevant config sections between the two processes before shipping.
Make routing loud. One log line per job that says which provider was resolved and why: pdf.parse -> ollama (no route, DefaultProvider fallback). That line would have turned an hour of confusion into a 30-second fix.
For expensive work, prefer fail-fast over fallback. A missing route for a cheap task can fall back. A missing route for vision parsing of a whole book should throw. A silent fallback is a decision the system makes without telling you. If the choice of provider matters, it should not be made silently.

The uncomfortable lesson on top: the bug was introduced by a fix I was proud of. Reliability work changes where code runs, and "where" is exactly what config is about.

Where in your system could a fallback take over silently, and how long would it take you to notice? Mine took a 106-page PDF and a CPU graph. I would love to hear yours in the comments.

I build TextStack, an open-source reader for technical books, in .NET. This bug is from the RAG indexing pipeline behind its "Ask this book" feature. github.com/mrviduus/textstack

Your RAG Eval Isn't Flaky. Your Retrieval Is Non-Deterministic.

Vasyl — Tue, 14 Jul 2026 14:48:00 +0000

Same query.
Same documents.
Same model.
And the RAG eval can still hand back a different Recall@8.

Not because the model is flaky. Because of an ORDER BY clause.

I didn't find this by watching a metric wobble. I found it reading the retrieval code, and realized the score would drift run to run even if the model never changed.

This came out of a habit I've adopted recently: I write the eval before the feature. Reviewing the retrieval pipeline behind my "Ask this Book" feature, I saw it: the retrieval layer wasn't deterministic.

Order isn't presentation. It's part of the input.

My RAG implementation is intentionally simple: plain PostgreSQL and .NET. Two retrieval strategies over the same table:

semantic search using pgvector
lexical search using PostgreSQL full-text search

The results are merged with Reciprocal Rank Fusion (RRF).

Here's the important part: RRF doesn't care about the retrieval scores. It only cares about rank.

If one retriever returns

A
B
C

instead of

B
A
C

RRF produces different fused scores. Different fused scores mean a different Top-K. Different Top-K means different Recall@K.

In RRF, order isn't a display detail. Order is data.

The bug

My lexical query ended like this:

ORDER BY score DESC

Looks perfectly reasonable. Except ts_rank_cd produces ties surprisingly often. Multiple chunks can have exactly the same score.

And SQL only guarantees the ordering you explicitly request. If multiple rows compare equal, PostgreSQL is free to return them in any order.

Nothing changed. Same database. Same query. Same model. Only the order of equally-ranked rows. Yet that's enough for RRF to assign different ranks, producing different fused scores and a different evaluation result.

The semantic retrieval had the same issue. Distance ties are much rarer than lexical ties, but "rare" isn't good enough for an evaluation pipeline.

The fix

The fix was almost embarrassingly small.

Before:

ORDER BY score DESC

After:

ORDER BY score DESC, id

A deterministic tie-breaker on both retrieval queries. Now equal-scoring rows always appear in the same order, RRF receives the same input every run, and the Top-K stays identical.

Notice what didn't happen. The retrieval didn't become better. It became reproducible.

Why this matters

We spend a lot of effort making the model deterministic during evaluation: temperature 0, fixed datasets, golden answers, reproducible prompts.

But it's easy to assume everything underneath the model is already deterministic. Often it isn't. Retrieval. Ranking. Sampling. Data loading. Any non-deterministic stage in the pipeline can quietly invalidate your eval.

A fluctuating eval isn't just annoying. It's dangerous. Eventually you stop trusting the number, even when it's pointing at a real problem.

The lesson I took away

Before debugging the model, debug determinism. An evaluation can only be as deterministic as the pipeline feeding it. Same query. Same rows. Same order. Only then can you trust what your eval is telling you.

I build TextStack, an open-source reader for technical books, in .NET. This is from the retrieval layer behind its "Ask this Book" feature. Code on github.com/mrviduus/textstack.

AI Wrote a Thread-Safe Counter. The CPU Made It 5x Slower.

Vasyl — Tue, 07 Jul 2026 13:00:00 +0000

Meet the cache line.

I asked an AI assistant for a simple thing: per-thread counters. Four threads, each incrementing its own slot in an array. No shared variables. No locks needed. The code it wrote was clean and correct:

long[] counters = new long[threadCount];
// thread t does: counters[t]++

Every thread writes only to its own element. There is no race here. Any code review would pass it. Every test would pass too.

Then I measured it against a version that does exactly the same work, and the correct code lost by 5x.

The numbers

This is from my machine (Apple Silicon, .NET 10, 4 threads, 200 million increments per thread):

Round 1:  adjacent (one cache line):  464 ms   padded (line per thread):  85 ms   ratio: 5.4x
Round 2:  adjacent (one cache line):  397 ms   padded (line per thread):  85 ms   ratio: 4.7x

Same loop. Same number of increments. Same "each thread touches only its own counter". The only difference between the two versions is where the counters live in memory.

What the hardware actually does

A CPU never reads one byte from memory. It moves data in fixed blocks called cache lines — 64 bytes on x86, 128 bytes on Apple Silicon. Ask for one long and the whole block it lives in travels into the core's cache. Think of a cook whose ingredients are in a basement fridge: going downstairs is expensive, so you never carry one carrot — you carry the whole crate.

(Don't take my word for the 128: run sysctl hw.cachelinesize on an M-series Mac.)

Usually this works for you. Array elements sit side by side, so scanning an array is fast: you touch one element and the next fifteen arrive in the same crate for free.

But with multiple cores there is a rule: to write into a cache line, a core must own it exclusively. The moment core 1 writes, every other core's copy of that line is declared stale. And the unit of ownership is not your variable. It is the whole line.

Now look at my four counters. Four long values, 8 bytes each, side by side — 32 bytes. They all fit in one cache line. Thread 1 increments its counter and takes ownership of the line. A nanosecond later thread 2 increments its own, different counter — and has to rip the same line back. The line ping-pongs between cores on every single write. Four threads that share nothing in the code are fighting over one crate in the hardware.

This is called false sharing. False, because no data is actually shared. The fight is real anyway.

Four counters in one cache line vs. one counter per line — same code, 5x difference.

The demo fix is one attribute

Give every thread its own cache line. Pad each counter so the next one starts in a different line:

[StructLayout(LayoutKind.Explicit, Size = 128)]
struct PaddedCounter
{
    [FieldOffset(64)] public long Value;
}

That is the entire difference between 464 ms and 85 ms. We pay a little memory — 128 bytes per counter instead of 8 — and get back the parallelism we thought we already had. (I use 128, not 64, for two reasons: Apple Silicon lines are 128 bytes, and on x86 the adjacent-line prefetcher likes to drag neighboring lines along.)

Run it yourself

The full demo is about 70 lines, no project file needed — with the .NET 10 SDK you can run a single file directly:

dotnet run -c Release FalseSharingDemo.cs

// FalseSharingDemo.cs — .NET 10 file-based app
using System.Diagnostics;
using System.Runtime.InteropServices;

const long Iterations = 200_000_000;
int threadCount = Math.Min(Environment.ProcessorCount, 4);

for (int round = 0; round < 2; round++)
{
    var slow = MeasureAdjacent(threadCount);
    var fast = MeasurePadded(threadCount);
    Console.WriteLine($"adjacent: {slow.TotalMilliseconds:F0} ms   padded: {fast.TotalMilliseconds:F0} ms   ratio: {slow / fast:F1}x");
}

static TimeSpan MeasureAdjacent(int threadCount)
{
    long[] counters = new long[threadCount];
    return RunThreads(threadCount, t =>
    {
        for (long i = 0; i < Iterations; i++)
            counters[t]++;
        return counters[t];
    });
}

static TimeSpan MeasurePadded(int threadCount)
{
    var counters = new PaddedCounter[threadCount];
    return RunThreads(threadCount, t =>
    {
        for (long i = 0; i < Iterations; i++)
            counters[t].Value++;
        return counters[t].Value;
    });
}

static TimeSpan RunThreads(int threadCount, Func<int, long> body)
{
    long sink = 0;
    var threads = new Thread[threadCount];
    var sw = Stopwatch.StartNew();
    for (int t = 0; t < threadCount; t++)
    {
        int id = t;
        threads[t] = new Thread(() => Interlocked.Add(ref sink, body(id)));
        threads[t].Start();
    }
    foreach (var th in threads) th.Join();
    sw.Stop();
    GC.KeepAlive(sink);
    return sw.Elapsed;
}

[StructLayout(LayoutKind.Explicit, Size = 128)]
struct PaddedCounter
{
    [FieldOffset(64)] public long Value;
}

Where this hides in real code

You will not write four counters in a loop at work. But you will write, or an AI will write for you:

Cache statistics. Almost every in-memory cache keeps hits, misses, evictions. The natural implementation is fields next to each other, or a long[] with one slot per shard, updated with Interlocked.Increment from every thread. Fields next to each other means one cache line. This is literally the demo above, running in your production service.

Sharded counters. The cruel version: you sharded a counter specifically to make it parallel, put the shards in one array — and they still share lines. You did the architecture work and the hardware quietly undid it.

LRU metadata. A compact long[] lastAccessTicks per cache slot means every cache read becomes a write into a hot shared array. A read-heavy cache that is slow because of writes is a fun thing to debug.

And sometimes the problem ships inside the library. ConcurrentDictionary — the base of most homemade .NET caches — internally keeps a counter per lock stripe in a plain array (_countPerLock in the source). Under very hot multi-threaded writes those neighbors can end up sharing lines. I have not benchmarked the real-world impact — but the layout is right there in the source, and now you know what to look for.

When you should NOT care

Honesty section. False sharing hurts when the writes are hot — millions of updates per second from several threads. If your cache updates its stats a thousand times per second, you will never notice, and padding everything "just in case" is cargo cult. The rule is the same as always: measure first. The demo above is the measurement; adapt it to your data layout and see if your ratio is 1.0x or 5x.

The actual point

The AI-generated code was not wrong. It compiled, it was race-free, it passed every test I could write for its correctness. An entire code review process could bless it. The 5x was invisible at every layer we normally check.

That is what changed with AI-assisted coding, and it is why hardware fundamentals became more valuable, not less. The model will happily generate a thread-safe counter, a sharded cache, an LRU eviction policy — and none of its correctness guarantees say anything about cache lines. Correctness and mechanical sympathy are different layers. Tests catch the first. Only understanding catches the second.

You do not need to memorize cache sizes. You need to know the crate exists. Keep data that is used together close. Keep data that is written by different threads apart. That one rule, read in both directions, is most of "cache-aware" programming.

The tools write the code now. Knowing why it is slow is still our job.

Originally published on vasyl.blog.

An AI Feature Has No "Tests Pass" Moment. So I Write the Eval First.

Vasyl — Tue, 23 Jun 2026 12:00:00 +0000

I was building an "Ask This Book" feature: readers can ask questions about a book while they're reading it.

One requirement sounded simple:

A reader on chapter 3 must never receive spoilers from chapter 30.

My first instinct was the same as everyone else's: tell the model not to spoil future chapters. Something like:

"Please don't reveal information from chapters the reader hasn't reached yet."

And honestly, it mostly worked.

The problem is that "mostly" is useless. A user only needs one spoiler.

That was the moment I realized the feature had no definition of done.

With normal software, something pushes back. The compiler complains. The tests fail. The types don't line up.

With an LLM feature, none of that happens. The output looks plausible by default — fluent, confident, well formatted — even when it's wrong.

So "it looked right in the demo" quietly becomes the finish line.

That's exactly why I write the eval before I write the feature.

The Eval Is the Specification

Most teams treat evals as QA. Build the feature, ship something that works, add evals later.

I increasingly think that's backwards. For AI systems, the eval is often the only concrete definition of success.

The moment I wrote the spoiler eval, I had to define failure: spoiler leakage must be zero. Not low. Not acceptable. Zero.

And that requirement immediately exposed a problem. No prompt can guarantee zero.

Prompts are probabilistic. Users can phrase questions differently. Models can interpret instructions differently. Future model updates can behave differently. You cannot get a hard guarantee from a soft instruction.

The Eval Changed the Architecture

Once the eval demanded zero spoilers, the solution stopped being a prompt problem. It became a retrieval problem.

Instead of telling the model not to reveal future chapters, I prevented future chapters from entering the context at all:

WHERE chapter_ord <= @maxChapterOrd```

Anything beyond the reader's progress never enters the retrieval set. The model can't leak information it never saw.

And the eval that checks it is just as blunt — a retrieved chunk past the reader's progress is a leak:



```csharp
// One retrieved chunk past the reader's progress = one spoiler leak.
public static int LeakCount(IEnumerable<RetrievedChunk> retrieved, int gateChapterOrd) =>
    retrieved.Count(c => c.ChapterOrd > gateChapterOrd);

Across the adversarial test cases, that number has to be zero. That's the moment the idea really clicked for me: the eval didn't test the design. It produced the design.

A measurable failure condition forced a better architecture than I would have built if I had started with prompt engineering.

The Same Thing Happened to Retrieval Quality

The spoiler requirement wasn't the only eval. I also defined two other targets before building the feature:

Retrieval must surface the correct passage near the top of the results.
Answers must remain grounded in the passages they cite.

Because those requirements were measurable, every change received a verdict instead of an opinion.

A single semantic search wasn't clearing the bar. So I ended up combining two retrieval approaches:

vector search for semantic similarity
full-text search for exact names, phrases, and quotations

The results are fused using Reciprocal Rank Fusion — less mysterious than it sounds. Each chunk scores the sum of 1/(k+rank) across the lists it appears in, so anything ranked highly by both retrievers floats to the top:

// ranked highly by both vector AND lexical -> floats to the top.
scores[item] += 1.0 / (k + i + 1); // i is 0-based; RRF rank is 1-based

I didn't choose hybrid retrieval because it's fashionable. I chose it because it moved the number. The eval said the system wasn't good enough. The architecture changed until it was.

A Note on the Stack

None of this is a no-dependencies flex. The judge that scores grounding is a custom evaluator on Microsoft.Extensions.AI.Evaluation:

public sealed class RubricEvaluator(string id, Rubric rubric) : IEvaluator

I lean on the Microsoft stack on purpose. What I keep hand-rolled is the part that decides quality — the retrieval, the fusion, the spoiler gate. The line I draw isn't "no libraries." It's no agent framework hiding the parts that determine whether the thing actually works.

Eval-First Development

Traditional software development gives us confidence almost for free. Compilers. Type systems. Unit tests. Integration tests.

AI systems don't. The difficult part isn't implementing the feature. The difficult part is defining what "correct" means.

That's why I increasingly think of eval-first development as the AI equivalent of TDD. With traditional software, tests verify the implementation. With AI systems, evals often define the implementation.

Build the feature first and the eval later, and the eval can only grade what you've already built. Build the eval first and it starts shaping the system itself.

It defines done. It tells you when you've regressed. And sometimes it forces a better architecture than the one you originally had in mind.

Otherwise you're not shipping a feature. You're shipping a guess that happened to demo well.

Want to go deeper on evals? I've written a separate, more hands-on series on building production AI on .NET: what evals actually are, error analysis, golden datasets, LLM-as-judge, and evals in CI and production. This post was originally published on my blog.

AI Evals, Part 5: From a Number to a Gate Evals in CI and Production

Vasyl — Wed, 17 Jun 2026 17:43:25 +0000

Part 5, the finale, of a series on building production AI on .NET. We've built the pieces — what evals are, error analysis, golden datasets, and a trustworthy judge. Now we make them earn their keep.

By now you can produce a defensible quality score for an AI feature. But a score you only look at is a vanity metric. The entire point of all that work is to make quality something your engineering process acts on automatically — the same way a failing unit test stops a bad commit. That means two homes for your evals: a gate before you ship, and monitoring after.

Home 1: CI — a safety net against regressions

Because TextStack's judge is a custom IEvaluator on Microsoft.Extensions.AI.Evaluation, an eval is just a dotnet test. The MEAI evaluator emits the rubric's axes plus an overall as numeric metrics, and a quality floor is expressed as a Pass/Fail interpretation on the overall:

// In the evaluator: the overall metric is interpreted Pass/Fail against a floor.
if (overallFloor is { } floor)
    overall.Interpretation = new EvaluationMetricInterpretation(
        RatingFor(score.Mean),
        failed: score.Mean < floor,
        reason: $"floor {floor:0.0} (mean {score.Mean:0.00})");

That catches gross breakage — "something is badly wrong." But the more valuable gate is relative: store a baseline score per feature, and fail the build when a change drops quality by more than a threshold versus that baseline. That turns "did this prompt change help?" into a red/green answer and makes improving a prompt a tight loop — change, run, compare, keep or revert. It's the AI equivalent of TDD.

Honest status from our codebase: the floor and on-demand runs exist today; the automatic baseline-versus-regression gate is the next step. I'm flagging that deliberately, because plenty of "we do eval-driven development" claims are really "we have a number nobody gates on." The hard 80% — the measuring instrument — is built; wiring the ratchet is the lighter remaining 20%.

The constraint CI forces: evals cost money

Every eval case is a real generation plus a real judge call. Running the full suite on every commit is slow and expensive, so evals have to be deliberate. TextStack's are opt-in: tagged so default CI skips them, and they self-skip when the provider isn't configured.

OPENAI_API_KEY=… dotnet test tests/TextStack.AiEvals --filter Category=Eval

Default CI stays green and free; the expensive truth runs on purpose. The pragmatic pattern: a small, cheap subset on pull requests for a fast signal, and the full suite nightly or pre-release. Treat eval spend like any cloud cost — budget it, don't let it run unbounded.

Home 2: Production — monitoring and guardrails

A curated golden set, however good, is a snapshot of inputs you imagined. Production sends inputs you didn't. So the offline gate is only half the system; the other half runs against live traffic.

This is where evals and observability become one thing. Every AI call in TextStack is tagged with its feature and recorded — cost, latency, tokens, errors — and runs persist to an eval_runs table surfaced on an internal /ai-quality dashboard (Traces and Evals tabs), with an admin "Run evals" button to trigger the suite on demand. Because the judge is the same component offline and online, you can sample real outputs per feature and score them with the identical rubric. Two modes fall out of that:

Background monitoring — sample a slice of live outputs, judge them, and watch the score over time to catch drift before users complain.
Guardrails — for high-stakes outputs, judge in the critical path and block, retry, or fall back when a result fails. (Use sparingly: it adds a judge call's worth of latency and cost to the request.)

The flywheel

Put the two homes together and you get a loop that compounds. Production surfaces a new failure mode → you do error analysis on it → it becomes a new golden case → your gate now defends against it → quality climbs → cleaner output produces cleaner traffic. Each turn makes the next regression harder to ship. That continuous-improvement flywheel — not any single dashboard — is the real product of an eval system.

The pitfalls

A number nobody gates on — if a bad score can't fail a build or page someone, it's decoration.
A fixed floor mistaken for a regression gate — a floor catches breakage, not a 2%-worse change. You want both.
Evals on every commit — the bill and the wait will kill the habit; subset on PRs, full suite nightly.
Offline-only — you'll ship regressions from inputs your golden set never imagined.
Guardrails everywhere — judging in the critical path is powerful but costs latency; reserve it for outputs that matter.
Online scores you never read — monitoring you don't look at is just a more expensive log.

The series, in one line each

That's the whole discipline, start to finish:

Evals are the test suite for non-deterministic code — graded judgement over a representative sample.
Error analysis comes first — read your failures and name them; the taxonomy decides what to measure.
The golden set is the ruler — representative, leak-free, fresh, and run through the real prompt and gateway.
The judge is a model too — defensive, dedicated, routed, and validated against humans with Cohen's κ.
A score must become a gate — CI to catch regressions before ship, monitoring to catch drift after.

None of it requires Python or a heavyweight platform. On .NET it's an ILlmService seam, a golden dataset in JSON, a custom IEvaluator on Microsoft.Extensions.AI.Evaluation, and an opt-in test category — built on a real product, in production. Done right, evals turn "I think this AI feature is fine" into "I can prove it, and I'll know the moment it stops being true." That's the difference between shipping AI and gambling with it.

TextStack is a reader that helps you finish the dense technical book you keep quitting — it builds every modern AI primitive (observability, evals, RAG, agents) as a real production feature on .NET. Try it at textstack.app, or read the code at github.com/mrviduus/textstack.

AI Evals, Part 4: LLM-as-Judge, Done Right

Vasyl — Wed, 17 Jun 2026 17:28:22 +0000

Part 4 of a series on building production AI on .NET. We've covered what evals are, error analysis, and golden datasets. Now: how do you turn a paragraph into a number you can trust?

You have a golden dataset and your feature's real output for each case. Now you need a score. But you can't assert == two paragraphs — there's no single right answer, and exact-match comparison is meaningless for prose. String-similarity metrics (BLEU, ROUGE) don't help either; they reward overlapping words, not correct meaning.

The pragmatic answer the field has converged on is LLM-as-judge: use a second, capable model to read the reference and the actual output and score it against a rubric. It's powerful, it scales, and — handled carelessly — it will hand you confident, biased numbers that feel rigorous and aren't. This post is about doing it right.

The basic shape

A judge takes the rubric and an evidence block (the inputs, the reference answer, and the model's actual output), and returns a structured verdict. In TextStack the judge is one feature-agnostic component built on Microsoft.Extensions.AI.Evaluation — Microsoft's official .NET evaluation library — implemented as a custom IEvaluator. The core is a single judge call asking for strict JSON:

var system =
    "You are a strict, fair evaluator of an AI feature's output. " +
    "Score each of three dimensions on an integer scale 1-5 (5 = excellent, 1 = poor):\n" +
    $"- d1 = {rubric.Dim1}\n- d2 = {rubric.Dim2}\n- d3 = {rubric.Dim3}\n" +
    "Return ONLY strict JSON: {\"d1\": int, \"d2\": int, \"d3\": int, \"rationale\": \"...\"}";

The rubric is a parameter, not a hardcode — three named axes passed in per feature. That's what lets one judge score Explain, Translate, distractors, and book metadata, each on the dimensions its own error analysis surfaced (Explain → accuracy / conciseness / usefulness; Translate → accuracy / fluency / register; and so on). One judge, many rubrics.

Three things that separate a toy judge from a production one

Parse defensively. Judges wrap their JSON in prose or code fences no matter how firmly you forbid it. Don't trust the whole string — extract the first {…} span:

var start = raw.IndexOf('{');
var end = raw.LastIndexOf('}');
if (start < 0 || end <= start)
    return new JudgeScore(0, 0, 0, "unparseable: no JSON object");

Fail to a number, not an exception. An unparseable or failed judge call returns a zero score with the reason attached, which drags the run's mean down instead of crashing it. A judge that silently throws is worse than one that scores zero — the zero is a visible signal you can investigate.

Use a dedicated, stronger judge — and route it like everything else. The model that judges should be more capable than the models that generate. TextStack generates features on small, cheap models but judges with a gpt-4.1-class model. And the judge call carries the same eval.judge feature tag and flows through the same gateway as production traffic, so it's traced and cost-accounted like any other call. Evaluating is itself an AI feature; treat it like one.

The biases that quietly wreck your judge

This is the part that separates people who use an LLM judge from people who can trust one. A judge is a language model, and it brings model-shaped biases to grading. Ignore them and your scores are precise and wrong.

Position bias. In pairwise comparisons ("is A or B better?"), judges favour whichever answer appears first (sometimes second) regardless of content. Mitigation: run each comparison both ways and average, or randomise order and watch the swap rate.

Verbosity bias. Judges reliably prefer longer, more elaborate answers even when the extra words add nothing — actively harmful for a feature like Explain whose rubric demands conciseness. Mitigation: name length explicitly in the rubric and watch for score creeping up with token count.

Self-preference bias. A judge scores text from its own model family higher. I'll be concrete about where TextStack sits here: features generated on a local model (distractors, book metadata) are judged cross-family by OpenAI — good, that's independent. But Explain and Translate are generated and judged within the OpenAI family (different sizes — gpt-4.1-nano to generate, gpt-4.1 to judge — but the same lineage), so some self-preference is still in play. The honest read: the absolute number is treated as soft; the deltas between runs are what we trust. A fully independent second judge is on the roadmap.

Sycophancy and scale compression. Judges drift toward agreeable, middling scores, clustering around 3–4 on a 1–5 scale and flattening your signal. Mitigation: anchor each dimension with a concrete description (not just a one-word label), always give the judge the reference answer as a yardstick, and consider a coarser scale if the judge can't use the full range reliably.

Your judge needs its own eval

Here's the step almost everyone skips: validate the judge against humans. You wouldn't ship a feature on an unvalidated model, and a judge is a model — so prove it agrees with human judgement before you trust its scores.

Hand-label a sample of outputs yourself, then measure agreement between you and the judge. The right metric is inter-rater agreement — Cohen's κ (kappa), which corrects for the agreement you'd get by chance — not raw percent-agreement, which flatters you when scores cluster. A judge around κ ≥ 0.6 against human labels is usable; near zero means it's rolling dice and your whole pipeline is theatre. Re-check it whenever you change the judge model or the rubric.

There's a design subtlety worth applying here: treat the judge prompt itself as something you iterate on against a labelled split. Tune the judge prompt on one slice of human-labelled cases, validate κ on a held-out slice — exactly the train/test discipline from the last post, applied one level up. The judge is software; it deserves the same rigour as the feature it grades.

This closes a loop people miss. The golden set evaluates the feature; a human-labelled slice evaluates the judge. Skip the second and you've just moved your trust problem one level up and hidden it from yourself.

The pitfalls

Trusting an unvalidated judge — measure κ against human labels or it's theatre.
Same model generating and judging — self-preference inflates the score; prefer a different (ideally cross-family) judge.
A weak judge model — the judge should be more capable than the generator, not the same one.
Ignoring position/verbosity bias — randomise order, penalise padding, anchor the rubric.
One-word rubric axes — "accuracy" alone means different things to the model each run; describe it concretely.
Throwing on a bad verdict — score it zero and surface it; don't let one parse failure kill the run.

The takeaway

LLM-as-judge is the only practical way to score prose at scale, but a judge is a model with a model's biases — so build it like production code (defensive parsing, a dedicated stronger model, routed and traced) and validate it like a model (human labels, Cohen's κ, a tuned-and-tested judge prompt). Do that and your scores mean something. Skip it and you've automated the production of confident nonsense.

Next, and last in the series: from a number to a gate — wiring evals into CI and online monitoring so quality regressions turn the build red, on Microsoft.Extensions.AI.Evaluation, without bankrupting your pipeline.

AI Evals, Part 3: Golden Datasets That Dont Lie

Vasyl — Tue, 16 Jun 2026 21:28:24 +0000

Part 3 of a series on building production AI on .NET. Part 1 was the overview; Part 2 was error analysis. Now we turn the failure taxonomy you built into something you can measure against — without quietly fooling yourself.

A golden dataset is a set of representative inputs, each paired with a reference answer a knowledgeable human would accept. It's the ruler you hold every model output against. And it is, in my experience, the single most important and most neglected asset in an eval pipeline — because a sloppy ruler doesn't announce itself. Your scores still come out green. They're just measuring the wrong thing.

This post is about building a golden set that tells the truth.

What it looks like in practice

In TextStack, each AI feature has ~30 hand-curated cases stored as plain JSON, loaded at runtime into a typed record that mirrors exactly what the production endpoint receives:

public record ExplainGolden(
    string Word,
    string Sentence,
    string? Genre,
    string TargetLang,
    string ExpectedExplanation);

Plain JSON on disk, deserialised case-insensitively. No database, no platform lock-in — the dataset is a checked-in artifact you can diff in code review:

var goldens = GoldenData.Load<ExplainGolden>("explain.json");

The format is the easy part. The honesty is in four properties of the content.

1. Representativeness — mirror reality, not the demo

Your set should reflect the real distribution of inputs your feature meets in production, including the hard, weird, and adversarial cases. This is where Part 2 pays off: the failure taxonomy tells you which kinds of input break things, so you deliberately stock the set with them.

The opposite — a set of only easy, happy-path cases — is the most common way an eval lies. The model aces them, your average climbs, and meanwhile the inputs that actually matter never get measured. Stratify on purpose: domains, lengths, languages, edge cases. For TextStack's Explain set that means technical passages and casual prose, common words and rare ones, several target languages — not thirty variations of the same easy lookup.

2. Reference quality — the ceiling you measure against

The reference answer defines what "good" means for that case, so a lazy reference caps the meaning of your whole score. If the reference for explaining idempotent is a paraphrased dictionary entry, your judge will happily reward dictionary entries — the exact failure mode you were trying to eliminate.

References should be written or vetted by someone who understands the domain. For Explain, that means genuinely good in-context explanations: what the word means here, in this sentence, the way you'd want it explained to you. The reference is the bar; set it where you actually want the product.

3. Leakage — keep a real train/test split

Here's the subtle statistical sin. If you tune your prompt against the same cases you score against, you're overfitting to the test, and your number is fiction — you've optimised for those thirty examples, not for the feature. It's the prompt-engineering version of training on your test set.

Keep a slice you never look at while iterating. Tune on one part; report on the held-out part. This feels heavy for thirty cases, but the discipline is what keeps the score meaningful as you iterate. The split is just as real for prompts as it is for model weights.

4. Size and freshness — a floor, and a living asset

Thirty cases is a deliberate floor, not a target: enough to catch gross regressions cheaply, small enough to run often and to keep every reference high quality. (It's statistically thin for detecting small changes — that's the next post's problem.) More important than size is that the set is alive: every new failure mode you find in production should earn a new case. A golden set that never changes slowly stops resembling reality, and a stale ruler is a lying ruler.

When you genuinely lack real examples — a brand-new feature with no traffic — you can bootstrap with synthetic cases (have a strong model generate realistic inputs across your taxonomy's dimensions). It's a legitimate starting point, but treat it as scaffolding: replace synthetic cases with real ones as traffic arrives, because real users are more creative than any generator.

The silent killer: dataset drift from production

Now the trap that quietly invalidates an otherwise perfect golden set, and the one I'd most want a reviewer to check for.

You write your feature's prompt in the API endpoint. You write the eval, and — naturally — you write the prompt again in the test. Two copies. Someone tweaks the production prompt for a hotfix and doesn't touch the test copy. From that moment your eval measures a prompt that no longer exists in production. The score stays green; the product changed underneath it. Nobody notices, because the test reports with total confidence.

The fix is structural, not disciplinary: extract the prompt into one builder that both production and the eval call. There is no second copy to drift.

// Built once, called by BOTH the endpoint and the eval — they cannot disagree.
public static class ExplainPrompt
{
    public static string BuildSystemPrompt(string? genre, string targetLang) => /* ... */;
    public static string BuildUserPrompt(string word, string sentence) => /* ... */;
}

The eval's case-to-request mapping wires that shared builder straight in, and crucially the request goes through the same model gateway production uses, selected by the feature's tag:

private static LlmRequest ToRequest(ExplainGolden g) => new(
    SystemPrompt: ExplainPrompt.BuildSystemPrompt(g.Genre, g.TargetLang),
    Messages: [new LlmMessage("user", ExplainPrompt.BuildUserPrompt(g.Word, g.Sentence))],
    MaxOutputTokens: 500,
    FeatureTag: "explain"); // same routing, same model, same path as prod

If you remember one thing from this post: an eval that runs a copy of the prompt is worse than no eval, because it manufactures false confidence. Same prompt, same gateway, same path — or you're measuring a ghost.

The pitfalls

A happy-path-only set — the score rises while the product falls. Stock it from your failure taxonomy.
Weak reference answers — they cap your score's meaning and can reward the very failure you're chasing.
Train/test leakage — tuning and scoring on the same cases overfits to fiction.
A frozen set — inputs drift; a dataset that never grows slowly measures a product that no longer exists.
Synthetic-forever — fine to bootstrap, dangerous to rely on; real traffic is weirder.
A duplicated prompt — the drift trap. One shared builder, through the real gateway.

The takeaway

A golden dataset is not a formality you generate once and forget. It's a carefully curated, honestly-split, continuously-refreshed ruler — and it has to run the real prompt through the real path or it measures nothing. Get the dataset right and every downstream number means something. Get it wrong and you've built an instrument that lies to you in green.

Next in the series: LLM-as-judge, done right — how to turn a paragraph into a trustworthy number, the biases that wreck judges, and why your judge needs its own eval.

AI Evals, Part 2: Error Analysis The Unglamorous Superpower Behind Good Evals

Vasyl — Fri, 12 Jun 2026 22:46:23 +0000

Part 2 of a series on building production AI on .NET. Part 1 covered what evals are and the Analyze → Measure → Improve lifecycle. This post is about the step everyone wants to skip: **Analyze.

When a team decides to "take evals seriously," the first thing they usually do is wrong. They open a dashboard tool, wire up a generic "correctness" score, and watch a number. It feels productive. It produces a chart. And it tells them almost nothing, because they skipped the step that decides what the chart should even measure.

That step is error analysis: reading your AI's actual outputs and naming, precisely, the ways they go wrong. It's unglamorous — no library, no dashboard, just you and a few dozen real examples. It is also, by a wide margin, the highest-leverage thing you will do in evals: error analysis is where the signal comes from. Everything downstream is just operationalising what you find here.

Why you can't skip straight to metrics

There's a gap between you and your running system that's easy to underestimate. Thousands of inputs flow through your AI feature daily, in shapes you never anticipated, and you have no realistic way to see them at scale. Call it the comprehension gap — the distance between the developer and a true understanding of what the data and the model are actually doing.

Metrics don't bridge that gulf; they presuppose it's already bridged. To measure "conciseness" you must first have noticed that verbosity is a failure mode worth caring about. If you pick your metrics before you've read your data, you're measuring your assumptions, not your product. The classic result: a dashboard glowing green while users quietly churn over a problem your metrics were never designed to catch.

Error analysis is how you cross the gulf. You trade scale for truth — you can't read everything, so you read a sample, carefully.

How error analysis actually works

It's a three-move loop, and the moves are deliberately low-tech.

1. Get a starting dataset and read it. Pull a sample of real (or realistic) outputs — 50 to 100 is plenty to start. Not the happy-path demo cases; the real distribution, including the weird inputs. Then actually read them. Slowly.

2. Open-code the failures. For each output that's wrong, write a short, free-text note describing what specifically is wrong — in your own words, no fixed categories yet. "Explained the word using a dictionary definition instead of the meaning it has in this sentence." "Translation is correct but the tone is far too formal for a casual chat." "The quiz distractor is so obviously wrong it gives the answer away." This is open coding: you're labelling reality, not forcing it into boxes.

3. Cluster the notes into a taxonomy. Once you have 40–50 notes, patterns emerge. Group them. Those groups are your failure taxonomy — a ranked list of how your feature fails, with rough frequencies. Now you know what to fix first (the common, severe modes) and, crucially, what your metrics should measure.

That's the whole secret. The taxonomy is the output, and it's worth more than any single score, because every later step — the rubric, the golden set, the judge — is downstream of it.

A mindset note: be a detective, not a judge (yet)

The hard part of error analysis isn't mechanical, it's psychological. You will be tempted to immediately assign a 1–5 score, or to jump to "the fix is to add a line to the prompt." Resist both. Scoring too early collapses rich information ("it's a 2") into a number that hides why. Fixing too early means you patch the first failure you see instead of the most common one.

Stay descriptive for as long as you can. Your only job in this phase is to understand and categorise. Judgement and repair come later.

A second trap is doing it alone. When two people label the same outputs, they disagree — and the disagreements are gold, because they reveal that "good" isn't actually defined yet. A short alignment session to resolve them sharpens your definition of quality before you bake it into a rubric. (Solo founders can approximate this by labelling, sleeping on it, and re-labelling cold.)

How error analysis shaped TextStack's evals

This isn't abstract for us. TextStack has seven AI surfaces, and every rubric we score against came directly out of reading failures, not out of a generic template.

Take Explain (tap a word, get a short in-context explanation). Reading real outputs surfaced a recurring failure: the model would produce a competent dictionary definition while ignoring the sentence the reader was actually looking at — useless for someone trying to understand this passage. That single observation is why the Explain rubric scores accuracy in context and usefulness to a learner as distinct axes, and explicitly penalises dictionary boilerplate under conciseness. The rubric is a direct transcription of the taxonomy.

Other surfaces produced different taxonomies, and therefore different axes:

Translate kept failing on register — accurate but wrong formality — so register became its own scored dimension alongside accuracy and fluency.
Vocabulary distractors (wrong answers in a quiz) failed by being implausible (too obviously wrong) or too similar to the right answer, so the rubric scores plausibility, distinctness, and difficulty.

We didn't invent those dimensions in a meeting. We read outputs until the dimensions were obvious. And because every AI call is traced and viewable on an internal /ai-quality page, error analysis isn't a one-time exercise — new production failures keep feeding new categories back into the taxonomy.

The pitfalls

Scoring before describing. A number erases the why. Open-code in words first.
Vague categories. "Bad output" isn't a category; "ignored the sentence context" is. Specific enough to act on.
Too small a sample, or only the easy cases. If you only read successes, you'll conclude everything is fine.
Fixing during analysis. Note the failure, move on. Triage after you can see the whole picture.
Labelling solo with no calibration. Disagreement is information; surface it before it hardens into a bad rubric.
Doing it once. Inputs drift. The taxonomy is a living document, refreshed from real traffic.

The takeaway

Error analysis is the part of evals with no tooling, no dashboard, and the highest payoff — and that's exactly why it gets skipped. Read your failures, name them in plain language, and cluster them into a taxonomy. That taxonomy tells you what to fix and what to measure. Skip it and you'll build a beautiful measurement system pointed at the wrong target.

Next in the series: golden datasets that don't lie — turning your taxonomy into a curated set of cases you can score against, without quietly fooling yourself.