A third of the measurement went dark and the report stayed green

Originally published at aideazz.xyz — cross-posted here with canonical link.

A field note from the AIdeazz AI Lab — a real incident on a live production system, written up from the logs. August 27, 2026.

A deprecated model made one of three AI answer engines return 404 for about three weeks. The weekly cron kept exiting 0 and reporting 0% citations, and the only tell was the denominator..

What it looked like from outside

The weekly AI citation probe reported aideazz.xyz cited in 0 of 12 AI answers, 0%. It had reported 0 of 18 on 3 August. Both runs named the same three engines in the same order. Nothing alerted, the cron exited 0 every Monday, and the stored trend rendered a clean flat line at zero. The 0% was being read as a visibility problem and planned against as one.

What was actually happening

OpenAI deprecated gpt-4o-search-preview, so every request on the openai-search leg returned HTTP 404. The engine caught the error per prompt, contributed zero measured probes, and the run summary averaged over the engines that did answer -- so a dead engine and an engine that genuinely found nothing produced the identical output. Only the all-engines-dead case was loud; a partial blackout had no signal at all. Reading the failure as a visibility problem instead of an instrument problem was the actual cost: three weeks of a metric nobody could act on.

The fix

Probed the provider directly rather than trusting configuration or the provider's own catalogue. gpt-5-search-api answers; gpt-4o-mini-search-preview is STILL LISTED in OpenAI's /v1/models endpoint and 404s when called, so the vendor's published inventory is not evidence either. Changed the default model in the tracker source rather than patching an env var on the box, so a rebuild cannot lose it. Then fixed the class of bug, not just the instance -- summarize() now names any engine that held a valid key and still measured zero probes as BLIND in the summary line, so the next partial blackout announces itself instead of averaging away.

How I know it worked

Live 3-engine run after deploy -- google-ai-overview 0/5, gemini-grounded 0/6, openai-search 0/6, total measured 17 (was 12), named without a link in 12%. Direct provider probe -- gpt-5-search-api returned a real grounded answer with 10 to 22 sources per prompt; gpt-4o-mini-search-preview returned model_not_found despite being listed. Deployed file checked on the box -- deprecated string count 0, new default count 1, BLIND warning count 1. pm2 restart cto-aipa --update-env, process uptime 4s, status online. Committed to main so a rebuild reproduces it.

The rule this earned

A zero and an un-measurable are different facts and must not share an output shape. Watch the denominator, not the value -- coverage collapsed from 18 to 12 while the headline stayed 0% and the engine list stayed unchanged. Any component that held a key and still measured nothing must say so by name. And capability discovery is not capability: a provider listing a model proves only that the list has not been updated, so probe the thing itself before believing either your config or their catalogue.

The named concepts behind it

Naming a failure mode is what makes it possible to recognise the same shape somewhere new, before it costs another weekend.

Null is not zero

A measurement that could not be taken is not a measurement of zero.

"We were cited 0 times" and "we could not check whether we were cited" are completely different facts about the world. One is a finding you plan against. The other is an outage wearing a finding's clothes. Systems collapse them constantly, because both come out of the pipe as the number 0.

This is not the same failure as [[silent-failure]], and the difference matters. A silent failure means something broke and swallowed the error. Null-is-not-zero can happen with nothing broken at all. Google Analytics reported form_submit: 0 on a site whose forms work perfectly -- the forms call preventDefault() and post over fetch, and the browser's automatic form tracking only fires on native submits. Nothing errored. The event was never observable. The 0 was structurally guaranteed and read for months as "nobody is converting".

A zero is dangerous precisely because it is plausible. A crash gets investigated. A 0% gets put in a report, then in a roadmap, and the team goes off to fix a problem that may not exist while the real one -- that the instrument is blind -- goes unexamined.

The tells, in order of usefulness:

• Watch the denominator, not the value. The headline number can stay flat while coverage silently collapses underneath it. If a report says 0 of 17 one week and 0 of 12 the next, the story is not "still zero", it is "a third of the measurement disappeared".
• Count the sources that answered, not the sources configured. A run listing three engines proves nothing about how many replied.
• Ask whether the event is even emittable. Before trusting a zero, confirm the thing being counted has a code path that can fire. Many do not.

The defence is to make coverage a first-class output. Never report a metric without reporting how much of the intended surface it was computed over, and make any component that measured nothing say so by name rather than contributing a harmless-looking 0. A run that measured nothing should be shaped differently from a run that measured zero -- loud, distinct, and impossible to average away.

The discipline this earns is the same one in [[verify-from-logs]], one step earlier: before you trust what the number says, prove the instrument could see.

Silent failure

The system did something reasonable, and told nobody.

The most expensive bug class there is, because the clock keeps running while everyone assumes things are fine.

A silent failure is not a crash. A crash is loud and gets fixed. A silent failure is a component making a defensible local decision -- drop this message, skip this record, return an empty string -- that nobody downstream is told about. From the outside, a system that is working perfectly and a system that is completely dead can produce the identical observation: nothing happened.

The defence is not "add more logging". It is to make the healthy state provable, so that "nothing happened" can be distinguished from "nothing was supposed to happen". Two things do that:

• Log the outcome, not the attempt. "sending notification" tells you nothing. "notification DELIVERED (id 4661)" versus "notification REJECTED 400" tells you everything.
• Run a canary. A synthetic transaction pushed through the real path on a schedule, which shouts when it does not come out the far end. Without one, you are relying on a customer to report your outage.

Verify from logs, not config

Configuration tells you what somebody intended. Logs tell you what happened.

A setting, an environment variable or a present API key is a statement of intent. It is evidence that somebody meant for a behaviour to occur. It is not evidence that the behaviour occurs.

The gap between the two is where the longest outages live, because reading the configuration feels like verification. It produces confident, wrong statements: the key is set, so the provider works; the schedule says every fifteen minutes, so it runs every fifteen minutes; the file was deployed, so the new code is running.

Each of those has a cheap, decisive check that costs seconds:

• Probe the dependency, do not read its credential. A key that exists proves nothing about the balance behind it.
• Grep for the action line, not the setup line. A startup banner proves the process started, not that it ever did its work.
• Compare timestamps after a deploy. If the running process is older than the file on disk, it is still executing the previous version from memory.

The rule this earns: never report a system's behaviour from its configuration. Grep the line that proves the behaviour happened, and quote it.

Liveness is not correctness

A dead job announces itself. A job that runs perfectly and emits slightly wrong output never will.

Almost every check you own measures liveness: did it run, did it return, did it exit zero, did it publish. Almost none measures correctness: was the thing it produced the right thing. These are different properties, and the gap between them is where the expensive incidents live.

The asymmetry is what makes this dangerous. A job that stops firing is loud — the output is missing, someone notices within a day. A job that fires on schedule and produces output that is subtly wrong is silent, and it stays silent for as long as nobody reads the output, because every signal you have is reporting the truth. The scheduler really did fire. The API really did return 200. The file really was written. Each check passes honestly while the only thing that matters fails.

Two shapes to watch for:

1. The safety net that adjusts instead of refusing. A guard catches a bad condition, then modifies the input so the operation can proceed — renaming a colliding key, truncating an over-long field, coercing a bad type. The error disappears from the logs and the bad condition ships anyway. A guard that never refuses is not a control; it is a laundering step, converting a real signal into a clean log line. Prefer failing closed: a skipped run is cheap and visible, a wrong run is expensive and invisible.
2. The record that drifts from the reality. Any check that compares against a cache, a state file, or a local ledger is only as good as that memory. When the memory can be truncated by a restart, a fresh machine, or a path that writes to one place and reads from another, the check degrades quietly and keeps returning "fine". Seed the memory from the artifacts themselves wherever you can, and periodically assert that the two still agree.

The practical defence is to add one check that reads the output rather than the exit code, and to make it something a human would actually notice — a count that should be stable, a uniqueness constraint, a spot comparison against what shipped last time. You are not trying to verify everything. You are trying to have at least one signal that fails when the job succeeds incorrectly.

---

This note is one entry in a running wiki of production engineering lessons — every concept linked to the incident that taught it — at aideazz.xyz/ai-ops-wiki.html.

No customer data, credentials, hostnames or internal record identifiers appear in these write-ups.