DEV Community: Panagiotis Gkilis

I Assumed My Retriever Failed at Stage One. The Bigger Failure Was at Stage Three.

Panagiotis Gkilis — Mon, 31 Aug 2026 16:38:31 +0000

Hierarchical retrieval is supposed to help on long documents. Pick the right chapter, then search inside it. Narrow the haystack, then find the needle.

On my long-book benchmark it came third of five, behind a chapter-summary chain and flat chunk RAG, and only barely ahead of naively grabbing the end of the book.

method	context precision	context recall
`naive_first_context`	0.1475	0.1458
`naive_last_context`	0.4150	0.3302
`flat_chunk_rag`	0.3375	0.4302
`chapter_summary_chain`	0.4000	0.4771
`hierarchical_book_rag`	0.3475	0.3365

My hypothesis was error compounding: if stage one picks the wrong chapter, every later stage searches the wrong text. That turned out to be true and incomplete — which is worse than being wrong, because it would have sent me to fix the right thing and stop there.

The ablation

Same corpus (240,767 words), same 80 gold questions, five variants, 400 scored rows. Two of the variants are given the correct chapter — diagnostics, not deployable methods — purely to measure how much is lost before versus after chapter selection.

method	recall	precision	hit@1	hit@3	hit@5
`hier_current`	0.3365	0.3475	0.2000	0.3875	0.4375
`hier_no_neighbors`	0.4771	0.4000	0.2000	0.3875	0.4375
`chapter_summary_chain`	0.4771	0.4000	0.2000	0.3875	0.4375
`hier_oracle_chapter`	0.7844	0.6796	1.0000	1.0000	1.0000
`hier_oracle_chapter_neighbors`	0.7688	0.6925	1.0000	1.0000	1.0000

Deleting a feature closed the entire gap. hier_no_neighbors lands on 0.4771 / 0.4000 — the same numbers as the baseline, to four decimals. The method I thought was structurally worse wasn't worse. It was carrying a stage that was hurting it.

And look at hit@k. Identical across the first three rows. Chapter selection did not change at all between hier_current and hier_no_neighbors — same code. Every point of recall that moved, moved downstream, in the stage that pads results with adjacent chunks.

Counting failures directly

failure type	count
`neighbor_dilution`	32
`wrong_chapter`	27
`right_chapter_wrong_chunk`	7
`ok`	14

My hypothesis accounted for 27 questions. The stage I hadn't suspected accounted for 32. I would have fixed chapter routing, seen a real improvement, and never looked at expansion — because the improvement would have confirmed the theory I walked in with.

Neighbour expansion is not simply bad

With the oracle chapter, adding neighbours moves recall 0.7844 → 0.7688 and precision 0.6796 → 0.6925. Recall down slightly, precision up. When you're already in the right chapter, expansion is roughly a wash and can help precision.

The damage happens when expansion runs on top of an uncertain chapter choice, crowding out the good chunks you did find with text that is only adjacent, not relevant. It is a stage whose value depends on the confidence of the stage above it — exactly the interaction a single end-to-end score cannot show you.

So the finding is not "turn off neighbour expansion." It is treat it as a tunable stage rather than a default, conditioned on router confidence.

The sequence that worked

Disable each optional stage in turn. Cheap, and it found the larger of my two problems.
Insert an oracle at each boundary. Not deployable, but it partitions the loss into before/after.
Only then attribute the failure.

If I had skipped to step three, I'd have published "hierarchical RAG underperforms chapter-summary retrieval on long narrative corpora" — well-supported by my headline numbers, and wrong about the cause.

What this does not show

One private narrative corpus, 80 gold questions written from the corpus rather than by independent annotators. Evidence-term overlap scoring, which is a lightweight audit signal and not full semantic correctness. No confidence intervals — the package didn't compute them and I won't imply precision I didn't measure. The oracle variants read gold labels and are not production-realistic. This is not a universal rule against hierarchical retrieval.

Paper: Diagnosing Hierarchical Retrieval Failure in Long-Document RAG
DOI: 10.5281/zenodo.20692450

One Integer Deleted the Stop Token From My Loss. The Curve Never Noticed.

Panagiotis Gkilis — Mon, 31 Aug 2026 16:37:39 +0000

For a long time I had a model that trained cleanly and produced nothing usable. The loss fell. Gradients were finite. Nothing crashed. It simply never learned to stop — every generation ran to the length cap and got truncated.

The cause was two lines of my own source that were individually correct.

The collision

In a neural codec language model the audio vocabulary has a fixed size, and the stop token is one extra class on top. So the output layer is one wider than the codebook:

nn.Linear(d_model, NUM_AUDIO_TOKENS + 1)   # 1025 classes: 0..1023 audio, 1024 = EOS
eos_id = NUM_AUDIO_TOKENS                  # 1024

Correct. EOS is the last class and the layer has room for it.

Then, elsewhere, the loss:

F.cross_entropy(logits, targets, ignore_index=NUM_AUDIO_TOKENS)

Also reasonable on its own. ignore_index is how you skip padding.

But the sentinel and the stop token are the same integer. Every position whose target was "stop" was discarded before the loss was computed. Not down-weighted — removed. The model was never once shown an example of stopping, across every epoch it ever ran.

PyTorch defaults ignore_index to -100 precisely because it must be a value that can never be a real class. Replace it with a real vocabulary constant and that guarantee is gone, silently: shapes valid, loss finite, run healthy.

The curves are identical

Minimal reproduction, two arms differing only in the sentinel value.

The broken arm finished at 0.0035. The fixed arm finished at 0.0034.

Same curve to any human, any dashboard, any threshold you would write. One has a working objective and one has an objective with a hole in it, and the loss cannot distinguish them — because the loss is computed over what survived the mask. A metric cannot report on examples it never received.

The second half: lowest loss, worse model

200 epochs at lr = 2e-5, evaluated every 50 on a fixed split held out by utterance (n = 32). Rank is the stop token's position among 1025 classes at the true terminal frame.

epoch	training loss	mean P(stop)	argmax = stop	self-terminated
50	2.199	0.4218	16/32	2/8
100	1.628	0.4655	18/32	6/8
150	1.302	0.2159	8/32	3/8
200	1.429	0.1818	5/32	3/8

Between epochs 100 and 150 the training loss improved by 20% while mean P(stop) fell 54%, top-1 stop accuracy went 18/32 to 8/32, and autonomous termination halved.

Selecting by lowest training loss returns epoch 150. The model that terminates reliably is epoch 100. The reversal was observed independently in a shorter run, which is why I am willing to state it.

"Save the checkpoint with the lowest validation loss" is the default in more or less every training script in existence, including mine. On this run it was actively the wrong rule, and the number it optimised looked better the whole way down.

For completeness, what fixing the collision bought on the real model, same held-out split:

checkpoint	mean P(stop)	argmax = stop	rank
random initialisation	0.001848	0/32	111.6
after correction	0.4655	18/32	2.1

End-to-end synthesis then terminated on its own at frame 203 against a 350-frame ceiling. Before the fix that was impossible by two independent mechanisms: the training-time collision above, and an inference-time mask that set the logit of every index at or beyond the codebook size — including end-of-sequence — to negative infinity before sampling.

One honest loose end: P(stop) plateaus in the range 0.35–0.47 across two learning rates and a 5.9x increase in training data (224 to 1313 utterances, with speaker, language, emotion and the held-out split all held constant). So the plateau is not a data-quantity limit. I have no confirmed explanation for it. Terminal timing in speech is genuinely ambiguous, and hedging with the stop token ranked second of 1025 may simply be correct behaviour.

The same integer, safe one stage over

The detail I find most instructive: the identical line is harmless in the next stage of the same model.

The autoregressive stage predicts the first codebook plus EOS — 1025 classes, so 1024 is a real class and using it as a sentinel is fatal. The non-autoregressive stage predicts audio codes only — 1024 classes, so 1024 is out of range and the exact same ignore_index is correct.

One + 1 in a different file decides whether that line destroys your objective. Both stages read identically at the call site. That is not a mistake you catch by reading carefully; it is a mistake you catch by checking a relationship between two numbers that never appear together.

Which is what a linter is for

This is now two rules in trainproof, my linter for training runs:

Sentinel collision. Compare the output layer's class count against ignore_index. If the sentinel is a valid class, fail. If it sits exactly one past the end, say so explicitly — because the same integer is fatal one class earlier, and that distinction deserves to be visible rather than silently passed.
Dead class. Accumulate which classes ever reach the loss as a positive target during the first epoch, then flag any class the output layer can emit but that never once appears as an answer.

My own two stages are the regression fixture — the fatal case and its safe twin, one integer apart. Not a synthetic example.

The design decision worth stating: the dead-class rule only fires when coverage is already broad and few classes are missing. One unseen class out of 1025 is a structural exclusion. Nine hundred unseen is a small sample. Without that guard the check screams on every short run and gets switched off — which is how good checks die.

If you take one thing from this

I do not think this is rare. Any codebase where a padding sentinel, an end-of-sequence id and a vocabulary size are all defined as named constants in different files can produce it, and none of your instrumentation will complain.

If you fine-tune anything with a custom ignore_index, go and check it against your output layer's width right now. It takes thirty seconds and the failure mode is completely silent.

Paper: The Loss Curve Is Not a Sufficient Statistic — Silent Objective Failures from Sentinel-Class Collisions in Neural Codec Language Models

DOI: 10.5281/zenodo.21864658

pip install trainproof — GitHub, MIT

Fourteen Speaker Encoders Heard the Same Voice. Their Error Rates Differed Five-Fold.

Panagiotis Gkilis — Mon, 31 Aug 2026 16:37:35 +0000

A speaker encoder turns a few seconds of speech into a vector, and the distance between two vectors is treated as an answer to "are these the same person." That number gets used for authentication, for voice-cloning pipelines, and increasingly as the yardstick in papers asking whether a synthetic voice preserved someone's identity.

The number is not neutral. It is a reading from an instrument, and I wanted to know how much the instrument was contributing.

So I ran fourteen of them over an identical, frozen list of 11,935 trials.

The corpus is why the question is answerable

Most emotional-speech corpora use different sentences for different emotions. A model can then learn angry vocabulary instead of angry delivery, and you cannot separate them afterwards.

This one can't make that mistake. Eight speakers each recorded the same 1,360 sentences in six phonation states — neutral, happy, angry, scared, shouting, whisper. Lexical content is fixed while phonation varies.

Enrolment is neutral speech. The test is whether the encoder still recognises the person while they shout or whisper. The trial list was frozen in writing before a single model loaded, and every encoder scored the identical list — so every comparison is paired.

The headline

Equal error rate spans 0.047 to 0.233 across the panel — same trials, same audio, nothing varying but the encoder. 29 of 91 pairwise comparisons survive Holm–Bonferroni correction.

If you are choosing an encoder for expressive audio, that five-fold spread is your decision. It is larger than most architectural differences people argue about.

CAMPPlus is last, without exception

Highest error of the fourteen, all 13 of its comparisons surviving family-wise correction — for every one of the eight speakers and every one of the six conditions. Margins run from +0.087 [+0.064, +0.127] against ECAPA-TDNN to +0.185 [+0.150, +0.215] against the lowest-error encoder.

The mechanism is the useful part. Its genuine scores are unremarkable; its impostor scores are the problem. It places different people at cosine 0.28–0.34 where ECAPA-TDNN and ReDimNet place them near 0.034. That is a compressed embedding space — different speakers land close together.

It is visible at neutral speech too (0.031 EER where several ReDimNet checkpoints reach 0.000). Expressive speech doesn't cause it. Expressive speech makes it visible.

A widely used open-source TTS system conditions on CAMPPlus. To be precise: that is a verified fact about its source code, not a measurement of its audio. I measured encoders in isolation on human recordings. Whether that choice degrades output is a separate experiment I have not run.

Parameter count does not order the panel

Spearman ρ(parameters, expressive EER) = +0.135 over fourteen encoders. The 20.8M model ranks 13th of 14. A 4.81M model ranks 1st. Published VoxCeleb1-O position doesn't transfer either — that leaderboard is computed on calm read speech, which is not the condition failing you.

I retracted my claim about the winner, incidentally. It survives as an argmin (selected in 89.4% of speaker resamples, leading all eight leave-one-speaker-out refits) but after correction it is separated from only 7 of 13. The defensible wording is lowest observed error.

A clean pre-registered negative

Spectral denoising does not materially change the measurement: |ΔEER| ≤ 0.039 over 10,765 twinned trials, and whispered speech — where removing aspiration noise was the stated worry — is among the least affected.

Ten retracted claims, published on purpose

An 11.7 dB spectral effect, asserted with an instrument whose F0 artifact budget is 12.09 dB. Sweeping F0 with the spectral envelope held fixed moved the descriptor further than the claimed effect. It was never a measurement.
"Shouting does not transfer." Judged on RMS after peak-normalising every clip, so the figure was crest factor, not loudness.
A tie-break defect in the EER estimator. argmin returns the first minimiser rather than the balanced operating point — invisible at unit weights, common under the integer speaker multiplicities a bootstrap runs at. Fixing it moved one comparison across Holm, 28 → 29 of 91. The correction favoured the paper, which is why it is published rather than quietly kept.

The rule I have now paid for three times: validate the instrument before measuring with it.

Statistics

The trial list is a directed graph on 8 speakers — genuine trials are self-loops, impostor trials the 56 ordered edges. The independent unit is the speaker, on both sides. So the resample is a vertex bootstrap: draw 8 vertices with replacement, take the induced sub-multigraph, weight the impostor edge by the product of multiplicities.

An earlier scheme resampled enrolment speakers only, dropping a speaker as enrollee while leaving them in as impostor. Fixing it widened marginal intervals (width ratio median 1.11, max 1.58) and cost two already-written claims. Paired differences barely moved, because both encoders score the same impostor clips and the dependence cancels in the difference.

B = 20,000, seed fixed, families declared before testing, Holm–Bonferroni at α = 0.05.

The companion result, which is the cheap fix

Enrolling on expressive material instead of neutral-only recovers the loss on all 14 encoders, every one surviving Holm.

The mechanism is the opposite of the intuitive one. Enrolling on a single expressive state helps that state and slightly harms the others — transfer-matrix diagonal median −0.082, off-diagonal median +0.010. Guessing which state a user will be in, and guessing wrong, is worse than enrolling on calm speech.

A pre-committed control ruled out the competing explanation: acoustic diversity alone delivers +0.002 where composition delivers −0.065. It is coverage, not expressiveness. Spread enrolment thinly across states.

What it does not claim

Nothing about any TTS system's output quality. Eight speakers, one studio, one recording chain. Enrolment and test share a session, so absolute error rates are optimistic and not comparable to VoxCeleb — only relative comparisons on identical trials are licensed. Architecture is fully confounded with training corpus; separating them requires training.

The generalisable part

An established metric is not automatically a neutral one. An encoder, an evaluator, a classifier, an LLM judge — any of them can become part of the experiment rather than a window onto it.

Before asking whether a system preserves identity under some transformation, ask whether your instrument has been validated under that transformation. Otherwise you attribute the encoder's behaviour to the speaker, and the number looks perfectly reasonable while you do it.

Paper: Speaker Encoders Disagree About Who You Are When You Shout — Zenodo, CC-BY-4.0.
DOI: 10.5281/zenodo.22158030
Code, data and the full decision record: github.com/Mormolykos/speaker-encoders-disagree
The eval library the analysis checks itself against: pip install spkproof

Two of My Instruments Were Wrong, in Opposite Directions

Panagiotis Gkilis — Fri, 28 Aug 2026 13:30:49 +0000

On 28 August 2026 I audited 38 live pages and fixed some ordinary defects. The two things worth writing down both came from my own tooling being broken. One script reported three pages I had just fixed as still broken, because it did not recognise a relative link. One date parser had thrown an exception on every crawl-statistics read since it was written, so the report that answers "is the crawler arriving at all" had never once returned a number.

This is a note about measurement rather than about search engines. The setting is an indexing problem on one of my hosts, but the transferable part is that I spent a day acting on numbers produced by instruments I had never checked, and both were wrong — one in each direction. One made a working fix look broken. One made a working crawler invisible.

Crawling and indexing are not the same thing

Everything below depends on this, and most writing on the subject collapses it into one word.

Crawling is retrieval. A crawler requests a URL and receives bytes. That is all it means. A page can be crawled every day for a year and appear in no search result ever.

Indexing is selection. Having retrieved the page, the engine decides whether to keep it in a form it will serve. That is a judgement, each engine makes it independently, and none of them will tell you the reasoning.

So there are at least three states worth separating: never fetched, fetched and declined, and fetched and kept. "My pages aren't indexed" describes the middle two identically, and their causes have nothing in common.

The audit, kept in proportion

On 28 August 2026 I fetched every URL in the sitemap — 38 pages on the affected host, 20 on my notebook site — over plain HTTP with no JavaScript executed, and produced a number per page rather than a verdict. That rule exists because an earlier audit returned "all clean" for months while the homepage was handing crawlers a fraction of its navigation in a form they could use.

It found real defects. Five pages had zero followable links: a crawler arriving there could go nowhere. The worst had two navigation controls, both <button onclick> — usable with a mouse, invisible to a crawler. Three more bound their only navigation to a JavaScript click handler, which has the same effect.

38 pages, fetched live	Before 28 Aug	After 28 Aug
Pages with zero followable links	5	0
Navigation a crawler cannot follow	2	0
Pages with an incorrect `h1` count	1	0
Missing robots directive	1	0
Canonical problems	0	0
Images with no `alt` attribute (of 199)	0	0
Non-200 responses	0	0

This is housekeeping. It is worth doing and it is not a finding.

Instrument one: a false negative that looked exactly like a failed fix

After deploying, I re-ran the audit against the live site. It reported that three of the pages I had just fixed still had zero followable links.

They did not. I fetched one by hand and the anchor was plainly in the served HTML.

The bug was mine. The script counted links by matching href values that started with / or http. The anchors I had added were relative — href="./" — so they matched nothing and counted as zero.

What makes this worth writing down is the shape of the failure. A measurement that silently under-reports is indistinguishable from the defect it is supposed to detect. Every downstream step would have been reasonable and wrong: conclude the deploy failed, hunt for a caching problem that did not exist, possibly "fix" three pages that were already correct and introduce a real defect doing it. Nothing in the output looked like an error. It looked like bad news.

The only thing that caught it was retrieving the page and reading the markup.

Instrument two: a report that had never returned a number

The second failure was older and much more expensive.

Bing's webmaster API serialises dates in a Microsoft-era format — /Date(1784190745000)/, epoch milliseconds wrapped in a string. My parser handled that correctly. But one endpoint, and as far as I can tell only one, returns a second variant carrying a UTC offset:

/Date(1786345200000-0700)/

My pattern captured "digits and minus signs". It swallowed the offset into the number, and the integer conversion raised an exception every time.

Because that variant appears in exactly one report, the failure was invisible everywhere else. Every other call worked. And the one report it broke was crawl statistics — the nearest available substitute for origin access logs, and the first thing to check when asking whether a crawler is showing up.

It had never once returned a value. I had been reasoning about crawler behaviour for weeks with the most direct instrument for it throwing an exception on every call. The fix is one line; the test asserts that both offset signs resolve to the same instant, because the value before the offset is already UTC.

What the repaired report showed

Date	Pages crawled that day (Bing)	Pages held in index (Bing)
2026-08-24	15	33
2026-08-25	27	33
2026-08-26	36	34
2026-08-27	23	34

Bing crawls this host every day, 15 to 36 pages, and holds 34 of roughly 38 in its index. Google holds none of them.

This is not a host a major crawler visits occasionally. It is one a major crawler works through continuously, and has done throughout the period I spent trying to work out why crawlers were not engaging with it.

Sorting the evidence

Verified directly from the live sites (2026-08-28): every row of the 38-page audit, fetched over HTTP with no JavaScript, re-verified after deployment.

Measured from Bing's API (2026-08-24 to 2026-08-28): the daily crawl volume and index count. One engine's figures about its own behaviour — good evidence about that engine, no evidence at all about any other.

Corrected tooling: two bugs, both mine, both fixed on 2026-08-28, one now covered by a regression test. The numbers here are post-correction. The pre-correction ones were wrong in a way that read as a genuine finding, which is the entire point.

Observation: two major engines, given the same host, the same bytes, the same permissive robots.txt and the same sitemap, reached opposite conclusions. One indexes nearly everything. One indexes nothing.

Unresolved: why. I do not know, and I am not going to name a mechanism to round the story off. Index selection is independent between engines, so an asymmetry is permitted and is not by itself evidence of a fault anywhere.

What the asymmetry does do is retire a family of explanations. A page a large crawler retrieves daily and chooses to keep is demonstrably reachable, parseable, renderable and acceptable to a serious indexing pipeline.

To be explicit, because the genre invites the opposite reading: nothing here fixed the zero. It was zero before the audit and zero after. This is not a recovery story.

The prediction, recorded before the outcome

The obvious next question is whether the fixes helped. I cannot answer that today — the changes went live on 28 August and there is no after to compare against.

So instead of waiting and then claiming a prediction I never wrote down, here it is in advance.

Baseline, 2026-08-28. Google: 0 pages indexed on the affected host. Bing: 34 held, 15–36 crawled per day. Audit defects all at zero.

I will re-measure on 2026-09-15, with the same script and the same API calls, and publish the result whichever way it goes.

What I expect: no material change in either engine. The defects I fixed were real, but none plausibly explains a host-wide zero, and Bing was already crawling this host thoroughly while every one of them was still present — fairly direct evidence they were not what stood in the way. Earlier in this investigation I also ran the standard remedies against a held-out control group and measured no effect: treatment 32%, control 33%.

If the numbers move, my model is wrong and that is the more interesting outcome. If they do not, I have a dated negative result instead of a vague impression.

What I would take from this

Check instruments against reality before trusting their output. Two of mine were wrong on the same day, in opposite directions. Both would have survived indefinitely if I had only read what they printed.
A silent nothing is worse than a loud failure. Make a missing measurement say missing — never zero, and never an empty report.
Separate crawled from indexed in your own vocabulary first. The fixes for "never fetched" and "fetched and declined" have almost nothing in common.
Count links on the page, not links to it. A page with no outgoing anchors is a dead end however many things point at it, and a click handler is invisible to the thing you are trying to persuade.
Fetch it and read it. Every real finding here came from the actual bytes. Every error came from believing a summary.
Write the prediction down first. It costs one paragraph and it is the difference between a result and a story.

The honest summary of the day: I improved a site that was already technically sound, repaired two instruments that had been quietly lying to me, and confirmed an asymmetry I can describe precisely and cannot explain at all.

Originally published at ai.bedvibe.studio/broken-instruments, where I will post the 15 September re-measurement against the prediction above.

I write up measurement and instrumentation work at ai.bedvibe.studio — including the control-group indexing experiment this one follows, and a crawler-behaviour observatory that logs which AI and search crawlers actually arrive. The studio itself is at bedvibe.studio.

My Agent Answers From 0.6% of Its Corpus and Reports It Like a Full Read

Panagiotis Gkilis — Tue, 25 Aug 2026 08:18:39 +0000

Originally published at ai.bedvibe.studio.

My portfolio RAG agent holds 1,003 indexed chunks. A question retrieves six and answers.

Six of 1,003 is 0.598%. Nothing in the response, the logs or the trace says the other 997 were never looked at. It reads exactly like an answer built from reading everything.

That is fine for "what does he say about X." It is not evidence for "does he mention X anywhere" — and there is nothing in a normal RAG stack that tells those two apart.

The claim retrieval cannot support

A compliance corpus: 2,431 policy documents. Someone asks whether there is a remote-work reimbursement policy. Retrieval returns eight chunks. None mentions reimbursement. The agent answers:

There is no remote-work reimbursement policy in the corpus.

That claim requires knowledge of 2,431 documents. It was made from eight. The agent did not lie and it did not hallucinate — it correctly reported what it found, and the shape of the sentence quietly upgraded a statement about eight documents into a statement about the corpus.

Absence of evidence in a retrieved fragment is not evidence of absence in the whole. Everyone knows this. No retrieval stack I have used records enough to enforce it, because the denominator is not carried anywhere near the answer.

"But our faithfulness score is 1.0"

This is the first objection and it deserves a straight answer: faithfulness cannot catch this, by construction.

Faithfulness asks whether the answer is supported by the retrieved context — did the model invent anything beyond what it was given. The compliance answer above invents nothing. It scores a perfect faithfulness and may be false about 2,423 documents.

Faithfulness scores answer against context. Coverage scores context against corpus. Different axes, and a system can be perfect on the first while completely silent on the second.

Metric	What it needs	Available at answer time?
Faithfulness	answer + context	Yes
Context precision	answer + context	Yes
Context recall	ground-truth annotations	No — offline evaluation only
Corpus coverage	corpus size + retrieved count	Yes — and it is not reported

Context recall is the metric that would catch it, and it needs labelled ground truth, so it lives in your evaluation harness and not in production. Corpus coverage needs two integers you already have.

The control is the shape of the sentence, not a percentage

This took longest to see, and it is why there is no threshold anywhere in the implementation.

0.598% is not a bad number. It is a bad number for one class of sentence. It is entirely adequate for "he mentions Rust" — you need the one chunk you are quoting and nothing else. It cannot support "he never mentions Rust." Same retrieval, same six chunks, opposite verdicts, because the claim changed.

A single coverage threshold cannot serve both. Set it low and it licenses the second sentence; set it high and it forbids the first.

Claim	Coverage required
"The policy says X"	Any — you need only the units you cite
"There is no policy about X"	Exhaustive
"All policies require X"	Exhaustive
"The most recent policy is…"	Exhaustive — the unread remainder may hold the true maximum
"There are three mentions"	Exhaustive — a count over a sample is an estimate

Every row needing exhaustive coverage has the same reason: it asserts something about the units that were not read. 999,999 of 1,000,000 is 99.9999% and still cannot establish absence — a threshold there would be a lie with a decimal point on it.

Three failures wearing one label

The second thing the missing denominator costs you is diagnosis. When a RAG answer is wrong, the post-mortem usually terminates at the model hallucinated. That sentence hides at least three separate engineering problems:

What actually happened	What to fix
The relevant chunk was never retrieved	The retriever, embedding or query — not the model
It was retrieved, then dropped during context assembly	Context construction — the model never saw it
It reached the model, which reasoned past it	The prompt or the model — the only reasoning failure

Three different people, three different days, one identical symptom. And the distinction is trivially recordable — retrieved, in_context, and whether the answer was right — but almost nobody logs the middle number, so the first two are permanently indistinguishable after the fact.

What I built

An MCP server for notchecked, the coverage-accounting schema I wrote after hitting the silent-skip failure in four domains. The library types the gaps a program leaves. The server does it for an agent, which is where the failure moved.

coverage_retrieval(
  target      = "portfolio corpus",
  query       = "Rust experience",
  corpus_size = 1003,
  retrieved   = 6,
  in_context  = 6,
  claim_type  = "absence",
)

→ scope:      6 of 1003 corpus units (0.598%)
  exhaustive: false
  claim_supported: false

  REFUSE_THIS_CLAIM: An ABSENCE claim asserts something about every unit
  you did NOT read, and 997 were never read. Downgrade the answer to what
  you found, or inspect the whole corpus.

retrieved = 0 is not a thin answer, it is a retrieval failure, and it records as one. Everything retrieved and then dropped is a context failure and records as that instead. Neither is a gap in the agent's reasoning, and calling them one sends someone to debug a prompt for a day.

The bug this found in its own implementation

I wrote a suite that replays six investigations of my own that produced wrong claims — a page judged from 3,000 of its 10,828 words, a search that had stripped the HTML so anything named only in an href was invisible, three different counts from three broken filesystem walks.

On its first run it failed, and it failed on my code rather than on the cases. exhaustive was a bare assertion: an agent could pass exhaustive: true alongside scope: "3,000 of 10,828 words" and the absence warning was dropped. The tool committed the exact failure it exists to prevent, one layer above the schema it protects. Where the scope carries "N of M", the contradiction is machine-visible and is now refused.

That is the third time this idea has caught its own implementation. I have stopped finding it funny.

What this does not do

If your retriever reports corpus_size=100000, retrieved=20, in_context=12, cited=3, this records those numbers and what they can support. It does not know whether the retriever chose the right twenty. It is not a retriever, a vector database, a reranker or a context assembler, and it is not competing with the one you have.

Three limits are recorded as passing tests rather than left out of the README: an exhaustive search of the wrong instrument is still exhaustive; the target list is self-declared, so nothing can know what the caller failed to think of; the retrieval counts are self-reported, with only internal consistency enforced.

What it removes is the silence. That is a smaller claim than "this makes agents honest," and it is the one the evidence supports.

I have measured this in one live system, my own. I have not established what other RAG deployments report, and I am not claiming it from a sample of one.

pip install notchecked · github.com/Mormolykos/notchecked — MIT, zero runtime dependencies, MCP over stdio written from the JSON-RPC wire format.

The eight coverage states were reviewed publicly by Boris Teplitsky, an IBM Certified Expert IT Architect who hit the same shape in infrastructure compliance and gave three corrections that changed the schema. They are frozen.

If you run RAG in production and you do record retrieval coverage alongside answers, I would like to know — that would make this a solved problem I had not found the solution to, which is a better outcome than being right.

I Shipped a Feature in May. Nobody Could Reach It Until Yesterday.

Panagiotis Gkilis — Tue, 25 Aug 2026 03:33:52 +0000

I shipped a feature in May. Cross-page session memory for a retrieval agent — follow a link, the conversation comes with you. It was deployed, tested, and running in production.

Yesterday I found out no visitor had ever been able to reach it.

The part that was actually broken

The obvious problem was that the chat widget existed on exactly one page out of thirty-nine. That is bad, but it is the kind of bad you notice.

The real problem was underneath. The widget never sent a session_id field. Or a scope field. The server had both, documented, tested, deployed — and the browser had simply never been wired to send either one.

So the feature was not "confined to one page." It was never invoked from anywhere. A sessions_active counter that had read 0 since May, on an endpoint that was working perfectly.

Both halves passed their own tests. The server tests posted a session id, because the test author knew the field existed. The widget tests asserted the chat rendered and sent a message. Nothing tested the seam, and the seam was the feature.

What that class of bug looks like

The failure has a shape worth naming: two components, each correct, and an integration nobody owns.

It does not throw. It does not log. The health endpoint is green. In my case the endpoint even reported the exact number that would have given it away — sessions_active: 0 — sitting in a JSON response I read regularly and never questioned, because zero active sessions on a low-traffic site is completely plausible.

That is the tell, and it generalises past my code: a metric whose broken value is indistinguishable from its healthy value tells you nothing. Zero sessions means "nobody is using it" and "it is impossible to use it," and the number is identical.

Proving the fix, rather than asserting it

Once the browser sent a session id, I did not want a green test. I wanted the thing a visitor does.

So: send one question with a session id from the datasets page. Then send a second question from the portfolio page with the client history deliberately empty — which is what a real navigation does to a JavaScript variable — and see whether the answer still knows what we were talking about.

page 1  /datasets/    "What speech datasets are there?"
page 2  /portfolio/   "Who recorded it?"     history: []

The second answer came back about the speech datasets. The server had carried it. That is a test I trust, because the thing I removed — the client's memory — is exactly the thing the feature claims to replace.

The bug I nearly shipped while fixing it

Rolling the widget out, I decided each page should tell the agent which section of the corpus it was on, so the datasets page ranks dataset material first.

Then I tested the datasets page with a piece of internal vocabulary — a codename from our own research notes — and it answered "the documents do not contain information about that." I read that as a coverage failure and widened the page's search to the whole corpus.

That was wrong, and the correction came from the person who owns the product rather than from me. Those codenames are internal research nomenclature for how a training corpus was built. They are not products. Nothing under those names is for sale. They appear zero times on the sales page and only in an engineering report.

The narrow scope was not a limitation. It was the mechanism keeping internal research vocabulary out of a commercial answer. By widening it I had connected a buyer's licensing question to the research corpus, in the name of fixing a defect that was the system working correctly.

Reverted. The lesson is cheap to state and I clearly needed to learn it expensively: do not test a sales surface with internal vocabulary, and do not read a narrow result as a broken one until you know what the narrowness is for.

What I would take from this

Ship a feature and then, before believing it works, ask what a visitor would have to do to reach it — out loud, as a sequence of clicks. Mine had no answer. There was no page to navigate from.

And look at your health metrics for one specific property: is there any value this number could report that means "broken" but reads as "quiet"? Those are the numbers that let a feature sit dead in production for three months while everything stays green.

The agent this is about is live: the architecture and its 32-case evaluation gate are documented at tts.bedvibe.studio/portfolio/agent-architecture, the eval results at tts.bedvibe.studio/portfolio/agent-evals, and the widget itself is on every page of tts.bedvibe.studio now — including, finally, more than one.

If you have found a feature of your own that was deployed and unreachable, I would like to hear how you noticed. In my case it was luck, and I would prefer a method.

My Validation Layer Was Correctly Deleting 16% of My Good Data

Panagiotis Gkilis — Tue, 25 Aug 2026 03:12:31 +0000

Originally published at ai.bedvibe.studio.

I built a real-time tracker in Rust — about two thousand lines — that reads a live ADS-B feed, keeps a Kalman-filtered track per aircraft, and screens every pair for closest approach against separation minima. Roughly 150 aircraft, a full cycle in under a millisecond.

It ran clean. Tests passed, the picture looked right, the numbers were plausible.

It was refusing about one measurement in nine, and the only reason I ever found out is that the rejections went to a counter instead of a log line.

The gate has a sub-second tolerance for clock error

The tracker runs an innovation gate: when a position arrives, the filter predicts where the aircraft should be, and if the measurement is too far from that prediction it is rejected as physically impossible rather than believed.

Once a track converges the innovation standard deviation settles around 36 m, so a five-sigma gate sits at roughly 180 m. An airliner at 250 m/s covers 180 m in 0.7 seconds.

So the gate's entire tolerance for a wrong timestamp is under one second. Any pipeline that mis-times its measurements by more than that will have them rejected — correctly, and invisibly.

The feed reports its own staleness. The pipeline dropped it.

Every ADS-B record carries a field saying how old that position already was when the response was generated. In the original build it was parsed into the contact struct and never read again — the only other place that field appeared in the entire codebase was as 0.0 in test fixtures. Every measurement was therefore stamped with the tracker's own cycle clock, as though it had been observed at the instant it landed.

This is the common case, not an exotic one. A field that is decoded and then unused looks identical to a field that is decoded and used, right up until you go looking for its second reference.

Here is what that field actually contains, sampled across two consecutive polls of the live feed:

reported age of position    median   0.31 s
                            p90      3.97 s
                            max     48.53 s

change per aircraft
between consecutive polls   -15.76 s  to  +3.00 s

re-served identical
positions                   17 of 135 contacts  (12.6%)

It is the jitter, not the lag

The intuitive diagnosis is that the lag itself is the problem. It is not, and the distinction turns out to be the whole thing.

A constant lag is invisible to a constant-velocity filter. If every measurement is uniformly two seconds old, the filter simply tracks a target that is uniformly two seconds behind. The innovations stay small. Nothing is rejected. The picture is late, but it is self-consistent.

That has a sharp consequence for anyone writing a regression test: a fixture built with a constant age will not reproduce the fault no matter how large the age. The test passes and the bug survives. The fixture has to carry the variation, or it is testing nothing.

Look at that middle row again. Between two polls two seconds apart, one aircraft's reported age fell by nearly sixteen seconds — a genuinely new observation arriving after a long gap. Stamped on arrival, both the stale one and the fresh one are marked "now", so the filter sees an aircraft that has apparently teleported. A steadily flying aeroplane appears to lurch, and a correctly functioning gate refuses to believe it.

Why the obvious fix makes it worse

This is the part worth reading, and the reason the change is architectural rather than a one-line patch.

The obvious repair: stamp each measurement with arrival − age, feed that to the filter, and guard against a measurement arriving from before the filter's current state, because you cannot predict backwards.

if observed_time <= track.epoch { reject }

That guard is fatal. Every cycle, the display loop advanced every track to the current wall-clock time so the picture and the collision screen would agree. So by the time a two-second-old measurement arrived, the track's clock already read later than the measurement, and the guard would have discarded it as stale. A patch aimed at an 11% rejection rate would have rejected considerably more — and from the outside it would have looked exactly like the fix working.

The general form: a Kalman filter's state is valid for exactly one instant. Once anything other than a measurement is allowed to advance that instant, every measurement is applied to a state from a different moment than the one it describes. A rendering loop is not usually thought of as mutating the estimator, which is precisely why this survives review.

The pre-fix code was not wrong about this. It was consistently wrong — the display and the measurements were stamped with the same fictional clock, so they agreed with each other. That is why correcting one half in isolation breaks it.

The rule that actually fixes it

One sentence: the filter's validity time advances only when a measurement arrives. Nothing else may move it.

The picture and the conjunction screen do not predict tracks forward — they take a view, which extrapolates a copy and leaves the filter parked at the moment it was last given evidence for. A test asserts that rendering the screen thirty times does not move a single filter.

Two properties follow, and both are load-bearing.

The absolute observation time is never reconstructed at all. The step between two observations is computed entirely from differences — elapsed time between two arrivals the process witnessed, plus the two ages the sensor reported. Nothing subtracts a duration from a monotonic clock, so there is no underflow path. That closes a real crash: Instant - Duration panics in Rust if the result would precede the clock's origin, and on Windows that origin is boot time.

Deduplication becomes free. A re-served snapshot has both its arrival and its reported age advance by the same amount, so it computes a step of exactly zero and is rejected by the same comparison that catches out-of-order data. One comparison covers both cases, which is a good sign the invariant is the right one.

Designing an A/B the upstream cannot distort

Measuring this against a live third-party feed is harder than it looks, and two designs had to be discarded before one held.

Running both binaries simultaneously fails: the corrected build recorded 32 polls and 32 HTTP errors — zero data — while the old one ran fine. That is not a regression, it is the upstream rate-limiting per IP. Running them sequentially fails differently: the results invert, because the API throttles progressively and whichever build runs first gets the fresher quota. Time-to-first-byte on three consecutive manual requests climbed 0.43 s, 0.91 s, 4.22 s.

Both would have produced a clean-looking number and a wrong conclusion. What caught them was running the corrected build alone, which worked — proving the failure was the environment, not the code under test.

The design that survives: alternating order, 45-second windows, 75-second cooldowns, comparing gated plots as a fraction of observations so traffic volume cancels.

window   build       observations   gated   rate
  1      corrected       2,137          3    0.14%
  2      pre-fix         2,752        308   11.19%
  3      corrected       2,208         81    3.67%
  4      pre-fix         2,755        578   20.98%

pooled   pre-fix       886 / 5,507         16.1%
         corrected      84 / 4,345          1.9%

Both corrected windows sit below both pre-fix windows with no overlap. The corrected build varies — 0.14% against 3.67% — and I would not claim a precise point estimate from four windows. The separation is the defensible result, not the number.

Two unrelated faults the counters surfaced

It was silently failing most of its polls. 888 observations at roughly 135 per response is about seven successful polls in sixty seconds, not thirty. Two-thirds of its requests were failing and the system reported nothing, because there were no sensor-health counters at all — a failed poll produces no data, which looks exactly like quiet airspace.

It never aged out a single track. Zero drops in both windows. Of course not: if every measurement is stamped with arrival time, every track's last-seen time is always now, so nothing is ever stale. A track whose aircraft stopped reporting forty seconds ago sat on the display looking as current as everything else.

Those counters were added to report on something else entirely — a limit on response body size. They immediately found a different fault. That is the lesson I would take from this project ahead of the timestamp one: observability added for one problem finds problems you were not looking for, and a system with no failure counters cannot tell you the difference between "nothing is happening" and "nothing is working".

What to check in your own pipeline

If you ingest anything from a source that reports its own staleness — market data, IoT sensors, GPS, log shipping, any polled API with a timestamp in the payload — check whether your pipeline records when the record arrived instead of when it was observed.

The failure mode is nasty specifically because it is quiet. Nothing crashes. No error is logged. Your validation layer does its job perfectly and deletes your good data, and if the rejections are counted anywhere at all you will read the number as evidence that the validation is working.

And a constant lag hides it completely. It is the jitter that bites — which means the systems most likely to have this bug are the ones whose feeds are usually fast.

MIT, 55 tests, clippy clean at -D warnings: github.com/Mormolykos/aether. Surveillance and state estimation only — no targeting, engagement or weapon functionality of any kind. The README carries the full measured results and a limitations section considerably longer than this post.

More measured write-ups of the same kind, including the retrieval work and the long-document RAG ablation, are at tts.bedvibe.studio/portfolio.

Your TTS Model Sounds Great — Until It Says "GPUB"

Panagiotis Gkilis — Tue, 25 Aug 2026 03:10:05 +0000

Originally published at ai.bedvibe.studio.

I built a text-to-speech product and kept getting burned by the same thing. On normal sentences the model sounded great. Then it would hit a number, a date, an acronym or a name, and quietly mangle it.

Worse, the metric everyone reaches for — Word Error Rate — was lying to me in both directions. It flagged perfectly good audio as broken because the script said 3:30 PM and the transcript said "three thirty pee em." And it missed real failures on short tokens, where the speech recogniser is as unreliable as the TTS.

So I wrote the QA framework I wished I had, packaged it as ttsproof, and then ran it as a blind study against a production TTS service so the results would be more than an opinion.

The two failures WER cannot see

A TTS pipeline breaks in two different ways, and a single WER number blurs both.

Structural defects. The clip is empty, truncated, three times too long, stuck in a repeated-chunk loop, clipping, or has a click at the tail. These have nothing to do with pronunciation — you can catch them with no model at all, straight from the waveform.

Pronunciation and content errors on the hard cases: numbers, decimals, dates, clock times, acronyms, single letters, URLs, names.

ttsproof splits them apart and handles each one honestly:

Structural checks, no model needed — empty or truncated audio, duration explosions, long internal silences, clipping, loop detection, end-of-clip artifacts. numpy and soundfile, nothing else.
Equivalence-aware WER/CER — the expected text and the ASR transcript are both canonicalised to spoken form before scoring, so 3:30 PM against "three thirty" stops counting as an error.
ASR-uncertainty quarantine — when the audio is structurally clean but the recogniser disagrees on a very short utterance, the sample is set aside for a human instead of being auto-failed. At that length the ASR is as likely to be wrong as the TTS.

The study: 390 samples, and a blind human check

I evaluated the method against a production neural TTS service — 130 edge cases × 3 voices = 390 samples — and published it as a citable technical report (DOI 10.5281/zenodo.20757553, CC-BY-4.0).

Zero structural audio-integrity defects across all 390 clips. That matters: it means every failure that did exist was pronunciation, exactly the kind WER mislabels.
Exact-match rate 0.769.
Then the honest part — a blind human review of the ASR-uncertain quarantine zone. Of 42 uncertain clips, 23 (55%) were ASR false negatives (the TTS said it right and the recogniser misheard) and 19 (45%) were genuine TTS mispronunciations. Fifteen control clips came back 15/15 correct, so the rater was reliable.

That 45/55 split is the entire argument for having a quarantine verdict at all. Auto-passing that zone ships 19 real mispronunciations. Naive ASR-WER auto-failing it wrongly kills 23 correct clips. Neither is acceptable, so ttsproof refuses to guess there.

What the real failures looked like

All 19 genuine failures were short isolated letters and acronyms, and the pattern is oddly specific:

Failure mode	Examples
A-vowel substitution	`NATO` → "NITO", `USA` → "USI", `CIA` → "CII"
Trailing appended phoneme	`GPU` → "GPUB", `EU` → "EUU"
Early truncation	`R` chopped short
Doubling	`X` said twice
Other substitution	`CEO` → "CEE", `Z` → "SZ"

The structural detectors did not fire on any of these. "GPUB" is intelligible speech, not a click. Structural checks and ASR quarantine are complementary; neither alone catches everything.

Benchmark any engine in one command

Beyond the study, ttsproof ships a corpus of 817 curated edge cases across 39 categories — numbers, currencies, dates, ISO timestamps, phone numbers, URLs, file paths, pronunciation-torture words (Worcestershire, synecdoche), proper names (Reykjavík, Nguyễn), Greek, Norwegian and more. The corpus is versioned independently of the software, so published scores stay comparable across tool updates.

It is engine-agnostic — point it at any TTS via a command template, or at a folder of audio you already generated:

ttsproof benchmark --cmd "mytts --text {text} --wav {out}"

You get a category scoreboard, a self-contained report.html with waveforms, an audio player and what the ASR actually heard, and a CI regression gate. Closed-source engines work too through a SpeechSDK wrapper — an integration a user suggested after the first release.

Try it

pip install ttsproof            # structural checks + metrics + corpus
pip install "ttsproof[asr]"     # + faster-whisper for pronunciation gating

Repo: github.com/Mormolykos/ttsproof (MIT)
The study: doi.org/10.5281/zenodo.20757553

It has already had its first outside contribution, a community fix for a real number-formatting bug, which is exactly what I hoped for. If your TTS breaks on something, open an issue with the case — the corpus grows from real failures.

The text-to-speech platform this came out of is live at tts.bedvibe.studio — ttsproof exists because I needed to QA that, and the model's engineering write-up covers what it is built on.

Why Corrupted Training Data Doesn't Show Up as High Loss

Panagiotis Gkilis — Tue, 25 Aug 2026 03:06:06 +0000

Originally published at ai.bedvibe.studio.

There is an assumption almost every practitioner carries without examining it: if your dataset has bad samples in it, the loss will tell you. Corrupted rows spike. Broken files stick out. Sort by per-sample loss, look at the top of the list, and there is your garbage.

I believed it too. Two separate failures in my own work say it is wrong, and they fail in the same direction — quietly.

The reproducible one: a dataset that cannot be learned

While validating trainproof I ran a controlled fault-injection study: one base setup, a Qwen2.5-3B QLoRA, run six ways, three seeds each, eighteen runs total. Every log ships in the repo so the verdicts can be checked rather than believed.

One configuration shuffled the dataset's labels into pure noise. The labels no longer corresponded to the inputs at all. This is not a hard dataset or a noisy dataset. It is a dataset that cannot be learned, because there is no relationship left in it to learn.

That run reduced its loss by 62%.

On its own curve it was textbook-healthy — a clean downward slope, no spike, no plateau, nothing a human or a rule would flag. It was learning nothing useful. It was memorising the statistics of noise, which any sufficiently large network will happily do. From a single run's loss curve it is indistinguishable from a real one.

That is where the assumption broke for me. Not "loss is a weak signal for this." Loss is not a signal for this at all, in isolation.

The production one, and what I can and cannot prove about it

The second failure came from real work rather than an experiment, and it is the one I think about more.

Building a text-to-speech corpus of roughly 110,000 recordings, a small number of the files were pure loud white noise. Not corrupted in the file-format sense — they opened fine, played fine, had valid headers and valid duration. They simply contained no speech. Just noise, at volume.

They did not surface as high-loss outliers.

Being precise about the epistemic status of this one matters: those training logs no longer exist. I cannot show the numbers and I am not going to reconstruct them from memory. It is a production observation, not a measurement. Treat it as the anecdote that sent me looking, and treat the fault-injection study above as the part carrying evidence.

But the two line up, and that is the reason to write it down.

Why it happens: noise has learnable statistics

The mechanism is the same in both cases, and the surprise goes away once you see it.

We treat "noise" as a synonym for "unpredictable" and then quietly assume a model will fail loudly on it. White noise is not unpredictable in the way that matters to a loss function. It is stationary and uniform. Its distribution is simple and consistent. A network fits that distribution quickly and cheaply — and fitting it registers as loss going down.

Shuffled labels are the same story from the other side. The mapping is destroyed, so nothing generalisable remains, but the network can still memorise the marginal statistics of the targets. Loss falls. Nothing spikes.

In both cases the model learned something. It just was not the thing you wanted. The loss function has no opinion about which of the two it did.

What to do instead

Two things follow and neither is exotic.

Compare against a baseline; do not judge in isolation. The shuffled-labels run becomes obvious the moment you put it beside a known-good run and look at the relative floor each one reaches — the corrupted run's floor sits in a different regime entirely. One run alone lies; two runs side by side tell the truth. That is why trainproof has a compare mode at all: not as a convenience, but because single-run rules provably cannot catch this class of failure.

Check the data as data, before training. This is the part I got wrong for longer than I would like. I was inferring dataset quality from the training curve, which is an indirect measurement of an indirect measurement. For audio the checks are cheap and direct: spectral flatness, dynamic range, silence ratio, speech-band energy. A pure white-noise file is trivially separable from speech by any of them — if you look at the audio. It is not separable by looking at the loss.

The general shape

This generalises past my two cases, and it is worth stating plainly:

A loss curve measures whether the model is fitting something. It does not measure whether that something is your task.

Any corruption that replaces your signal with a distribution the model can fit will pass every loss-shaped check you own. Truncated samples, silence, duplicated rows, label noise, wrong-language text, empty targets — none of these are guaranteed to spike. Some of them are guaranteed not to.

The failures that burn the most GPU hours are not the ones that crash. They are the ones that look exactly like success.

The fault-injection logs are all in the trainproof repo, including the shuffled-labels run and a 9.8-hour Coqui XTTS fine-tune that diverged on its own with nobody touching it.

If you have hit a failure mode a deterministic check would have caught, tell me on the repo — it goes in, with credit.

The 110,000-recording speech corpus in the second case belongs to a 730M-parameter TTS model I trained from scratch; how it was built is documented at tts.bedvibe.studio/portfolio/bedvibe-tts.

The Same Bug, Four Times, Three of Them Mine

Panagiotis Gkilis — Tue, 25 Aug 2026 02:20:15 +0000

Originally published at ai.bedvibe.studio.

Most validation tooling has two states: it passed, or it failed. Everything that was not actually evaluated has to be forced into one of them — and it is wrong in both directions.

I did not work that out from first principles. I worked it out by shipping the same defect four times.

One: a run that learned nothing, reported as healthy

In trainproof, a training run whose loss was exactly 0.0 on every step returned PASS.

Every loss-shape check in that tool is guarded against dividing by zero. A curve that is identically zero trips every guard, so all of them skipped — silently. And then the report listed those same skipped checks as having run. A run that learned nothing passed, accompanied by a list of the checks that had cleared it.

That is the whole problem in one artifact. The tool had no way to say I could not judge this, so the absence of a judgement rendered as a favourable one.

Two: the same tool, one loop earlier

The fix for that was a report field listing which checks ran and which did not. Good. Then a nastier version turned up somewhere else.

trainproof doctor walks a directory twice — once to discover candidate logs, once to judge them. The judging pass reported anything it could not read. The discovery pass had except Exception: pass.

So a file that raised while being found never became a candidate, and never appeared in the report at all. Plainly visible on disk. Absent from the output. Indistinguishable from a file that passed.

Same failure as the first, one loop earlier than I had been looking. That is when I stopped treating it as a bug.

Three: not my system

I wrote the first two up publicly. Someone working in infrastructure compliance replied with the identical shape from a domain I know nothing about.

A compliance framework document is mostly prose. Most of it describes things no generated artifact can satisfy or violate — staff training, review procedures, who signs what. Only a fraction maps to something a machine can check.

The trap is reporting against the framework's name. Do that and everything unevaluated looks identical to everything that passed, and the ninety per cent that was never in scope disappears from the output entirely.

That exchange is why this became a library rather than another trainproof feature. One person hitting a shape twice is a habit. Two people hitting it in unrelated domains is a primitive.

Four: found while I was busy being pleased with myself

Then I ran an evaluation harness of my own over a retrieval experiment and found three instances in a single afternoon.

It recorded model refusals under a failure type asserting an ordering the model had never given. It scored ten refusals as correct, because the expected phrase appeared inside the sentence explaining what could not be determined. And it missed eight correct answers because its negative pattern required a comma.

One absent value. Wrong in both directions. Inside the instrument I was using to judge my own hypothesis.

The states

Three instead of two — checked, could-not-check, never-in-scope — and eight once you ask two more questions of each gap: who can fix it, and can it ever change.

State	Owner	What it means
`CHECKED`	—	a determination was made; the verdict is separate
`NOT_CHECKED / DATA_DEGENERATE`	the data	the signal is present and unusable
`NOT_CHECKED / CHECKER_FAILED`	your tooling	the checker raised, timed out, or could not observe
`NOT_CHECKED / WAIVED`	a named person	in scope, deliberately not evaluated, accepted
`NOT_CHECKED / PREREQUISITE_FAILED`	another target	something upstream failed first
`OUT_OF_SCOPE / CALLER`	the caller	not requested
`OUT_OF_SCOPE / DATA_TRANSIENT`	the deployment	does not apply yet
`OUT_OF_SCOPE / DATA_PERMANENT`	nobody	no artifact of this kind can ever evidence it

Coverage is not verdict. CHECKED is not a result — it says a determination was made, not what it was. The verdict vocabulary belongs to your domain (pass/warn/fail, compliant/non-compliant) and hangs off CHECKED rather than sitting beside the not-checked states. Collapsing those two axes is failure one above. The constructor enforces the separation: a gap cannot carry a verdict, and a checked record cannot carry a skip reason.

What the review changed

Boris Teplitsky, from the compliance side, put three corrections into the schema that I would not have found:

Permanence is relative to a target, never absolute. "Nobody, never" is not a property of a control — it is a property of pairing that control with a kind of artifact. A Kubernetes control is permanently out of scope only while the target has no Kubernetes; change the target and it becomes a row. So OUT_OF_SCOPE / DATA_PERMANENT now requires a permanent_wrt field naming the reference. Unqualified, two reports on the same framework disagree and both are correct.

Rows for the checkable subset, one count for the rest. A framework document is hundreds of pages of which a few paragraphs concern anything an artifact can evidence. One row each makes the report noise. So the excluded corpus can be a single counted rule — and the count cannot be stated without the rule that produced it, and it enters the denominator, so excluding 412 of 415 reports 99% rather than a flattering silence.

The taxonomy starts after the unit exists. In a linter, a row is a check somebody wrote. In compliance, turning prose into checkable requirements is the hard part and it is a judgment call. Nothing in the library governs that, and it should not read as if the rows arrive by themselves.

That last one is scope, not code, and it was the correction I was most wrong about.

And one the change found in my own reporter

Running the README example through the new bulk path printed 413 out of scope (100% of all targets) while one target had in fact been checked — .0% rounded 99.5 up. A reader takes "100% out of scope" as "nothing was measured."

That is this library's own failure mode, committed by its own renderer, one layer above the schema it protects. Fixed both directions: 100% only when everything is excluded, 0% only when nothing is.

MIT, no dependencies, 74 tests: github.com/Mormolykos/notchecked

If you have hit this shape in a fifth domain, I would like to hear it — the case list is a test file now, so a new case goes in with credit.

A Hundred Per Cent Success Rate and One Usable Answer

Panagiotis Gkilis — Tue, 25 Aug 2026 01:56:48 +0000

Originally published at ai.bedvibe.studio.

I built a gateway that sits in front of several inference backends — authentication, per-key spend quota, token metering, retries, failover, streaming. The ordinary shape. Then, before trusting any of it, I broke it twice on purpose and measured what the broken version reported about itself.

Both breakages produced clean logs. That is the part worth writing down.

The first fault: a gateway that cannot say no

I removed admission control — no bound on how many requests may be in flight at once — and sent thirty-two callers at the gateway simultaneously. Each caller was willing to wait 700 milliseconds and would walk away after that.

Every single request returned HTTP 200. Thirty-two successes out of thirty-two. A dashboard reading that gateway would show a flawless service.

Thirty-one of those answers arrived after the caller had stopped waiting. One was usable.

The queue had not disappeared when I removed the bound. It had moved. Off the gateway, where I could see it and refuse it, and into the backend, where it was invisible and where everybody's clock was already running. The gateway kept faithfully reporting the outcome of work that no longer had an audience.

With a bound and a fast 429 instead: eight answered, twenty-four refused within milliseconds, and six usable answers instead of one.

	Unbounded	Bounded
HTTP 200 returned	32	8
Success rate	100%	25%
Answers that arrived in time	1	6

Six times the useful output, while the headline metric fell by seventy-five points. If you are optimising the success rate, you will optimise your way back into the first column.

Where the bound came from

I did not pick eight because it felt right. I measured the backend until it stopped getting faster:

Concurrent requests	Throughput
1	189 tokens/sec
8	433 tokens/sec
32	474 tokens/sec — 9% more, for 3.3x the latency

The knee is at eight. Past it you are not buying throughput, you are buying queue. So the admission limit is eight, and it is a measurement rather than a preference — which also means it has an expiry date: change the model or the hardware and the number has to be taken again.

The second fault: failover that works in the log and fails for the caller

Everybody writes failover for a backend that dies. Dying is easy — the connection refuses, you try the next one. So I made a backend go slow instead: three seconds to respond, against a caller willing to wait one and a half.

The gateway logged eight successes. Every one of them landed at three seconds, after the caller was gone. The healthy backend was never called once.

The cause was that my retry budget was a count of attempts plus a per-backend timeout. Neither of those knows whether anybody is still waiting. A retry policy expressed in attempts is a policy about the server's patience, and the server is not the one who is waiting.

So the deadline now belongs to the request, and travels with it. While another backend is still available, the current one may spend at most sixty per cent of whatever time remains. You have to reserve time for the thing you are failing over to, or the failover is decoration that runs after the deadline it was supposed to protect.

	Attempt-counted	Deadline-owned
Logged successes	8	8
Arrived in time	0	8
p50 latency	3001 ms	1046 ms

Note the first row. The log is identical in both columns. Every observability surface I had said the failover was working, for as long as I only asked it whether requests succeeded.

Three things I did not plan

The two faults above were deliberate. These three found me.

The first working call billed 32 tokens and returned an empty string. The local model is a reasoning model, and the server returns its reasoning in a different field from its answer. Read only the answer field and you show the user nothing while charging them for a full generation. Nothing errors, and the token accounting is correct — that is what makes it nasty.

A concurrency bug that hid inside a green test suite. I created the lock that bounds concurrency in the constructor. Python binds those to whichever event loop touches them first, so a second loop makes it throw — on admission, before any model is called, while the health endpoint still says ok. It surfaced as a warning on a worker thread and the suite reported 79 passed. Then my fix had the same bug in a new costume: I keyed it by the loop's memory address, and Python reuses the address of a closed loop. Both are fixed, and the suite now turns a swallowed background exception into a failure.

I guessed the wrong hotspot and the trace corrected me. I was certain the overhead was the quota check — three database reads on every single request, right there in the hot path. It was 0.146 ms, about four per cent. Three quarters of the overhead was writing one metering row, because the database was flushing to disk before answering. One setting: 2.999 ms down to 0.230 ms, and the gateway's total overhead from 3.47 ms to 0.60 ms.

The number I refused to publish

Measured end to end, through the whole stack, the gateway added +1.26 ms against a noise floor of ±5.38 ms.

That is not a measurement. It is a number smaller than the uncertainty around it, and quoting it would be quoting noise with a decimal point on it. The benchmark now says so out loud instead of printing the figure.

It is also the entire argument for instrumenting inside the system. You cannot find a one-millisecond effect by subtracting two noisy end-to-end numbers. You find it with a span around the thing you suspect — which is how the metering flush turned up, and how I learned my confident guess about the quota check was wrong.

The one I was asked to build and didn't

The brief said to put vLLM on this machine. I looked at the machine first.

vLLM has no Windows build, so reaching it means a WSL2 install of roughly ten gigabytes of CUDA — on a box where system RAM, not VRAM, is the bottleneck. And the GPU is not idle: it is shared with a text-to-speech service that sits behind a live API taking real traffic, holding 7.4 of its 16 GB. vLLM's default configuration reserves ninety per cent of the card. The good outcome is that it refuses to start. The bad outcome is that I take down something that earns money, to demonstrate a tool.

So I said so, in an architecture decision record, along with what the decision costs: continuous batching is what vLLM is actually for, and I did not measure it. Measuring something else and calling it vLLM would be a lie with benchmarks attached.

What I built instead is the layer above the engine, and that is the part that transfers. Routing, quota, metering, admission, deadlines and failover know nothing about which engine is underneath. The proof is structural rather than rhetorical: the two backends in it speak completely different protocols and each one is a single file.

What I actually take from this

Success rate is the metric that hides an outage. Both faults produced a clean log. One produced a perfect log. The number I care about now is how many answers arrived while somebody was still there to read them, and every dashboard I build gets that number next to the success rate, not instead of it.

The second thing is smaller and I keep relearning it: a fault that returns an error is a fault you will fix this afternoon. A fault that returns a plausible success is one you ship. Both of the faults here were of the second kind, and I only found them because I planted them and then went looking for the specific evidence that they had happened — not because anything alerted.

Ninety-four tests, seven seconds, no model, no GPU, no API key and no network. Zero API calls were spent on any of it.

Source: github.com/Mormolykos/basalt.

If you have a load-shedding or failover story where the log looked fine, I would like to hear it — particularly the ones where the metric that lied was one you had deliberately chosen.

The retrieval agent this gateway was built to sit in front of is live, with its architecture and evaluation gate written up at tts.bedvibe.studio/portfolio/agent-architecture.

Ten Tests, Written Before the Code. The One That Failed Taught Me the Most

Panagiotis Gkilis — Thu, 20 Aug 2026 07:07:32 +0000

There is a number in solid mechanics that is exactly 3.

Take an infinite plate with a circular hole, pull it in one direction, and the stress right at the edge of the hole is exactly three times the stress far away. Kirsch worked it out analytically in 1898. It is not approximately 3. It is 3.

I wrote a finite element solver from scratch and never told it that number. It came back with 3.00002.

What "from scratch" means here

No FEA library. No scipy. The Delaunay mesher, the isoparametric elements, the global assembly and the conjugate-gradient solver are all my own code.

That constraint was the point. Calling a library and getting 3 proves the library works.

Configuration                                  K_t        error
Annulus, P2 curved, finest mesh (21,220 dof)   2.99981    0.0063 %
Annulus, extrapolated to h = 0                 3.00002    0.0005 %
Finite plate, extrapolated                     2.99970    0.0101 %
Conservative Richardson variant                2.99937    0.0211 %

Hitting the number is the weak evidence

This is the part I want to argue for, because it applies well beyond numerical code.

Landing on 3 could be luck. A bug that happens to cancel, a mesh that happens to flatter you, a fudge factor someone tuned until the output looked right. One number matching one expected value is the weakest test you can run — and it is the test most of us stop at.

The strong evidence is the rate. Finite element theory predicts exactly how fast the error must shrink as the mesh gets finer, and the rate is different for each element type. You cannot fake that by accident.

Element        L2 rate  predicted   H1 rate  predicted   energy  predicted
P1 (linear)      2.026      2         1.019      1        2.000      2
P2 (quadratic)   3.249      3         2.013      2        3.988      4

Six numbers, six predictions, all recovered. That is what says the implementation is right rather than coincidentally close.

Writing the tests before the code

I registered ten acceptance gates in a literature audit before any code existed. Quadrature exactness, patch tests, mesh validity, checking the iterative solver against a direct one, scale invariance, convergence orders for both element types.

The reason is not discipline for its own sake. It is that I know myself. If I write the criteria after seeing the output, I will unconsciously write criteria the output passes. Registering them first means the standard cannot move.

Nine gates passed. One did not.

Gate I failed, and it is published

Gate I concerned a variational crime — an approximation in how the curved boundary is handled. It failed. I tested two hypotheses for why and both were refuted.

What is published is an arithmetic account consistent with every number I measured, plus a plain statement that the discrepancy cannot be settled from this experiment and is not settled. The literature figure I had written the gate against turned out to come from a search summary rather than a paper I had actually opened. That is in the record too.

I could have deleted the gate. Nobody would have known it existed, because I wrote it myself, before the code, in a document nobody had read.

The gate that passed for the wrong reason

This one I am more pleased with than the failure.

Gate H passed. Then I went back and criticised it: the acceptance band I had written was the wrong shape for a one-sided theorem. It should have been a one-sided bound, and I had written a two-sided one. The result was fine, but the test would have passed things it should have caught.

No reviewer forced that. A passing test that passes for the wrong reason is a bug in your test suite that no failure will ever reveal to you.

Making the central claim checkable instead of asserted

The whole thing rests on "the target was never an input." That is easy to say and easy to lie about.

So the README documents it as something you can verify: grep the source for the target value, and the only hits you find are ones that never feed the computation. You do not have to believe me. You can run grep in about four seconds.

The full run reproduces with python fem_003.py in roughly fifty seconds, with a --smoke mode for a quick shake-out, and it ships its own evidence — gate report, convergence CSVs, hoop-stress CSV, mesh and convergence plots.

The honest framing

This is reproduction and validation, not discovery. Kirsch published the analytic solution in 1898 and the convergence theory is in every textbook. Nothing here is new.

The contribution is building the thing and demonstrating that it lands on a number it was never given, with the rates it was supposed to have, against criteria written before the first line of code.

If you take one thing from this: write your acceptance criteria before you write the thing, publish the ones that fail, and go back and check whether the ones that passed deserved to.

Full write-up: Ten tests, written before the code

Code, data and the full gate report: 10.5281/zenodo.21892064