DEV Community: Panagiotis Gkilis

I audited my own ML linter and had to withdraw its best evidence

Panagiotis Gkilis — Mon, 21 Sep 2026 20:54:14 +0000

I maintain trainproof, a deterministic linter for ML training runs. No model scores your run — every verdict is a rule that either fires or doesn't, and every finding prints the numbers behind it.

Before releasing 0.22.0 I put it through two independent forensic audits, one of them working blind. They took away the single best piece of evidence I had.

The claim

Every fault in my validation gallery is injected on purpose — learning rate cranked 100×, fp16 overflow, shuffled labels. Injected faults are clean, and that's the known weakness of any fault-injection study. So the one log I was proudest of was the one nobody touched: a 9.8-hour Coqui XTTS fine-tune from my own work that, I said, "diverged on its own."

It was on the README. It was in two places in my long-form write-up. It was the thing that made the study stand on something real.

It was computed from a truncated prefix

The log I shipped stops at step 72,900 of a run that went to 125,039. I had truncated it, published the verdict computed from the truncation, and cited the trainer's own bookkeeping as corroboration — the last BEST MODEL line in the prefix is best_model_49880.pth, which does sit a long way behind the end of the prefix.

The rest of the run was still on disk. In the complete log:

the trainer promotes best_model_124700.pth — step 124,700 of 125,039, 99.7% of the way through
all six retained held-out evaluations improve, the last one included: 4.8813 → 2.5894

The bookkeeping I cited as corroboration was corroborating a sentence about the first 58% of a run.

What replaces it, exactly

The temptation here is to overcorrect, and that would be the same mistake pointing the other way. So, precisely:

What the evidence supports: the per-micro-batch training display loss ended above its own minimum. That is a property of that one series.

What it does not support: that the model got worse. And it does not support that the model got better either — no audio, MOS or perceptual evaluation of that run was kept. Held-out loss improved. That is all that was measured.

The part that matters more than my mistake

trainproof still returns FAIL on that run, and 0.22.0 does not fix it.

TP-DIVERGE reads one training series. That run's event file carries two:

series	points	ends
per-micro-batch loss	1,251	1.88× above its minimum
epoch-aggregated loss	5	at its minimum

The reader resolves competing series by density. The 1,251-point series wins; the 5-point series is discarded before any rule sees it. An improving held-out curve sitting in the same file cannot overrule the survivor.

I could have changed the threshold until this particular run went green. That is tuning a rule against a single example until it produces the answer I now prefer, and it would have been the worst available option. So the run still FAILs, the limitation is written into the README, the rules document and the release's machine-readable record, and it is the top item for the next architecture instead of something quietly smoothed over.

What else the audits found

Same principle decided each one: remove a claim the tool can't support, or make an advertised behaviour real — never strengthen a claim to close a gap.

A missing optional dependency was a verdict about your work. No sentencepiece installed? trainproof tokenizer exited 1 with a FAIL — the tool reporting its own missing package as a fault in your tokenizer. Now NOT-CHECKED, exit 2. FAIL has to mean something about your run, never "the tool had a problem."

The HuggingFace callback could change the run it was watching. It read the first batches of labels at on_train_begin, which means opening a fresh iterator over your training dataloader. With a map-style loader and a random sampler drawing from a generator, that consumes generator state and shifts your batch order. A measurement tool perturbing its own experiment. objective_check now defaults to False.

Four evidence strings described something other than what was measured — including a zero-learning-rate finding that printed 100.0% of steps have lr=0 for a run where every value was -1e-4.

No rule ID, threshold or detection predicate changed in 0.22.0. It adds nothing. 497 tests, each correction pinned by a test that fails against the pre-repair code.

What I'm not claiming

No rule in trainproof is calibrated. There is no false-positive rate for any check, because nothing here has been run against a population where the answer is independently known. The gallery is a set of faults I injected on purpose — a regression suite, not a sample.

What the tool offers is determinism and evidence: the same log gives the same verdict, every finding prints the numbers behind it, and a check that could not run says NOT-CHECKED instead of PASS.

Read a finding's evidence as the measurement and its message as an interpretation that may exceed it. That sentence is in the README now because the audits showed it needed to be.

pip install trainproof
trainproof doctor .   # discovers and judges every training log it finds

Release 0.22.0 is archived at 10.5281/zenodo.22882964. The full write-up, including what the audits found that I haven't covered here, is on my notebook.

If a deterministic check would have caught a failure that burned your GPU hours, tell me on the repo — it goes in, with credit.

Panagiotis (Panos) Gkilis — founder, BedVibe Studios, the operating brand of BED VIBE GKILIS, a sole proprietorship registered in Norway (org. no. 935 267 897).

Main hub · Portfolio · Work · Research & articles · About · LinkedIn · GitHub · ORCID

Genuine Human Speech, Called Synthetic. Nothing Was Generated.

Panagiotis Gkilis — Fri, 18 Sep 2026 03:27:45 +0000

I did not set out to test synthetic-speech detectors. I was validating my own voice pipeline and needed to know whether the processing stage changed what a provenance checker would say about the output.

It did. And it did so on recordings where nothing had been generated at all.

The premise being tested

A synthetic-speech detector is used as though its score answers one question: was this audio generated? Those scores are entering forensic, journalistic and platform-moderation settings on that reading. In those settings a false positive is not a benchmark loss — it is a genuine recording being called fake, with consequences for the person who made it.

Here is the structural problem. Detectors are trained on corpora where synthetic audio and particular signal-processing histories are almost perfectly confounded. Every generated clip in a text-to-speech corpus has passed through a vocoder or a neural codec. Almost no bona fide clip has. A model minimising training loss has no incentive to separate what its training data never separated.

So I ran the confound backwards.

The design

Hold everything fixed — speaker, words, performance, room, microphone. Apply a benign transformation that produces no generated speech tokens. Measure the paired change in a frozen pretrained detector's score on the same utterance.

The transformations are deliberately unremarkable and the statistics deliberately conservative. The question is not whether an adversarial perturbation can fool a classifier. Of course it can. The question is whether ordinary processing of genuine speech is enough to move a deployed one.

Thirteen detectors, declared and hashed before any of them was scored. Four pre-registered experiments on the same 47 utterances from three speakers:

experiment	transformation	family size	Holm survivors
Detector panel	neural-codec round trip	13	9
Processing 2×2	denoise / EQ+compression / both	39	30
Reconstruction ladder	Griffin-Lim ×4, BigVGAN, Vocos, 2 rate controls	78	61
Codec family	EnCodec 6 kbps, 12 kbps, DAC 44.1 kHz	39	33

Two design decisions were fixed in advance and both matter.

No AUC gate. Running the causal test only on detectors that discriminate on this corpus would be circular — by my own hypothesis, discrimination on this corpus is substantially a measure of codec sensitivity, so gating on it would preferentially admit the detectors most likely to show the effect. Every detector ran on every condition regardless of its AUC.

A negative control on purpose. The panel includes a detector already known to be inverted, so it is visibly not a set of models picked to agree.

Genuine speech, called synthetic, with zero generation

A codec encode-decode of real studio recordings moved 9 of 13 detectors past Holm correction.

Three of those were fully prospective — nothing about their behaviour had been observed when the protocol was frozen — and all three moved on 47 of 47 utterances, at matched-pairs rank-biserial of exactly −1.000, the maximum a rank statistic can reach.

It held across three professional microphones capturing the same physical performance simultaneously, so it is not a capture-channel artefact. It held on a speaker outside the corpus.

And it is not about neural codecs. Griffin-Lim — phase retrieval published in 1984, no neural network anywhere — moved 12 of 13 detectors in the same direction. An ordinary denoise-and-master chain, the kind any production pipeline applies without thinking about it, moved two detectors by −2.95 and −4.22 native units.

The phenomenon is about reconstruction and processing history, not about codecs being special.

There is no best detector to pick

The obvious mitigation is to choose a better detector. The measurements close that door.

detector family	codec round trip	ordinary denoise + EQ + compression
codec-trained	collapses	barely moves
ASVspoof-era	null	collapses

The two families fail on opposite inputs. The ASVspoof-era models could not separate real speech from synthetic speech on this material at all — and ordinary mastering still moved them on 41 and 46 of 47 genuine human recordings.

Choosing a detector is choosing which false positive to accept, not avoiding one.

Which transformation moves which detector tracks each checkpoint's documented training exposure across three architectures. I report that as an association among frozen checkpoints, not as a causal effect of training data — the checkpoints differ in frontend construction and optimisation as well as in corpus, and no isolating intervention was performed.

The prediction I got wrong

I predicted that a detector trained on codec audio would resist a codec pass.

It collapsed hardest of anything in the panel.

That prediction is in the frozen pre-registration, so it is in the paper. An independent adversarial audit then ran three rounds against the frozen estate, parsing the stored raw scores directly rather than executing my code. It reproduced every statistic — and thirteen of my own claims were withdrawn as a result. The withdrawals are published, because a corrected claim is more informative than one that was never challenged:

withdrawn	replaced by
"the mechanism is analysis–resynthesis"	no mechanism identified
"any operation that discards phase flips these detectors"	the tested operations do
"ASVspoof-era detectors are blind to codecs"	insensitive to this codec specifically
"Griffin-Lim moves the panel harder than any codec"	EnCodec also reaches median −1.000

What this does not establish

Detectors are not broken in general. The supported claim is narrower: on this material, detector output can be strongly affected by a recording's reconstruction and signal-processing history rather than uniquely identifying synthetic provenance.
No mechanism is identified. The paper says so explicitly.
No listening test was run. Nothing here says any transformation is perceptually transparent.
Three speakers, 47 utterances, neutral speech, one language. Enough for a paired effect. Not a population claim.
Seven of thirteen detectors come from one research group. That is not seven replications.

Not an evasion method

This is a detector-robustness and forensic-attribution result and I do not present it as anything else. No parameter anywhere in this work was chosen, searched or tuned against a detector output.

The one time an intervention was tuned to a statistic, it made the audio more separable rather than less. That experiment is retained in the paper deliberately, because a failed attempt that points the opposite way to the hypothesis is worth more than a quiet deletion.

The part that generalises past audio

An instrument's score is not interpretable without knowing the provenance of what it was computed on.

Before trusting a detector panel, the question worth asking is which detectors on it can do the task on your material at all — and what else, besides generation, moves them. That is the question the companion tool answers, and it is the reason I wrote it as an instrument for measuring detectors rather than as another detector.

Reproducibility

The deposit is 26 files: the paper, the frozen per-utterance detector scores, the result and pre-registration freeze records, the five figures, and the two scripts that regenerate the figures and the rank-biserial bootstrap intervals. Both scripts were verified to run from the deposit alone and to reproduce every figure and interval file byte-for-byte.

The 11.5 GiB audio evidence estate is not released. Speaker and sample identifiers are retained deliberately as provenance under signed contract; only workstation path prefixes are stripped.

Article (canonical): https://ai.bedvibe.studio/called-synthetic/
DOI: https://doi.org/10.5281/zenodo.22819223 — all versions: https://doi.org/10.5281/zenodo.22819222

CC BY 4.0.

Three of Twenty Decoders Actually Stream, and My Quality Metric Was Beaten by an Algorithm from 1984

Panagiotis Gkilis — Thu, 17 Sep 2026 12:24:44 +0000

If you are building streaming text-to-speech, you have to pick a decoder — the thing that turns a representation into a waveform. I tried to pick one from the literature and could not, so I measured twenty of them under one frozen protocol on identical audio.

Three things came out of it. Two are about the decoders. The most useful one is about my own measuring stick, which failed a test I had written in advance specifically to catch it.

The problem with the published record

Real-time factors quoted without naming the hardware. "Streaming" claimed for architectures that need the whole utterance before they emit anything. Quality figures from different corpora, at different sample rates, through different measurement paths. Each number is probably fine on its own. None of them compose into a decision.

So: one GPU, one corpus, one protocol, twenty arms. Representation → waveform only. Generator and autoregressive timing are a different question, and a decode-only number reported as end-to-end TTS performance is a failed report.

One framing decision changed the candidate set before any measurement happened. I nearly excluded several codecs for not being causal — then realised only the decoder needs to be causal for a TTS build. The encoder runs at training time on complete utterances; at inference the generator emits tokens directly and no encoder sits in the serving path. NVIDIA's NanoCodec makes it concrete: non-causal encoder, causal HiFi-GAN decoder. "Is this codec causal?" is the wrong question.

Five gates, each frozen before it ran

gate	question
0 — candidate	which decoders are runnable, licensed and comparable at all?
1 — functionality	does it produce structurally valid audio?
2 — streaming	does it truly stream, or is it stateless chunking?
3 — performance	latency, real-time factor, first-audio, jitter, memory
4 — quality	what does the round trip cost the signal?
Q5 — identity	does the speaker survive, and do they stay one person?

The protocol for each gate was written and hashed before the data it governs existed. Two later freezes were not prospective — Gate 4's analysis layer and three revisions of the Q5 spec were post-measurement repairs, and the records say so rather than pretending otherwise. That distinction matters more than it sounds like it does, and I got it wrong in the first published version of this work.

Gate 2: "streaming" is a property almost nothing has

An arm counts as truly incremental only if both hold:

stateful chunked decode reproduces full-context output to max|err| ≤ 1e-3, and
state is load-bearing — stateless error is at least 10× the stateful error.

The second condition exists because the first can be satisfied by an arm that ignores state completely. If you only check condition 1, a stateless chunker passes and you ship it believing it streams.

Of twenty arms: three are truly incremental, all three explicitly causal FocalCodec configurations. Fifteen are stateless chunking. One streams its network but never emits audio from partial input. One was blocked on platform and reports no number at all rather than a fabricated one.

The three that pass do so with state ratios of 3,530 to 36,740 against a threshold of 10. State is not marginally load-bearing there; it is decisive.

The negative controls are what make that mean anything. The non-causal FocalCodec configurations run through the identical stateful code path. Their ratio is exactly 1 — carrying state changes nothing. If a negative control had passed, the experiment would have been void, not the arm promoted.

Several of the fifteen never claimed to stream. The classification measures what they do; it is not an accusation about what they advertised.

Gate 3: speed is not validity

First-audio latency (TTFA) across the arms that have one spans 1.37 ms to 27.64 ms at the ~80 ms chunk anchor. MelFlow does not have one: it emits spectrogram frames, and producing playable PCM from a partial stream needs an overlap-add stage this study records as not established. Its 375.91 ms is a steady-state median, a different quantity, reported separately and never mixed into the TTFA range. (I mixed them in v1.0.0. More on that below.)

Real-time margin spans 64× down to 0.2×.

But two arms produce streamed output that fails the study's own validity gate at every chunk size tested. A third fails at the anchor and passes at larger sizes — its smallest viable configuration is 640.8 ms, not 80 ms.

The sharpest case: vocos_mel24 has the fastest first-audio in the whole study and no achievable streaming configuration at all. Its timing numbers are real. What they timed is not a valid streaming configuration. If your benchmark reports latency without a validity gate, this is the arm that wins it.

One more result governs how any of these numbers may be read. A controlled probe — byte-identical tokens, same GPU, one arm run in every environment — found a 50.2% difference attributable purely to the transformers version, device-side, at identical torch. So one arm in this study may not be compared numerically against the others at all. That is not a statement about whether it is fast. It is a statement that the comparison does not exist.

Gate 4: the metric failed the test I wrote to catch it

This is the part worth your time even if you never touch audio.

Gate 4 ranks on one metric, mel-cepstral distance, chosen after nine variants were implemented and attacked. Alongside it, the pre-registration installed a validity instrument: put Griffin-Lim in the panel. Griffin-Lim is a phase-retrieval algorithm from 1984 with zero trained parameters. The pre-registration stated in advance what it would mean if Griffin-Lim scored close to the trained neural codecs — a finding about the metric, not about the decoders.

It did not score close. It scored best, in six states of six.

The obvious explanation was tested and refuted. The hypothesis that the metric's low-level frame floor favours a magnitude-matching algorithm failed: splitting each cell at its own median energy, Griffin-Lim leads in both halves, and by more in the loud half.

What remains is an explanation consistent with the result rather than a demonstrated mechanism, and the difference is worth being careful about. Mel-cepstral distance here is derived from the magnitude spectrum and is blind to phase by construction. Griffin-Lim iterates toward consistency of the linear STFT magnitude and optimises nothing else, accepting whatever phase error that leaves. Aligned in domain — but not the same quantity, and nothing in this package demonstrates that Griffin-Lim minimises the released cepstral distance exactly.

What is established does not depend on the mechanism: a metric whose ranking puts an untrained phase-retrieval algorithm first is not, on its own, a sufficient authority for cross-decoder quality.

That is the whole lesson, and it generalises well past audio. The pre-registration naming this outcome in advance is the only reason it reads as a finding rather than an embarrassment — and the reason no metric was swapped in afterwards to produce a tidier table. If you are choosing anything on a published quality number, the question worth asking first is whether that metric has ever been shown a floor it should beat.

Q5: does the speaker survive?

A single similarity score conflates two different failures, so this measures them separately. Retention: how close a reconstruction sits to its own source. Dispersion: how tightly different reconstructions of the same speaker cluster. A decoder can move a voice consistently — a stable new identity — or make the speaker wander. Those are not the same problem and they do not have the same fix.

Three separately calibrated speaker encoders, chosen mechanically from a previously frozen fourteen-encoder study before any of these numbers existed. Cosine values are encoder-specific and never averaged. The three encoders show broadly similar ordering with some pairwise reversals, which is exactly why a bare cosine without its encoder name is not a result.

The streaming detector compares each arm against its own offline decode on the same recording, so no ceiling or floor enters and no cross-arm ranking is implied. Two distinct quantities, never quoted as one range: S1, the offline↔streamed cosine, and S2, the paired retention delta.

For the three causal arms, across all three encoders: S1 0.999866–0.999946, S2 −3.9×10⁻⁵ to −5.3×10⁻⁶. A near-zero median additional change, under the tested encoders, supported recordings and imposed chunking regime. Small, and consistently negative — this study establishes no minimum detectable change and no equivalence threshold, so that is not a claim that the difference is below what the instrument can see.

The other thirteen lose −0.078 to −0.742 on S2, which is nowhere near that scale. For one arm the offline-versus-streamed cosine itself reaches −0.013: the streamed output is essentially orthogonal to that arm's own offline decode in this embedding space. That says nothing about waveform or linguistic relationships, which were not measured.

This agrees with Gate 4's separate streaming detector — an aligned maximum-error detector on the waveform — which found the same three arms at 0.69–0.74% error where every other arm sat at 105–225%. Two measurements sharing no implementation, the same three survivors.

They do share their input: the same reconstructions of the same corpus. So the agreement is corroborating evidence, not proof. Two implementations agreeing makes an independent error in both less likely; it cannot rule out an error in either, and it says nothing about an artefact of the material both were computed on.

What this does not establish

Perceptual quality was never measured. No listening test was run. Which decoder sounds best is not established and no sentence here implies it. Two metrics ranked a phase-blind algorithm first; both are blind to the same thing.
The identity result is about speaker-encoder representations, not about how audio sounds or about cloning quality.
A large majority of cells fall outside the quality metric's validated support domain. Support is carried as a covariate, never as a silent exclusion.
For one decoder family, dispersion and band-limiting cannot be separated in this design, so no causal claim is made about it.
Seven speakers, one studio, one recording chain, English. Absolute values are not comparable to VoxCeleb-scale benchmarks.
Thirteen arms were driven with chunked context they were never designed for. The result is the cost of streaming a decoder that was not built to stream — not a defect report.

The part I got wrong, and why it is in the article

Six rounds of independent adversarial review ran against this work. Four before publication found three defects that changed conclusions — a reference that admitted pairs the measurement forbade, a ceiling measured on the wrong duration, and a pair-grouping convention that was order-dependent.

Then a fifth round ran after publication, against the published package, and found what four earlier rounds structurally could not: they had all audited the private research tree, not what shipped from it.

All three of my advertised public reproduction commands failed from a clean copy of the release. The scripts resolved their data relative to where they sat in my private tree — one directory above the artifacts — and the release ships them flat. The claim audit opened a deliberately withheld file unconditionally. And the verifier I pointed readers at was the manifest of the private tree, which cannot verify a redacted package and duly reported 26 of 32 artifacts as drift. A first-audio range had borrowed a steady-state number for its upper end. Three sentences claimed more than the measurements carry.

A sixth round then attacked the correction itself and found five more, two of them claims in the repair that outran their evidence in the same way as the originals.

The pattern has now recurred six times in different disguises: verification that checks what is present rather than what is required. The estimator without its reference. The instrument without its evidence. The manifest without its inventory. A package that verified its own private origin rather than what it shipped. A privacy scanner that listed the paths it knew about instead of declaring what counts as private. A verifier that printed its own integrity fingerprint without ever comparing it to anything.

That last one is my favourite, because it looks like working code. It computed the hash. It printed the hash. It printed the recorded hash right underneath. It never compared them, so corrupting the fingerprint exited 0. Printing a number beside another number is not a check. Only a comparison that can fail is one.

All of it is corrected in v1.0.1, itemised with the original wording beside the corrected wording. No measurement changed — 51,495 non-string scalar leaves were compared across every JSON artifact common to both versions, and none differ. v1.0.0 stays permanently available, because deleting the version that was wrong would be the same mistake in a different disguise.

Reproducing it

Three commands run from the public package with no private input: regenerate every table, re-trace every headline claim to the artifact it came from, and re-hash the bytes that shipped.

That is a consistency check, and the README now says so rather than overclaiming. It is not independent reconstruction — rebuilding the aggregates and their confidence intervals needs the withheld per-recording rows and speaker embeddings, which are not published because an embedding of a named consenting human is biometric data. One published claim is marked WITHHELD rather than PASS in the public audit for exactly that reason. Source audio, embeddings and the 35.65 GB of reconstructions stay private; the reconstructions are retained because human listening and adversarial detectability are declared later phases.

Article (canonical): https://ai.bedvibe.studio/decoder-benchmark/
Paper + code + full audit trail: https://github.com/Mormolykos/decoder-vocoder-benchmark
Release: https://github.com/Mormolykos/decoder-vocoder-benchmark/releases/tag/v1.0.1
DOI (v1.0.1): https://doi.org/10.5281/zenodo.22811349 — all versions: https://doi.org/10.5281/zenodo.22798415
Corrections: https://github.com/Mormolykos/decoder-vocoder-benchmark/blob/main/CORRECTIONS_v1.0.1.md

CC BY 4.0.

The Compiler Got 5% Slower. The Benchmark Called It a 10% Regression a Quarter of the Time.

Panagiotis Gkilis — Fri, 04 Sep 2026 20:43:20 +0000

A benchmark suite is an instrument. You point it at two versions of a compiler, it tells you whether the new one got worse, and somebody merges or reverts a pull request on the strength of that answer.

If the compiler is stochastic and the suite does not pin its seed, that answer is a draw from a distribution nobody measured.

I measured the distribution.

The finding

At the pinned revision I inspected, IBM's Benchpress fixes the seeds that build its circuits — seed=12345, fourteen occurrences — and passes no seed_transpiler to the code that compiles them. Circuit construction is deterministic; compilation is not.

On bv_n140 mapped to a heavy-hex lattice, going from Qiskit 1.4.3 to 2.0.0, across 400 seeds per version and 21 OS processes — a separate measurement from the pre-registered experiment below, and not part of the pre-registered endpoint:

quantity	value	95% interval
long-run mean change	+5.37%	+4.27% to +6.50%
called a ≥10% regression, 3 runs/version	24.4%	19.5% to 31.1%

The true change is nowhere near the threshold. The verdict crosses it a quarter of the time anyway.

I did not trust that at first, because all 200 original seeds came from one contiguous block inside one process — so per-process state, hash randomisation and allocator layout were all held constant. A disjoint set of 200 seeds spanning 5.9 million to 1.08 billion, across ten fresh processes with differing PYTHONHASHSEED, gave 22.6% [16.4, 30.6] against 26.3% [18.5, 34.2] from the contiguous set. Each point falls inside the other's interval.

Running it more does not fix it

The obvious response is "run it more times". Pooling all 400 seeds:

runs per version	1	3	5	8	10	20
false-positive rate	34.58%	24.38%	18.56%	12.94%	10.36%	3.74%

Twenty runs per version is 20 × 2 versions × roughly two hours per suite run — the issue's own figure, which I could not independently confirm — so on the order of 80 compute hours. It still leaves 3.74%.

Fixing seed_transpiler removes the variance attributable to the transpiler seed at a single run. It does not make one run correct. A fixed seed makes the answer repeatable, including when that answer is wrong: on bv_n280, recorded seed 663193 gives A = 1040 against B = 1157 — a +11.25% call against a measured θ of +4.89%. Frozen, reproducible, and the wrong side of the threshold.

Taking the minimum of three runs instead of the mean — a common instinct — makes it worse. Every rate roughly doubles.

The pre-registered part

One circuit is an anecdote. So I wrote down the selection rule, the analysis, the endpoint and the labels and committed them before the first tracked commit of the 39-circuit raw dataset. The commit ordering is checkable in the repository.

What that ordering supports is a frozen follow-up design after exploration, not blindness to earlier outcomes — and the paper corrects that distinction twice. A census and exploratory results preceded the pre-registration: an earlier commit the same day already identifies heavy-hex as the worst case, and the ≤10 s selection rule is computed from that pre-existing census. Git records when files were committed; it does not record when their outputs were first looked at.

The rule selected every qasmbench-large circuit whose twelve-seed heavy-hex runtime in that census was under ten seconds: 39 circuits, 200 recorded seeds per arm, 15,600 primary transpilations. 36 resolved; 26 were eligible after pre-registered boundary exclusions.

Over those 26:

12/26 met the pre-registered endpoint — a risk interval excluding zero
7/26 carry a risk of at least 5%
4/26 carry a risk of at least 10%

The median circuit admits an ambiguity band of about 10.9 percentage points — the width of the interval of true change over which the three-run rule's call probability runs from 5% to 95%. That band is built from the measured residuals of this version pair under a stated model. It is not claimed to be version-pair-independent.

What I withdrew

That 46.2% headline rate is the number I trust least, so I want to say why rather than let someone find it.

The 26 circuits are not 26 independent observations. They come from eleven algorithm families and within a family the results are nearly all-hit or all-miss. A cluster bootstrap over families gives [17.4%, 81.0%] against the Wilson interval's [28.8%, 64.5%]. The effective sample size is far below 26, so no interval I can compute is informative about a population of circuits.

The suite-level rate is withdrawn. 12/26 stands as a descriptive count of these 26 circuits and nothing more.

Three other claims went the same way. One was a correlation of ρ = +0.876 between "compilation is stochastic" and risk, which I had presented as a discovered mechanism. Circuits whose two arms are constant have zero risk by arithmetic, so a correlation computed over a sample containing them measures a definition. Simulated data containing no compiler at all reproduces the statistic.

All four withdrawals are still in the repository, with the evidence that defeated them.

Checking the checker

The paper argues that measurement apparatus should be checked rather than trusted, so it would be poor form not to turn that on myself.

The repository ships a thirteen-stage read-only verifier and a suite of 44 deliberate corruptions, each of which must be rejected by a named stage for a named reason — not merely cause a nonzero exit. The paper's own account of the review of its claims is four rounds of hostile review plus two first-principles audits, which removed five claims from the work. Every audit round after that attacked the apparatus instead. Every one of them ended with a green verifier that the next reader then broke, and not one of them changed a scientific number. What they broke was the checking.

The best example: a claim scan that could not see the sentence "Seven of the twenty-six eligible circuits have an interval that excludes zero" — because twenty-six was not a number it recognised, so no claim object was ever created and the fail-closed classifier was never asked. Four rounds of widening a recogniser lost to the fifth spelling. The fix was to stop asking "can we recognise every English sentence that means 7 of 26" and instead register every reader-visible unit of the frozen manuscript by hash, so an unregistered block is refused before anything reads a word of it.

That is evidence the verifier can turn red. It is not a proof that nothing gets through, and the paper says so in those words.

What this does not say

It does not say Qiskit's transpiler got worse, or that Benchpress is a bad suite, or that any published decision was wrong. It measures one property of one benchmarking protocol: an unseeded stochastic compiler, sampled three times, produces regression verdicts that disagree with its own long-run behaviour at rates that are not small.

Limits, stated plainly: one SDK, one version pair, one topology family, one machine. The +10% threshold is mine — Benchpress defines none. seed_transpiler is not a universal remedy either: it removed false positives on the circuits I tested but was worse on three of the four showing false negatives.

If you maintain a benchmark like this

The cheap version of the check: run the same comparison twenty times without changing anything, and look at the spread of verdicts, not the spread of values. If the verdict moves, the number of runs is part of your instrument and it belongs in the write-up.

Version 4 is the audited release. Paper, code, raw per-seed data and the assurance apparatus:

DOI — 10.5281/zenodo.22848938
Tagged release — github.com/Mormolykos/qvalidation @ v4.0.0
Full write-up — ai.bedvibe.studio/decision-risk/

CC-BY-4.0.

I Assumed My Retriever Failed at Stage One. The Bigger Failure Was at Stage Three.

Panagiotis Gkilis — Mon, 31 Aug 2026 16:38:31 +0000

Hierarchical retrieval is supposed to help on long documents. Pick the right chapter, then search inside it. Narrow the haystack, then find the needle.

On my long-book benchmark it came third of five, behind a chapter-summary chain and flat chunk RAG, and only barely ahead of naively grabbing the end of the book.

method	context precision	context recall
`naive_first_context`	0.1475	0.1458
`naive_last_context`	0.4150	0.3302
`flat_chunk_rag`	0.3375	0.4302
`chapter_summary_chain`	0.4000	0.4771
`hierarchical_book_rag`	0.3475	0.3365

My hypothesis was error compounding: if stage one picks the wrong chapter, every later stage searches the wrong text. That turned out to be true and incomplete — which is worse than being wrong, because it would have sent me to fix the right thing and stop there.

The ablation

Same corpus (240,767 words), same 80 gold questions, five variants, 400 scored rows. Two of the variants are given the correct chapter — diagnostics, not deployable methods — purely to measure how much is lost before versus after chapter selection.

method	recall	precision	hit@1	hit@3	hit@5
`hier_current`	0.3365	0.3475	0.2000	0.3875	0.4375
`hier_no_neighbors`	0.4771	0.4000	0.2000	0.3875	0.4375
`chapter_summary_chain`	0.4771	0.4000	0.2000	0.3875	0.4375
`hier_oracle_chapter`	0.7844	0.6796	1.0000	1.0000	1.0000
`hier_oracle_chapter_neighbors`	0.7688	0.6925	1.0000	1.0000	1.0000

Deleting a feature closed the entire gap. hier_no_neighbors lands on 0.4771 / 0.4000 — the same numbers as the baseline, to four decimals. The method I thought was structurally worse wasn't worse. It was carrying a stage that was hurting it.

And look at hit@k. Identical across the first three rows. Chapter selection did not change at all between hier_current and hier_no_neighbors — same code. Every point of recall that moved, moved downstream, in the stage that pads results with adjacent chunks.

Counting failures directly

failure type	count
`neighbor_dilution`	32
`wrong_chapter`	27
`right_chapter_wrong_chunk`	7
`ok`	14

My hypothesis accounted for 27 questions. The stage I hadn't suspected accounted for 32. I would have fixed chapter routing, seen a real improvement, and never looked at expansion — because the improvement would have confirmed the theory I walked in with.

Neighbour expansion is not simply bad

With the oracle chapter, adding neighbours moves recall 0.7844 → 0.7688 and precision 0.6796 → 0.6925. Recall down slightly, precision up. When you're already in the right chapter, expansion is roughly a wash and can help precision.

The damage happens when expansion runs on top of an uncertain chapter choice, crowding out the good chunks you did find with text that is only adjacent, not relevant. It is a stage whose value depends on the confidence of the stage above it — exactly the interaction a single end-to-end score cannot show you.

So the finding is not "turn off neighbour expansion." It is treat it as a tunable stage rather than a default, conditioned on router confidence.

The sequence that worked

Disable each optional stage in turn. Cheap, and it found the larger of my two problems.
Insert an oracle at each boundary. Not deployable, but it partitions the loss into before/after.
Only then attribute the failure.

If I had skipped to step three, I'd have published "hierarchical RAG underperforms chapter-summary retrieval on long narrative corpora" — well-supported by my headline numbers, and wrong about the cause.

What this does not show

One private narrative corpus, 80 gold questions written from the corpus rather than by independent annotators. Evidence-term overlap scoring, which is a lightweight audit signal and not full semantic correctness. No confidence intervals — the package didn't compute them and I won't imply precision I didn't measure. The oracle variants read gold labels and are not production-realistic. This is not a universal rule against hierarchical retrieval.

Paper: Diagnosing Hierarchical Retrieval Failure in Long-Document RAG
DOI: 10.5281/zenodo.20692450

One Integer Deleted the Stop Token From My Loss. The Curve Never Noticed.

Panagiotis Gkilis — Mon, 31 Aug 2026 16:37:39 +0000

For a long time I had a model that trained cleanly and produced nothing usable. The loss fell. Gradients were finite. Nothing crashed. It simply never learned to stop — every generation ran to the length cap and got truncated.

The cause was two lines of my own source that were individually correct.

The collision

In a neural codec language model the audio vocabulary has a fixed size, and the stop token is one extra class on top. So the output layer is one wider than the codebook:

nn.Linear(d_model, NUM_AUDIO_TOKENS + 1)   # 1025 classes: 0..1023 audio, 1024 = EOS
eos_id = NUM_AUDIO_TOKENS                  # 1024

Correct. EOS is the last class and the layer has room for it.

Then, elsewhere, the loss:

F.cross_entropy(logits, targets, ignore_index=NUM_AUDIO_TOKENS)

Also reasonable on its own. ignore_index is how you skip padding.

But the sentinel and the stop token are the same integer. Every position whose target was "stop" was discarded before the loss was computed. Not down-weighted — removed. The model was never once shown an example of stopping, across every epoch it ever ran.

PyTorch defaults ignore_index to -100 precisely because it must be a value that can never be a real class. Replace it with a real vocabulary constant and that guarantee is gone, silently: shapes valid, loss finite, run healthy.

The curves are identical

Minimal reproduction, two arms differing only in the sentinel value.

The broken arm finished at 0.0035. The fixed arm finished at 0.0034.

Same curve to any human, any dashboard, any threshold you would write. One has a working objective and one has an objective with a hole in it, and the loss cannot distinguish them — because the loss is computed over what survived the mask. A metric cannot report on examples it never received.

The second half: lowest loss, worse model

200 epochs at lr = 2e-5, evaluated every 50 on a fixed split held out by utterance (n = 32). Rank is the stop token's position among 1025 classes at the true terminal frame.

epoch	training loss	mean P(stop)	argmax = stop	self-terminated
50	2.199	0.4218	16/32	2/8
100	1.628	0.4655	18/32	6/8
150	1.302	0.2159	8/32	3/8
200	1.429	0.1818	5/32	3/8

Between epochs 100 and 150 the training loss improved by 20% while mean P(stop) fell 54%, top-1 stop accuracy went 18/32 to 8/32, and autonomous termination halved.

Selecting by lowest training loss returns epoch 150. The model that terminates reliably is epoch 100. The reversal was observed independently in a shorter run, which is why I am willing to state it.

"Save the checkpoint with the lowest validation loss" is the default in more or less every training script in existence, including mine. On this run it was actively the wrong rule, and the number it optimised looked better the whole way down.

For completeness, what fixing the collision bought on the real model, same held-out split:

checkpoint	mean P(stop)	argmax = stop	rank
random initialisation	0.001848	0/32	111.6
after correction	0.4655	18/32	2.1

End-to-end synthesis then terminated on its own at frame 203 against a 350-frame ceiling. Before the fix that was impossible by two independent mechanisms: the training-time collision above, and an inference-time mask that set the logit of every index at or beyond the codebook size — including end-of-sequence — to negative infinity before sampling.

One honest loose end: P(stop) plateaus in the range 0.35–0.47 across two learning rates and a 5.9x increase in training data (224 to 1313 utterances, with speaker, language, emotion and the held-out split all held constant). So the plateau is not a data-quantity limit. I have no confirmed explanation for it. Terminal timing in speech is genuinely ambiguous, and hedging with the stop token ranked second of 1025 may simply be correct behaviour.

The same integer, safe one stage over

The detail I find most instructive: the identical line is harmless in the next stage of the same model.

The autoregressive stage predicts the first codebook plus EOS — 1025 classes, so 1024 is a real class and using it as a sentinel is fatal. The non-autoregressive stage predicts audio codes only — 1024 classes, so 1024 is out of range and the exact same ignore_index is correct.

One + 1 in a different file decides whether that line destroys your objective. Both stages read identically at the call site. That is not a mistake you catch by reading carefully; it is a mistake you catch by checking a relationship between two numbers that never appear together.

Which is what a linter is for

This is now two rules in trainproof, my linter for training runs:

Sentinel collision. Compare the output layer's class count against ignore_index. If the sentinel is a valid class, fail. If it sits exactly one past the end, say so explicitly — because the same integer is fatal one class earlier, and that distinction deserves to be visible rather than silently passed.
Dead class. Accumulate which classes ever reach the loss as a positive target during the first epoch, then flag any class the output layer can emit but that never once appears as an answer.

My own two stages are the regression fixture — the fatal case and its safe twin, one integer apart. Not a synthetic example.

The design decision worth stating: the dead-class rule only fires when coverage is already broad and few classes are missing. One unseen class out of 1025 is a structural exclusion. Nine hundred unseen is a small sample. Without that guard the check screams on every short run and gets switched off — which is how good checks die.

If you take one thing from this

I do not think this is rare. Any codebase where a padding sentinel, an end-of-sequence id and a vocabulary size are all defined as named constants in different files can produce it, and none of your instrumentation will complain.

If you fine-tune anything with a custom ignore_index, go and check it against your output layer's width right now. It takes thirty seconds and the failure mode is completely silent.

Paper: The Loss Curve Is Not a Sufficient Statistic — Silent Objective Failures from Sentinel-Class Collisions in Neural Codec Language Models

DOI: 10.5281/zenodo.21864658

pip install trainproof — GitHub, MIT

Fourteen Speaker Encoders Heard the Same Voice. Their Error Rates Differed Five-Fold.

Panagiotis Gkilis — Mon, 31 Aug 2026 16:37:35 +0000

A speaker encoder turns a few seconds of speech into a vector, and the distance between two vectors is treated as an answer to "are these the same person." That number gets used for authentication, for voice-cloning pipelines, and increasingly as the yardstick in papers asking whether a synthetic voice preserved someone's identity.

The number is not neutral. It is a reading from an instrument, and I wanted to know how much the instrument was contributing.

So I ran fourteen of them over an identical, frozen list of 11,935 trials.

The corpus is why the question is answerable

Most emotional-speech corpora use different sentences for different emotions. A model can then learn angry vocabulary instead of angry delivery, and you cannot separate them afterwards.

This one can't make that mistake. Eight speakers each recorded the same 1,360 sentences in six phonation states — neutral, happy, angry, scared, shouting, whisper. Lexical content is fixed while phonation varies.

Enrolment is neutral speech. The test is whether the encoder still recognises the person while they shout or whisper. The trial list was frozen in writing before a single model loaded, and every encoder scored the identical list — so every comparison is paired.

The headline

Equal error rate spans 0.047 to 0.233 across the panel — same trials, same audio, nothing varying but the encoder. 29 of 91 pairwise comparisons survive Holm–Bonferroni correction.

If you are choosing an encoder for expressive audio, that five-fold spread is your decision. It is larger than most architectural differences people argue about.

CAMPPlus is last, without exception

Highest error of the fourteen, all 13 of its comparisons surviving family-wise correction — for every one of the eight speakers and every one of the six conditions. Margins run from +0.087 [+0.064, +0.127] against ECAPA-TDNN to +0.185 [+0.150, +0.215] against the lowest-error encoder.

The mechanism is the useful part. Its genuine scores are unremarkable; its impostor scores are the problem. It places different people at cosine 0.28–0.34 where ECAPA-TDNN and ReDimNet place them near 0.034. That is a compressed embedding space — different speakers land close together.

It is visible at neutral speech too (0.031 EER where several ReDimNet checkpoints reach 0.000). Expressive speech doesn't cause it. Expressive speech makes it visible.

A widely used open-source TTS system conditions on CAMPPlus. To be precise: that is a verified fact about its source code, not a measurement of its audio. I measured encoders in isolation on human recordings. Whether that choice degrades output is a separate experiment I have not run.

Parameter count does not order the panel

Spearman ρ(parameters, expressive EER) = +0.135 over fourteen encoders. The 20.8M model ranks 13th of 14. A 4.81M model ranks 1st. Published VoxCeleb1-O position doesn't transfer either — that leaderboard is computed on calm read speech, which is not the condition failing you.

I retracted my claim about the winner, incidentally. It survives as an argmin (selected in 89.4% of speaker resamples, leading all eight leave-one-speaker-out refits) but after correction it is separated from only 7 of 13. The defensible wording is lowest observed error.

A clean pre-registered negative

Spectral denoising does not materially change the measurement: |ΔEER| ≤ 0.039 over 10,765 twinned trials, and whispered speech — where removing aspiration noise was the stated worry — is among the least affected.

Ten retracted claims, published on purpose

An 11.7 dB spectral effect, asserted with an instrument whose F0 artifact budget is 12.09 dB. Sweeping F0 with the spectral envelope held fixed moved the descriptor further than the claimed effect. It was never a measurement.
"Shouting does not transfer." Judged on RMS after peak-normalising every clip, so the figure was crest factor, not loudness.
A tie-break defect in the EER estimator. argmin returns the first minimiser rather than the balanced operating point — invisible at unit weights, common under the integer speaker multiplicities a bootstrap runs at. Fixing it moved one comparison across Holm, 28 → 29 of 91. The correction favoured the paper, which is why it is published rather than quietly kept.

The rule I have now paid for three times: validate the instrument before measuring with it.

Statistics

The trial list is a directed graph on 8 speakers — genuine trials are self-loops, impostor trials the 56 ordered edges. The independent unit is the speaker, on both sides. So the resample is a vertex bootstrap: draw 8 vertices with replacement, take the induced sub-multigraph, weight the impostor edge by the product of multiplicities.

An earlier scheme resampled enrolment speakers only, dropping a speaker as enrollee while leaving them in as impostor. Fixing it widened marginal intervals (width ratio median 1.11, max 1.58) and cost two already-written claims. Paired differences barely moved, because both encoders score the same impostor clips and the dependence cancels in the difference.

B = 20,000, seed fixed, families declared before testing, Holm–Bonferroni at α = 0.05.

The companion result, which is the cheap fix

Enrolling on expressive material instead of neutral-only recovers the loss on all 14 encoders, every one surviving Holm.

The mechanism is the opposite of the intuitive one. Enrolling on a single expressive state helps that state and slightly harms the others — transfer-matrix diagonal median −0.082, off-diagonal median +0.010. Guessing which state a user will be in, and guessing wrong, is worse than enrolling on calm speech.

A pre-committed control ruled out the competing explanation: acoustic diversity alone delivers +0.002 where composition delivers −0.065. It is coverage, not expressiveness. Spread enrolment thinly across states.

What it does not claim

Nothing about any TTS system's output quality. Eight speakers, one studio, one recording chain. Enrolment and test share a session, so absolute error rates are optimistic and not comparable to VoxCeleb — only relative comparisons on identical trials are licensed. Architecture is fully confounded with training corpus; separating them requires training.

The generalisable part

An established metric is not automatically a neutral one. An encoder, an evaluator, a classifier, an LLM judge — any of them can become part of the experiment rather than a window onto it.

Before asking whether a system preserves identity under some transformation, ask whether your instrument has been validated under that transformation. Otherwise you attribute the encoder's behaviour to the speaker, and the number looks perfectly reasonable while you do it.

Paper: Speaker Encoders Disagree About Who You Are When You Shout — Zenodo, CC-BY-4.0.
DOI: 10.5281/zenodo.22158030
Code, data and the full decision record: github.com/Mormolykos/speaker-encoders-disagree
The eval library the analysis checks itself against: pip install spkproof

Two of My Instruments Were Wrong, in Opposite Directions

Panagiotis Gkilis — Fri, 28 Aug 2026 13:30:49 +0000

On 28 August 2026 I audited 38 live pages and fixed some ordinary defects. The two things worth writing down both came from my own tooling being broken. One script reported three pages I had just fixed as still broken, because it did not recognise a relative link. One date parser had thrown an exception on every crawl-statistics read since it was written, so the report that answers "is the crawler arriving at all" had never once returned a number.

This is a note about measurement rather than about search engines. The setting is an indexing problem on one of my hosts, but the transferable part is that I spent a day acting on numbers produced by instruments I had never checked, and both were wrong — one in each direction. One made a working fix look broken. One made a working crawler invisible.

Crawling and indexing are not the same thing

Everything below depends on this, and most writing on the subject collapses it into one word.

Crawling is retrieval. A crawler requests a URL and receives bytes. That is all it means. A page can be crawled every day for a year and appear in no search result ever.

Indexing is selection. Having retrieved the page, the engine decides whether to keep it in a form it will serve. That is a judgement, each engine makes it independently, and none of them will tell you the reasoning.

So there are at least three states worth separating: never fetched, fetched and declined, and fetched and kept. "My pages aren't indexed" describes the middle two identically, and their causes have nothing in common.

The audit, kept in proportion

On 28 August 2026 I fetched every URL in the sitemap — 38 pages on the affected host, 20 on my notebook site — over plain HTTP with no JavaScript executed, and produced a number per page rather than a verdict. That rule exists because an earlier audit returned "all clean" for months while the homepage was handing crawlers a fraction of its navigation in a form they could use.

It found real defects. Five pages had zero followable links: a crawler arriving there could go nowhere. The worst had two navigation controls, both <button onclick> — usable with a mouse, invisible to a crawler. Three more bound their only navigation to a JavaScript click handler, which has the same effect.

38 pages, fetched live	Before 28 Aug	After 28 Aug
Pages with zero followable links	5	0
Navigation a crawler cannot follow	2	0
Pages with an incorrect `h1` count	1	0
Missing robots directive	1	0
Canonical problems	0	0
Images with no `alt` attribute (of 199)	0	0
Non-200 responses	0	0

This is housekeeping. It is worth doing and it is not a finding.

Instrument one: a false negative that looked exactly like a failed fix

After deploying, I re-ran the audit against the live site. It reported that three of the pages I had just fixed still had zero followable links.

They did not. I fetched one by hand and the anchor was plainly in the served HTML.

The bug was mine. The script counted links by matching href values that started with / or http. The anchors I had added were relative — href="./" — so they matched nothing and counted as zero.

What makes this worth writing down is the shape of the failure. A measurement that silently under-reports is indistinguishable from the defect it is supposed to detect. Every downstream step would have been reasonable and wrong: conclude the deploy failed, hunt for a caching problem that did not exist, possibly "fix" three pages that were already correct and introduce a real defect doing it. Nothing in the output looked like an error. It looked like bad news.

The only thing that caught it was retrieving the page and reading the markup.

Instrument two: a report that had never returned a number

The second failure was older and much more expensive.

Bing's webmaster API serialises dates in a Microsoft-era format — /Date(1784190745000)/, epoch milliseconds wrapped in a string. My parser handled that correctly. But one endpoint, and as far as I can tell only one, returns a second variant carrying a UTC offset:

/Date(1786345200000-0700)/

My pattern captured "digits and minus signs". It swallowed the offset into the number, and the integer conversion raised an exception every time.

Because that variant appears in exactly one report, the failure was invisible everywhere else. Every other call worked. And the one report it broke was crawl statistics — the nearest available substitute for origin access logs, and the first thing to check when asking whether a crawler is showing up.

It had never once returned a value. I had been reasoning about crawler behaviour for weeks with the most direct instrument for it throwing an exception on every call. The fix is one line; the test asserts that both offset signs resolve to the same instant, because the value before the offset is already UTC.

What the repaired report showed

Date	Pages crawled that day (Bing)	Pages held in index (Bing)
2026-08-24	15	33
2026-08-25	27	33
2026-08-26	36	34
2026-08-27	23	34

Bing crawls this host every day, 15 to 36 pages, and holds 34 of roughly 38 in its index. Google holds none of them.

This is not a host a major crawler visits occasionally. It is one a major crawler works through continuously, and has done throughout the period I spent trying to work out why crawlers were not engaging with it.

Sorting the evidence

Verified directly from the live sites (2026-08-28): every row of the 38-page audit, fetched over HTTP with no JavaScript, re-verified after deployment.

Measured from Bing's API (2026-08-24 to 2026-08-28): the daily crawl volume and index count. One engine's figures about its own behaviour — good evidence about that engine, no evidence at all about any other.

Corrected tooling: two bugs, both mine, both fixed on 2026-08-28, one now covered by a regression test. The numbers here are post-correction. The pre-correction ones were wrong in a way that read as a genuine finding, which is the entire point.

Observation: two major engines, given the same host, the same bytes, the same permissive robots.txt and the same sitemap, reached opposite conclusions. One indexes nearly everything. One indexes nothing.

Unresolved: why. I do not know, and I am not going to name a mechanism to round the story off. Index selection is independent between engines, so an asymmetry is permitted and is not by itself evidence of a fault anywhere.

What the asymmetry does do is retire a family of explanations. A page a large crawler retrieves daily and chooses to keep is demonstrably reachable, parseable, renderable and acceptable to a serious indexing pipeline.

To be explicit, because the genre invites the opposite reading: nothing here fixed the zero. It was zero before the audit and zero after. This is not a recovery story.

The prediction, recorded before the outcome

The obvious next question is whether the fixes helped. I cannot answer that today — the changes went live on 28 August and there is no after to compare against.

So instead of waiting and then claiming a prediction I never wrote down, here it is in advance.

Baseline, 2026-08-28. Google: 0 pages indexed on the affected host. Bing: 34 held, 15–36 crawled per day. Audit defects all at zero.

I will re-measure on 2026-09-15, with the same script and the same API calls, and publish the result whichever way it goes.

What I expect: no material change in either engine. The defects I fixed were real, but none plausibly explains a host-wide zero, and Bing was already crawling this host thoroughly while every one of them was still present — fairly direct evidence they were not what stood in the way. Earlier in this investigation I also ran the standard remedies against a held-out control group and measured no effect: treatment 32%, control 33%.

If the numbers move, my model is wrong and that is the more interesting outcome. If they do not, I have a dated negative result instead of a vague impression.

What I would take from this

Check instruments against reality before trusting their output. Two of mine were wrong on the same day, in opposite directions. Both would have survived indefinitely if I had only read what they printed.
A silent nothing is worse than a loud failure. Make a missing measurement say missing — never zero, and never an empty report.
Separate crawled from indexed in your own vocabulary first. The fixes for "never fetched" and "fetched and declined" have almost nothing in common.
Count links on the page, not links to it. A page with no outgoing anchors is a dead end however many things point at it, and a click handler is invisible to the thing you are trying to persuade.
Fetch it and read it. Every real finding here came from the actual bytes. Every error came from believing a summary.
Write the prediction down first. It costs one paragraph and it is the difference between a result and a story.

The honest summary of the day: I improved a site that was already technically sound, repaired two instruments that had been quietly lying to me, and confirmed an asymmetry I can describe precisely and cannot explain at all.

Originally published at ai.bedvibe.studio/broken-instruments, where I will post the 15 September re-measurement against the prediction above.

I write up measurement and instrumentation work at ai.bedvibe.studio — including the control-group indexing experiment this one follows, and a crawler-behaviour observatory that logs which AI and search crawlers actually arrive. The studio itself is at bedvibe.studio.

My Agent Answers From 0.6% of Its Corpus and Reports It Like a Full Read

Panagiotis Gkilis — Tue, 25 Aug 2026 08:18:39 +0000

Originally published at ai.bedvibe.studio.

My portfolio RAG agent holds 1,003 indexed chunks. A question retrieves six and answers.

Six of 1,003 is 0.598%. Nothing in the response, the logs or the trace says the other 997 were never looked at. It reads exactly like an answer built from reading everything.

That is fine for "what does he say about X." It is not evidence for "does he mention X anywhere" — and there is nothing in a normal RAG stack that tells those two apart.

The claim retrieval cannot support

A compliance corpus: 2,431 policy documents. Someone asks whether there is a remote-work reimbursement policy. Retrieval returns eight chunks. None mentions reimbursement. The agent answers:

There is no remote-work reimbursement policy in the corpus.

That claim requires knowledge of 2,431 documents. It was made from eight. The agent did not lie and it did not hallucinate — it correctly reported what it found, and the shape of the sentence quietly upgraded a statement about eight documents into a statement about the corpus.

Absence of evidence in a retrieved fragment is not evidence of absence in the whole. Everyone knows this. No retrieval stack I have used records enough to enforce it, because the denominator is not carried anywhere near the answer.

"But our faithfulness score is 1.0"

This is the first objection and it deserves a straight answer: faithfulness cannot catch this, by construction.

Faithfulness asks whether the answer is supported by the retrieved context — did the model invent anything beyond what it was given. The compliance answer above invents nothing. It scores a perfect faithfulness and may be false about 2,423 documents.

Faithfulness scores answer against context. Coverage scores context against corpus. Different axes, and a system can be perfect on the first while completely silent on the second.

Metric	What it needs	Available at answer time?
Faithfulness	answer + context	Yes
Context precision	answer + context	Yes
Context recall	ground-truth annotations	No — offline evaluation only
Corpus coverage	corpus size + retrieved count	Yes — and it is not reported

Context recall is the metric that would catch it, and it needs labelled ground truth, so it lives in your evaluation harness and not in production. Corpus coverage needs two integers you already have.

The control is the shape of the sentence, not a percentage

This took longest to see, and it is why there is no threshold anywhere in the implementation.

0.598% is not a bad number. It is a bad number for one class of sentence. It is entirely adequate for "he mentions Rust" — you need the one chunk you are quoting and nothing else. It cannot support "he never mentions Rust." Same retrieval, same six chunks, opposite verdicts, because the claim changed.

A single coverage threshold cannot serve both. Set it low and it licenses the second sentence; set it high and it forbids the first.

Claim	Coverage required
"The policy says X"	Any — you need only the units you cite
"There is no policy about X"	Exhaustive
"All policies require X"	Exhaustive
"The most recent policy is…"	Exhaustive — the unread remainder may hold the true maximum
"There are three mentions"	Exhaustive — a count over a sample is an estimate

Every row needing exhaustive coverage has the same reason: it asserts something about the units that were not read. 999,999 of 1,000,000 is 99.9999% and still cannot establish absence — a threshold there would be a lie with a decimal point on it.

Three failures wearing one label

The second thing the missing denominator costs you is diagnosis. When a RAG answer is wrong, the post-mortem usually terminates at the model hallucinated. That sentence hides at least three separate engineering problems:

What actually happened	What to fix
The relevant chunk was never retrieved	The retriever, embedding or query — not the model
It was retrieved, then dropped during context assembly	Context construction — the model never saw it
It reached the model, which reasoned past it	The prompt or the model — the only reasoning failure

Three different people, three different days, one identical symptom. And the distinction is trivially recordable — retrieved, in_context, and whether the answer was right — but almost nobody logs the middle number, so the first two are permanently indistinguishable after the fact.

What I built

An MCP server for notchecked, the coverage-accounting schema I wrote after hitting the silent-skip failure in four domains. The library types the gaps a program leaves. The server does it for an agent, which is where the failure moved.

coverage_retrieval(
  target      = "portfolio corpus",
  query       = "Rust experience",
  corpus_size = 1003,
  retrieved   = 6,
  in_context  = 6,
  claim_type  = "absence",
)

→ scope:      6 of 1003 corpus units (0.598%)
  exhaustive: false
  claim_supported: false

  REFUSE_THIS_CLAIM: An ABSENCE claim asserts something about every unit
  you did NOT read, and 997 were never read. Downgrade the answer to what
  you found, or inspect the whole corpus.

retrieved = 0 is not a thin answer, it is a retrieval failure, and it records as one. Everything retrieved and then dropped is a context failure and records as that instead. Neither is a gap in the agent's reasoning, and calling them one sends someone to debug a prompt for a day.

The bug this found in its own implementation

I wrote a suite that replays six investigations of my own that produced wrong claims — a page judged from 3,000 of its 10,828 words, a search that had stripped the HTML so anything named only in an href was invisible, three different counts from three broken filesystem walks.

On its first run it failed, and it failed on my code rather than on the cases. exhaustive was a bare assertion: an agent could pass exhaustive: true alongside scope: "3,000 of 10,828 words" and the absence warning was dropped. The tool committed the exact failure it exists to prevent, one layer above the schema it protects. Where the scope carries "N of M", the contradiction is machine-visible and is now refused.

That is the third time this idea has caught its own implementation. I have stopped finding it funny.

What this does not do

If your retriever reports corpus_size=100000, retrieved=20, in_context=12, cited=3, this records those numbers and what they can support. It does not know whether the retriever chose the right twenty. It is not a retriever, a vector database, a reranker or a context assembler, and it is not competing with the one you have.

Three limits are recorded as passing tests rather than left out of the README: an exhaustive search of the wrong instrument is still exhaustive; the target list is self-declared, so nothing can know what the caller failed to think of; the retrieval counts are self-reported, with only internal consistency enforced.

What it removes is the silence. That is a smaller claim than "this makes agents honest," and it is the one the evidence supports.

I have measured this in one live system, my own. I have not established what other RAG deployments report, and I am not claiming it from a sample of one.

pip install notchecked · github.com/Mormolykos/notchecked — MIT, zero runtime dependencies, MCP over stdio written from the JSON-RPC wire format.

The eight coverage states were reviewed publicly by Boris Teplitsky, an IBM Certified Expert IT Architect who hit the same shape in infrastructure compliance and gave three corrections that changed the schema. They are frozen.

If you run RAG in production and you do record retrieval coverage alongside answers, I would like to know — that would make this a solved problem I had not found the solution to, which is a better outcome than being right.

I Shipped a Feature in May. Nobody Could Reach It Until Yesterday.

Panagiotis Gkilis — Tue, 25 Aug 2026 03:33:52 +0000

I shipped a feature in May. Cross-page session memory for a retrieval agent — follow a link, the conversation comes with you. It was deployed, tested, and running in production.

Yesterday I found out no visitor had ever been able to reach it.

The part that was actually broken

The obvious problem was that the chat widget existed on exactly one page out of thirty-nine. That is bad, but it is the kind of bad you notice.

The real problem was underneath. The widget never sent a session_id field. Or a scope field. The server had both, documented, tested, deployed — and the browser had simply never been wired to send either one.

So the feature was not "confined to one page." It was never invoked from anywhere. A sessions_active counter that had read 0 since May, on an endpoint that was working perfectly.

Both halves passed their own tests. The server tests posted a session id, because the test author knew the field existed. The widget tests asserted the chat rendered and sent a message. Nothing tested the seam, and the seam was the feature.

What that class of bug looks like

The failure has a shape worth naming: two components, each correct, and an integration nobody owns.

It does not throw. It does not log. The health endpoint is green. In my case the endpoint even reported the exact number that would have given it away — sessions_active: 0 — sitting in a JSON response I read regularly and never questioned, because zero active sessions on a low-traffic site is completely plausible.

That is the tell, and it generalises past my code: a metric whose broken value is indistinguishable from its healthy value tells you nothing. Zero sessions means "nobody is using it" and "it is impossible to use it," and the number is identical.

Proving the fix, rather than asserting it

Once the browser sent a session id, I did not want a green test. I wanted the thing a visitor does.

So: send one question with a session id from the datasets page. Then send a second question from the portfolio page with the client history deliberately empty — which is what a real navigation does to a JavaScript variable — and see whether the answer still knows what we were talking about.

page 1  /datasets/    "What speech datasets are there?"
page 2  /portfolio/   "Who recorded it?"     history: []

The second answer came back about the speech datasets. The server had carried it. That is a test I trust, because the thing I removed — the client's memory — is exactly the thing the feature claims to replace.

The bug I nearly shipped while fixing it

Rolling the widget out, I decided each page should tell the agent which section of the corpus it was on, so the datasets page ranks dataset material first.

Then I tested the datasets page with a piece of internal vocabulary — a codename from our own research notes — and it answered "the documents do not contain information about that." I read that as a coverage failure and widened the page's search to the whole corpus.

That was wrong, and the correction came from the person who owns the product rather than from me. Those codenames are internal research nomenclature for how a training corpus was built. They are not products. Nothing under those names is for sale. They appear zero times on the sales page and only in an engineering report.

The narrow scope was not a limitation. It was the mechanism keeping internal research vocabulary out of a commercial answer. By widening it I had connected a buyer's licensing question to the research corpus, in the name of fixing a defect that was the system working correctly.

Reverted. The lesson is cheap to state and I clearly needed to learn it expensively: do not test a sales surface with internal vocabulary, and do not read a narrow result as a broken one until you know what the narrowness is for.

What I would take from this

Ship a feature and then, before believing it works, ask what a visitor would have to do to reach it — out loud, as a sequence of clicks. Mine had no answer. There was no page to navigate from.

And look at your health metrics for one specific property: is there any value this number could report that means "broken" but reads as "quiet"? Those are the numbers that let a feature sit dead in production for three months while everything stays green.

The agent this is about is live: the architecture and its 32-case evaluation gate are documented at tts.bedvibe.studio/portfolio/agent-architecture, the eval results at tts.bedvibe.studio/portfolio/agent-evals, and the widget itself is on every page of tts.bedvibe.studio now — including, finally, more than one.

If you have found a feature of your own that was deployed and unreachable, I would like to hear how you noticed. In my case it was luck, and I would prefer a method.

My Validation Layer Was Correctly Deleting 16% of My Good Data

Panagiotis Gkilis — Tue, 25 Aug 2026 03:12:31 +0000

Originally published at ai.bedvibe.studio.

I built a real-time tracker in Rust — about two thousand lines — that reads a live ADS-B feed, keeps a Kalman-filtered track per aircraft, and screens every pair for closest approach against separation minima. Roughly 150 aircraft, a full cycle in under a millisecond.

It ran clean. Tests passed, the picture looked right, the numbers were plausible.

It was refusing about one measurement in nine, and the only reason I ever found out is that the rejections went to a counter instead of a log line.

The gate has a sub-second tolerance for clock error

The tracker runs an innovation gate: when a position arrives, the filter predicts where the aircraft should be, and if the measurement is too far from that prediction it is rejected as physically impossible rather than believed.

Once a track converges the innovation standard deviation settles around 36 m, so a five-sigma gate sits at roughly 180 m. An airliner at 250 m/s covers 180 m in 0.7 seconds.

So the gate's entire tolerance for a wrong timestamp is under one second. Any pipeline that mis-times its measurements by more than that will have them rejected — correctly, and invisibly.

The feed reports its own staleness. The pipeline dropped it.

Every ADS-B record carries a field saying how old that position already was when the response was generated. In the original build it was parsed into the contact struct and never read again — the only other place that field appeared in the entire codebase was as 0.0 in test fixtures. Every measurement was therefore stamped with the tracker's own cycle clock, as though it had been observed at the instant it landed.

This is the common case, not an exotic one. A field that is decoded and then unused looks identical to a field that is decoded and used, right up until you go looking for its second reference.

Here is what that field actually contains, sampled across two consecutive polls of the live feed:

reported age of position    median   0.31 s
                            p90      3.97 s
                            max     48.53 s

change per aircraft
between consecutive polls   -15.76 s  to  +3.00 s

re-served identical
positions                   17 of 135 contacts  (12.6%)

It is the jitter, not the lag

The intuitive diagnosis is that the lag itself is the problem. It is not, and the distinction turns out to be the whole thing.

A constant lag is invisible to a constant-velocity filter. If every measurement is uniformly two seconds old, the filter simply tracks a target that is uniformly two seconds behind. The innovations stay small. Nothing is rejected. The picture is late, but it is self-consistent.

That has a sharp consequence for anyone writing a regression test: a fixture built with a constant age will not reproduce the fault no matter how large the age. The test passes and the bug survives. The fixture has to carry the variation, or it is testing nothing.

Look at that middle row again. Between two polls two seconds apart, one aircraft's reported age fell by nearly sixteen seconds — a genuinely new observation arriving after a long gap. Stamped on arrival, both the stale one and the fresh one are marked "now", so the filter sees an aircraft that has apparently teleported. A steadily flying aeroplane appears to lurch, and a correctly functioning gate refuses to believe it.

Why the obvious fix makes it worse

This is the part worth reading, and the reason the change is architectural rather than a one-line patch.

The obvious repair: stamp each measurement with arrival − age, feed that to the filter, and guard against a measurement arriving from before the filter's current state, because you cannot predict backwards.

if observed_time <= track.epoch { reject }

That guard is fatal. Every cycle, the display loop advanced every track to the current wall-clock time so the picture and the collision screen would agree. So by the time a two-second-old measurement arrived, the track's clock already read later than the measurement, and the guard would have discarded it as stale. A patch aimed at an 11% rejection rate would have rejected considerably more — and from the outside it would have looked exactly like the fix working.

The general form: a Kalman filter's state is valid for exactly one instant. Once anything other than a measurement is allowed to advance that instant, every measurement is applied to a state from a different moment than the one it describes. A rendering loop is not usually thought of as mutating the estimator, which is precisely why this survives review.

The pre-fix code was not wrong about this. It was consistently wrong — the display and the measurements were stamped with the same fictional clock, so they agreed with each other. That is why correcting one half in isolation breaks it.

The rule that actually fixes it

One sentence: the filter's validity time advances only when a measurement arrives. Nothing else may move it.

The picture and the conjunction screen do not predict tracks forward — they take a view, which extrapolates a copy and leaves the filter parked at the moment it was last given evidence for. A test asserts that rendering the screen thirty times does not move a single filter.

Two properties follow, and both are load-bearing.

The absolute observation time is never reconstructed at all. The step between two observations is computed entirely from differences — elapsed time between two arrivals the process witnessed, plus the two ages the sensor reported. Nothing subtracts a duration from a monotonic clock, so there is no underflow path. That closes a real crash: Instant - Duration panics in Rust if the result would precede the clock's origin, and on Windows that origin is boot time.

Deduplication becomes free. A re-served snapshot has both its arrival and its reported age advance by the same amount, so it computes a step of exactly zero and is rejected by the same comparison that catches out-of-order data. One comparison covers both cases, which is a good sign the invariant is the right one.

Designing an A/B the upstream cannot distort

Measuring this against a live third-party feed is harder than it looks, and two designs had to be discarded before one held.

Running both binaries simultaneously fails: the corrected build recorded 32 polls and 32 HTTP errors — zero data — while the old one ran fine. That is not a regression, it is the upstream rate-limiting per IP. Running them sequentially fails differently: the results invert, because the API throttles progressively and whichever build runs first gets the fresher quota. Time-to-first-byte on three consecutive manual requests climbed 0.43 s, 0.91 s, 4.22 s.

Both would have produced a clean-looking number and a wrong conclusion. What caught them was running the corrected build alone, which worked — proving the failure was the environment, not the code under test.

The design that survives: alternating order, 45-second windows, 75-second cooldowns, comparing gated plots as a fraction of observations so traffic volume cancels.

window   build       observations   gated   rate
  1      corrected       2,137          3    0.14%
  2      pre-fix         2,752        308   11.19%
  3      corrected       2,208         81    3.67%
  4      pre-fix         2,755        578   20.98%

pooled   pre-fix       886 / 5,507         16.1%
         corrected      84 / 4,345          1.9%

Both corrected windows sit below both pre-fix windows with no overlap. The corrected build varies — 0.14% against 3.67% — and I would not claim a precise point estimate from four windows. The separation is the defensible result, not the number.

Two unrelated faults the counters surfaced

It was silently failing most of its polls. 888 observations at roughly 135 per response is about seven successful polls in sixty seconds, not thirty. Two-thirds of its requests were failing and the system reported nothing, because there were no sensor-health counters at all — a failed poll produces no data, which looks exactly like quiet airspace.

It never aged out a single track. Zero drops in both windows. Of course not: if every measurement is stamped with arrival time, every track's last-seen time is always now, so nothing is ever stale. A track whose aircraft stopped reporting forty seconds ago sat on the display looking as current as everything else.

Those counters were added to report on something else entirely — a limit on response body size. They immediately found a different fault. That is the lesson I would take from this project ahead of the timestamp one: observability added for one problem finds problems you were not looking for, and a system with no failure counters cannot tell you the difference between "nothing is happening" and "nothing is working".

What to check in your own pipeline

If you ingest anything from a source that reports its own staleness — market data, IoT sensors, GPS, log shipping, any polled API with a timestamp in the payload — check whether your pipeline records when the record arrived instead of when it was observed.

The failure mode is nasty specifically because it is quiet. Nothing crashes. No error is logged. Your validation layer does its job perfectly and deletes your good data, and if the rejections are counted anywhere at all you will read the number as evidence that the validation is working.

And a constant lag hides it completely. It is the jitter that bites — which means the systems most likely to have this bug are the ones whose feeds are usually fast.

MIT, 55 tests, clippy clean at -D warnings: github.com/Mormolykos/aether. Surveillance and state estimation only — no targeting, engagement or weapon functionality of any kind. The README carries the full measured results and a limitations section considerably longer than this post.

More measured write-ups of the same kind, including the retrieval work and the long-document RAG ablation, are at tts.bedvibe.studio/portfolio.

Your TTS Model Sounds Great — Until It Says "GPUB"

Panagiotis Gkilis — Tue, 25 Aug 2026 03:10:05 +0000

Originally published at ai.bedvibe.studio.

I built a text-to-speech product and kept getting burned by the same thing. On normal sentences the model sounded great. Then it would hit a number, a date, an acronym or a name, and quietly mangle it.

Worse, the metric everyone reaches for — Word Error Rate — was lying to me in both directions. It flagged perfectly good audio as broken because the script said 3:30 PM and the transcript said "three thirty pee em." And it missed real failures on short tokens, where the speech recogniser is as unreliable as the TTS.

So I wrote the QA framework I wished I had, packaged it as ttsproof, and then ran it as a blind study against a production TTS service so the results would be more than an opinion.

The two failures WER cannot see

A TTS pipeline breaks in two different ways, and a single WER number blurs both.

Structural defects. The clip is empty, truncated, three times too long, stuck in a repeated-chunk loop, clipping, or has a click at the tail. These have nothing to do with pronunciation — you can catch them with no model at all, straight from the waveform.

Pronunciation and content errors on the hard cases: numbers, decimals, dates, clock times, acronyms, single letters, URLs, names.

ttsproof splits them apart and handles each one honestly:

Structural checks, no model needed — empty or truncated audio, duration explosions, long internal silences, clipping, loop detection, end-of-clip artifacts. numpy and soundfile, nothing else.
Equivalence-aware WER/CER — the expected text and the ASR transcript are both canonicalised to spoken form before scoring, so 3:30 PM against "three thirty" stops counting as an error.
ASR-uncertainty quarantine — when the audio is structurally clean but the recogniser disagrees on a very short utterance, the sample is set aside for a human instead of being auto-failed. At that length the ASR is as likely to be wrong as the TTS.

The study: 390 samples, and a blind human check

I evaluated the method against a production neural TTS service — 130 edge cases × 3 voices = 390 samples — and published it as a citable technical report (DOI 10.5281/zenodo.20757553, CC-BY-4.0).

Zero structural audio-integrity defects across all 390 clips. That matters: it means every failure that did exist was pronunciation, exactly the kind WER mislabels.
Exact-match rate 0.769.
Then the honest part — a blind human review of the ASR-uncertain quarantine zone. Of 42 uncertain clips, 23 (55%) were ASR false negatives (the TTS said it right and the recogniser misheard) and 19 (45%) were genuine TTS mispronunciations. Fifteen control clips came back 15/15 correct, so the rater was reliable.

That 45/55 split is the entire argument for having a quarantine verdict at all. Auto-passing that zone ships 19 real mispronunciations. Naive ASR-WER auto-failing it wrongly kills 23 correct clips. Neither is acceptable, so ttsproof refuses to guess there.

What the real failures looked like

All 19 genuine failures were short isolated letters and acronyms, and the pattern is oddly specific:

Failure mode	Examples
A-vowel substitution	`NATO` → "NITO", `USA` → "USI", `CIA` → "CII"
Trailing appended phoneme	`GPU` → "GPUB", `EU` → "EUU"
Early truncation	`R` chopped short
Doubling	`X` said twice
Other substitution	`CEO` → "CEE", `Z` → "SZ"

The structural detectors did not fire on any of these. "GPUB" is intelligible speech, not a click. Structural checks and ASR quarantine are complementary; neither alone catches everything.

Benchmark any engine in one command

Beyond the study, ttsproof ships a corpus of 817 curated edge cases across 39 categories — numbers, currencies, dates, ISO timestamps, phone numbers, URLs, file paths, pronunciation-torture words (Worcestershire, synecdoche), proper names (Reykjavík, Nguyễn), Greek, Norwegian and more. The corpus is versioned independently of the software, so published scores stay comparable across tool updates.

It is engine-agnostic — point it at any TTS via a command template, or at a folder of audio you already generated:

ttsproof benchmark --cmd "mytts --text {text} --wav {out}"

You get a category scoreboard, a self-contained report.html with waveforms, an audio player and what the ASR actually heard, and a CI regression gate. Closed-source engines work too through a SpeechSDK wrapper — an integration a user suggested after the first release.

Try it

pip install ttsproof            # structural checks + metrics + corpus
pip install "ttsproof[asr]"     # + faster-whisper for pronunciation gating

Repo: github.com/Mormolykos/ttsproof (MIT)
The study: doi.org/10.5281/zenodo.20757553

It has already had its first outside contribution, a community fix for a real number-formatting bug, which is exactly what I hoped for. If your TTS breaks on something, open an issue with the case — the corpus grows from real failures.

The text-to-speech platform this came out of is live at tts.bedvibe.studio — ttsproof exists because I needed to QA that, and the model's engineering write-up covers what it is built on.