Part one ended at a plateau: 34 of 46 cases passing at the honest configuration, recall@3 thirteen points above recall@1, and a hypothesis. The right page was almost always retrieved and simply was not first, so the problem had to live wherever "first" gets decided, and in a hybrid engine that place is rank fusion. RRF's conventional k = 60 is calibrated for TREC-scale runs fusing thousand-deep lists. On a 19-document corpus the reciprocal-rank curve is nearly flat (rank 1 versus rank 2 is 1/61 versus 1/62, a gap of 0.0003), so fusion degenerates toward "best average rank wins" and a decisive first place in one engine cannot survive mediocrity in the other. Nobody calibrates k. It looked like the constant nobody had checked.
I built a sweep to test it. rrf_k threaded through ScoreOpts, seeded from rrf::K so there is exactly one copy of the constant. --sweep-rrf-k and --sweep-rrf-alpha on eval, running all 46 cases per grid point. And a self-validation requirement: the k=60 / alpha=0 cell must reproduce the baseline byte for byte, or the harness itself is suspect. It did.
Here is what the grid did to the hypothesis:
| recall@1 | alpha 0 | alpha 0.5 | alpha 1 | alpha 2 |
|---|---|---|---|---|
| k = 2 | 74% | 74% | 72% | 67% |
| k = 4 | 74% | 72% | 72% | 67% |
| k = 8 | 74% | 74% | 72% | 72% |
| k = 16 | 74% | 74% | 74% | 72% |
| k = 32 | 74% | 74% | 76% | 74% |
| k = 60 | 74% | 74% | 76% | 74% |
(This table is re-measured on the 46-case union at 42593bf: all twenty-four cells reproduced exactly, including the flat first column.)
Read the first column. 74% at every k from 2 to 60. Fusion resolution is not where recall@1 lives on this corpus, and the hypothesis died in one table. The k that nobody calibrates turns out not to need calibrating here, and the sweep built to confirm the obvious instead falsified it in a single run.
Two smaller findings came along. The alpha=2 corner at low k drops to 67%: confidence-weighted fusion on a sharp curve actively amplifies keyword mistakes, which is the knob's danger zone documented empirically rather than reasoned about. And the grid's best cell, k=32 / alpha=1 at 76%, is exactly one case above baseline, which on a 46-case set I dismissed as argmax noise. Argmax over a 24-cell grid will always find a cell one case up by chance. That dismissal gets revisited below, and the revisit is the most instructive mistake of the arc.
A negative result recorded with its grid is a deliverable. This sweep produced no tuning change and was still the most valuable run of its week, because of what it printed next.
What the sweep actually bought
The per-kind table redrew the map of where the problem was:
| kind | n | recall@1 | recall@3 |
|---|---|---|---|
| exact | 14 | 86% | 100% |
| navigational | 9 | 100% | 100% |
| negative | 4 | 50% | 50% |
| paraphrase | 19 | 58% | 79% |
(Measured at 42593bf, no flags, summed over the gate and known-failures files: 12/14, 9/9, 2/4, 11/19, total 34/46, matching the baseline exactly. One instrument trap worth recording: the sweep re-runs its best cell verbosely after the grid, so a per-kind table trailing sweep output belongs to the best cell, not the baseline. An earlier draft of this table was the k=32/alpha=1 cell for exactly that reason.)
Navigational is solved outright and exact is close behind. That is BM25F doing the job it was built for in part one: the "getting started"-class failures are gone, and both kinds sit at 100% by recall@3. The negatives are leaking at exactly 50%, meaning the relevance floor lets junk through on half the off-topic queries. And paraphrase at 58% is the ceiling, with four of its nineteen misses absent from the top 3 entirely, which is retrieval, not ordering, and no fusion or gating knob reaches it.
The deficit had names, and none of them was fusion.
The floor versus the gate
The two leaking negatives ("toddler bedtime routine", "bicycle chain lubricant") were semantic cosines sneaking over the derived relevance floor, 0.28 at 128 dims. The obvious fix: raise the floor. min_cos 0.34 cleared both.
The per-case diff is why that fix was rejected. It read +2/-1, and the casualty was "make the download smaller for visitors", a legitimate paraphrase whose expected winner's best chunk sat at cosine 0.313, inside the same 0.28-to-0.34 band as the noise. The raised floor deleted the right answer's evidence and the fused order collapsed onto the hub page. A second fingerprint confirmed the mechanism: "elasticlunr" flipped from hybrid to keyword-only, a legitimate exact query whose entire semantic corroboration fell below the raised floor.
On this corpus at this dimensionality, the 0.28-to-0.34 band contains noise and signal simultaneously. No absolute per-document threshold separates two populations occupying the same interval. That sentence is the whole reason the next mechanism exists.
The corroboration gate, built in part one and shipped disarmed, is the distributional version of the same judgment. Instead of asking "is this cosine high," it asks "does anything stand out from this query's own field," measured as top-minus-median over raw cosines, applied only when the keyword engine contributed nothing. Off-topic queries on a homogeneous corpus produce a flat pack; real paraphrases produce one document above the field.
The calibration is the part worth stealing. Part one shipped the gate with diagnostics: --explain prints the gap and top values per query. Toddler's measured gap was 0.059. The sweep missed it at min_gap 0.04 and caught it at 0.06, and the per-case table was identical to baseline at every value from 0.06 through 0.12 except for the one intended flip. Instrument and knob agreeing to the third decimal. When explain and eval move together like that, calibration is a lookup, not a search, and that is the entire argument for shipping a disarmed mechanism with armed diagnostics. The value chosen was 0.08, roughly 35% above the measured noise gap. The strong_cos escape hatch stayed off: nothing on this corpus needed exempting, and a hatch that ships armed for a problem the corpus does not present is a knob without a constituency.
The gate is casualty-free by construction on the floor's failure mode: the min_cos casualty had keyword confidence 1.00, and the gate never fires on corroborated queries. The two mechanisms judge disjoint populations, which is why one of them could be armed and the other could not.
And the second negative survived every gap value, which is the honest counterexample the section needs. "Bicycle chain lubricant" is a structurally different noise mode: one document ringing at a nonsense query, standing out from the pack, which a flatness statistic reads as signal by definition. Any gap value that caught it would also suppress every legitimate semantic-only query whose winner stands out. A statistic designed to detect flat noise cannot detect pointed noise. Disposition: a known-failures entry under a new cause label, spurious resonance, with the blindness documented as inherent rather than pending. Not every failure gets fixed. Some get confessions.
Alpha's redemption
One explain output changed a verdict I had already published to myself.
The data-chops-open failure was the case rrf_alpha was designed against, in its purest observed form. The keyword engine had the compound term right: the correct page at kw#1, score 7.135, on df=2 evidence, about as unambiguous as keyword evidence gets. The semantic engine crowned an honestly adjacent page. And unweighted RRF's arithmetic preferred kw#3 + sem#1 over kw#1 + sem#4, because on a flat curve mediocrity in both engines beats excellence in one. Doing the weighted arithmetic by hand from the explain columns showed alpha=1 flipping it:
2/61 + 1/64 = 0.04841 > 2/63 + 1/61 = 0.04814
That hand calculation forced the revision. The grid's k=60 / alpha=1 cell had shown one case above baseline and been dismissed as argmax noise. It was almost certainly this case. The corrected principle, and both directions of it fired inside one arc: one flipped case without a mechanism is noise; one flipped case with an explain-verified mechanism is a finding. The k=32 cell stays dismissed. The alpha=1 cell gets rehabilitated the moment its case has arithmetic behind it. I applied the noise heuristic one round too early, and the only reason the mistake got caught is that the diagnostic loop kept running after the verdict.
The combined run, min_gap 0.08 plus alpha 1, was predicted at two up, zero down. It came back three up, one down. The bonus was "what should i commit and what should i ignore" flipping to PASS on the expected page, and its known-failures comment, written 2026-08-08 in the same commit that re-baselined the gate (2205069), read: "Verify with chops-search query when weighted RRF lands: if the keyword list already had manage-the-model first, this comes along for free." It did, and it did. A dated prediction in a fixture comment paying off is the strongest single argument in the file for writing diagnoses down instead of just fixing things.
The casualty was "cli", a navigational regression test, and its explain was the best single artifact of the arc. The page titled CLI held kw#3. Ahead of it at kw#1, scoring 4.324, sat a page containing zero occurrences of the word "cli", carried entirely by summed damped prefix expansions: "client", "client-side". And the fused-order arithmetic showed the case had been passing at baseline by 0.032266 to 0.032258.
A margin of eight millionths. Alpha did not break a solid case. It revealed a case that had been passing by luck, balanced on a coin edge that any keyword up-weight would tip. A regression test passing by 8e-6 is not a passing test; it is a coin that has not been flipped yet. The fix recorded the margin so nobody trusts that case as load-bearing again, and the per-case diff discipline is the only reason the coin was ever noticed before it landed wrong in production.
Two keyword fixes, both defensible with the fixture deleted
The "cli" explain exposed a principle-level bug, not a tuning problem. Completions of a half-typed word are competing hypotheses about what the user means, not corroborating evidence. A document matching several completions of "cli" has not multiplied its evidence for anything. The fix scores the trailing word's hypothesis set, the exact match plus its expansions, as a per-document maximum instead of a sum, while distinct typed words still sum as independent evidence. Confidence deliberately kept summing: it answers "does the keyword side have any evidence at all" rather than "how much does one document have", and its floor was calibrated against that arithmetic.
Counterfactual analysis before building showed best-of alone would not rescue "cli": with the phantom kw#1 demoted, the install page's genuine description-plus-body evidence still edged the CLI page's lone title hit. The enabler was the second fix, a title-cover tier, an idea from the first brainstorm of the whole arc that finally found its case. Documents whose titles contain every typed query word rank ahead of documents whose titles do not, BM25F order within each tier. Ordinal on purpose, so it composes with rank fusion untouched instead of fighting the saturation curve, and the trailing word can be covered through an expansion, so the tier works mid-keystroke.
The tier carries one deliberate semantic inversion, pinned by a named test rather than slipped in: for covered queries, title coverage outranks the tag weight. w_tag encodes topical authority; coverage encodes navigational intent; the navigational claim wins across tiers and the weights decide within. The old "tag outranks title" test was split into its two claims, score arithmetic (unchanged, still pinned) and rank order (now inverted for covered queries, pinned as deliberate). And the safety property got its own test: an all-false cover reproduces the plain ranking byte for byte, so queries the tier does not touch score exactly as they did before.
Predictions and outcomes, because pre-registering them is the house style: the tier and best-of were predicted to reach 35 with no outcome changes from the tier itself, and 37 with alpha armed. Actual: 36 and 37. The overshoot was best-of alone flipping "rebuild automatically when i publish a post" to PASS, unpredicted, and alpha then taking it back, weighted fusion re-amplifying the sense-collision evidence ("content rebuild" in the cache-invalidation sense) that best-of had demoted. Alpha's honest ledger on the merged set: +2 / -1, net +1, with the casualty being a case whose own diagnosis already prescribes a content fix no fusion weight can substitute for. Armed anyway, with the taketh-away recorded in the case's notes.
The fixture debt comes due
Running the gate file at the honest configuration for the first time showed two cases down from the recorded 37/37, and the immediate read was "two regressions". The audit said otherwise.
The 37/37 had been measured at --min-cos 0.34, the hand-raised floor. The file's own header had warned that "any --fail-under here gates a configuration nobody actually runs", and the negatives section had predicted that these exact two cases were the most likely to flip when the gate started measuring what ships. Both had been failing at every honest configuration all along. Their green was measurement debt, and the honest floor collected it on the predicted schedule. The gate file's true history, restated without the debt: 33/37 at this chapter's start, +2 from the calibration and keyword work, two never-real passes now visible as the failures they always were.
A fixture file that names, in advance, which of its own passes are fake is doing science. The predictions coming due is what the apparatus was for.
The litigation round that preceded this settled the open fixture verdicts by reading the actual corpus, and two pre-reading guesses reversed on contact with the pages, which is the argument for the procedure over judgment-from-the-diff. "model2vec-rs" was not widened: the crate name appears in exactly one place in the corpus, the returned winner never mentions it, and blessing it would be blessing a noise winner. "data-chops-open" turned out not to be a fixture problem at all but the fusion case above. "which files land in the deployed site" was widened to include the configuration page, weakly held and recorded as such. The verdict rule that did the work: would you add this URL to expect if the case were currently passing? Corpus edits made solely to flip a test stayed off the table, with the one carve-out the known-failures file itself blesses: docs fixes that make the docs genuinely better count as documentation work that happens to be measured.
The ledger, and the cell I refused
The configuration ladder on the merged 46-case set, all at the derived floor: baseline 34, gate armed 35, keyword fixes 36, alpha armed 37. Baseline and gate pin to 42593bf, the keyword fixes and alpha to c9905d8, and all four rungs re-measure exactly. 80% recall@1, up from 74% at the chapter's start, with every point traceable to a mechanism: a gate, a tier, a scoring correction, or the measurement getting honest. Zero points came from resweeping weights. The tuning surface was flat; the recall was hiding in the fusion arithmetic, the expansion semantics, the fixtures, and the prose.
On the gate file: 35/37 at the shipping candidate configuration, and its two failures were exactly the two standing verdicts, which landed: bicycle demoted to the known-failures file on 2026-08-19 (0731290), where its confession lives, and "which files" widened to include the configuration page on 2026-08-21 (46c27c5), weakly held and recorded as such. At the chapter's shipping configuration against the settled fixture, the gate reads 36/36: the c9905d8 engine, --min-gap 0.08 --rrf-alpha 1, fixture as of 46c27c5, 100% at recall@1 and recall@3.
The engine that started this series at 65% now passes its whole gate, and every case it cannot win lives in the known-failures file with a signed confession.
And one number I turned down. The configuration matrix contained a tempting 97% cell: everything above plus --min-cos 0.34, 36/37 at c9905d8, and it re-measures exactly, casualty included: at that floor, "make the download smaller for visitors" slips to third place in the file the gate cannot read. Rejected, because its casualty lives in the known-failures file, so the gate file cannot see the collateral. Shipping a configuration whose damage is hidden by the file split would be optimizing the scoreboard against the honesty the two-file design exists to protect. The split earns its keep in both directions: it keeps known failures from blocking CI, and it keeps CI from laundering them.
The asterisk, and what closed it
Everything above was measured through CLI flags. The browser constructs its scoring from index.bin, which at that point carried the field weights and nothing else, so live search was still running the baseline column: ungated, unweighted, junk on toddler-class queries. The CI gate was about to certify a configuration nobody's visitor ran, which is the measured-at-0.34 debt in a new denomination.
The persistence batch closed it. min_gap, rrf_alpha, and the min_cos override became chops-search.toml keys, ride into index.bin next to the field weights under one version bump, and get read out by the engine at construction. The config file states the scoring, the build bakes it, the browser runs it, and a bare eval measures it: one configuration in four places, none of them shell history. The boundary rule that came with it: persist what you calibrated, default what you did not. rrf_k stays compiled because its sweep was flat; strong_cos stays off because no case has ever needed the hatch. A knob earns a config key the day a sweep produces a value worth shipping, and on that day it also earns an artifact field, because a config key that does not reach the browser recreates the exact gap the batch closed.
The final verification is the one this whole series was pointed at: type "toddler bedtime routine" into the live search box and watch it return nothing.
Landing the batch meant rewriting two files, and the review of that rewrite found seven things the new code had quietly stopped doing, none of which any passing test could see. That is its own story, and it is the next post.
One prediction, pre-registered here so it can fail in public: a comprehensive multi-knob sweep is planned, and the expected result is flat, everywhere, except possibly chunk_penalty, which has never been swept systematically and is the one knob a future chunk_chars change could wake up. The coupled grids (alpha crossed with k, min_gap crossed with strong_cos) should reproduce the flat-column result. If that prediction fails, the failure is the finding, and it gets the same write-up a win would.
The prediction has since been tested, and recording the outcome here keeps the chronology honest, because the paragraph above was written first. A calibration round on 2026-08-14 swept every knob against this gate and woke exactly the knob this post's closing prediction named: chunk_penalty, pinned at 0.12 the next day (534e1dd), the same commit that retired the hand-raised floor for good. The sweep regenerates from the repo alone, corpus, fixture, and harness all being committed, and its verdict re-litigates in one command: at the post-verdict fixture, the pin's per-case ledger against the old 0.02 is +1/-0. It wins Content-Range and costs nothing. And two weeks of documentation growth then did what documentation growth does on a dogfooded corpus: today's bare eval at HEAD reads 33/36, three failures whose winners are the freshly written measurement pages out-competing the fixture's expectations, none of them the penalty's fault by the same per-case diff. The gate caught the corpus moving. That is what it is for.
The engine, the eval harness, both fixture files, and every sweep in this post are in the repo: github.com/gitbadger-clan/chops-search. Part one, the evidence-changing work that got the corpus to the plateau this post started from, is here.
Top comments (0)