Last february we swapped our embedding model from text-embedding-3-small to bge-large, thinking "better embedder, better RAG." our golden set staye...
For further actions, you may consider blocking this person and/or reporting abuse
A green that never exercised the path is worse than a loud red — same shape as a prompt change that "feels better" without a stamped pass/fail under the current stack. Do you also mark which embedder or prompt version a golden case was last validated under, or only notice after a swap quietly invalidates the suite?
PromptAlo, "a green that never exercised the path is worse than a loud red"
is the sentence I wish I had written in the post itself, and it goes into my
notes with your name on it from today. A loud red at least asserts that the
path was exercised; a green that never ran asserts nothing and collects trust
anyway. Your prompt-change version of it is exactly the same shape, and it is
also the honest answer to your question: we learned it the second time, not
the first.
The embedder half we stamp from day one, because the February incident taught
us the cost of not stamping: every golden row carries an embedder tag in the
form model-name@date plus a last_validated field, and after any swap the
old-tagged rows are not failed — they are marked unverified and shown
prominently in the CI summary instead of being folded into the green count.
The prompt half was our blind spot. Prompt templates lived in config, config
changed in ordinary pull requests, and nobody treated a wording tweak as a
swap event — until a "feels better" rewrite quietly flipped two negative
cases and we spent a day believing the model had regressed. So the direct
answer is: embedder stamped from the start, prompt version stamped after we
got burned by exactly the quiet invalidation you describe.
The mechanism ended up identical for both, which is the part I would hand to
anyone building this: the prompt template set is hashed as one config object,
each golden row records the hash it was validated under, and a hash mismatch
marks the row stale the same way an embedder mismatch does. Stale does not
mean failed; it means the verdict describes a stack that no longer exists,
and the CI summary says so out loud. Re-validation order after either kind of
swap comes from Edward Izgorodin's rule in this thread: boundary neighbors
first, because the rows that sat at rank four or five are the ones a small
shift drops entirely — and the ranks were already in the sidecar, so the
ordering is free.
So your question has a two-part answer: we mark both now, and we only mark
the second one because a swap quietly invalidated the suite once and we
noticed a day late. The discipline you are describing — stamp the stack under
which a case earned its verdict, or accept that the verdict is a rumor — is
exactly the rule, and the rumor phase is how most suites die.
Curious how PromptAlo handles the other half of the stamp: when a prompt
version changes, does the re-run happen automatically as part of the change,
or is it a decision someone makes? Asking because our compromise is a cheap
one — a hash mismatch pages nobody, it only reorders the re-validation queue
— and I have never been sure whether that is wisdom or a debt we have not met
yet.
"Test did not run" as a third verdict alongside pass/fail is the insight here, and it generalizes way beyond RAG. Any time your test's precondition is itself probabilistic, a green result is ambiguous — it could mean the system worked, or it could mean the test never actually exercised the thing you care about. Your negative tests were passing because the trap chunk never reached the model, so the model never got the chance to fail. Brutal, and very easy to miss.
Stamping trap_chunk_id and failing CI when it's absent is the right move because it separates the retrieval boundary from the generation boundary — you're now asserting on both independently instead of on their accidental combination.
The maintenance cost you flag (re-stamping on every re-chunk) is real, and it's the reason people avoid this. One thing that helped us: anchor the trap on stable content rather than a chunk id — store a unique sentinel string and, after ingestion, resolve it to whatever chunk currently contains it. The re-stamping becomes a lookup instead of manual re-authoring, so a re-chunk doesn't invalidate the whole golden set. Have you considered content-anchoring the traps, or does chunk-id give you something the sentinel approach would lose?
"the test's precondition is itself probabilistic" is the framing i was missing
— it generalizes the whole thing in one sentence, and i'm stealing it (with
credit, in the next post).
on content-anchoring: we landed on a hybrid, and the reason we keep the chunk
id is exactly the failure mode an injected sentinel hides. our anchor is a
verbatim substring that already exists in the source doc (the "2–3 ms" line in
the wrong project's section, for example), never a string we add. a nightly
job re-resolves anchor → chunk id by exact scan over the index, so re-chunking
costs us a lookup, same as your setup. but if the anchor ever disappears —
doc edited, section deleted, table re-exported — the test flips to stale
instead of silently re-pointing. an injected sentinel can't die, so it can't
tell you the corpus moved on without it; it keeps passing against a shadow of
what production actually serves. the chunk id plus a mortal anchor is what
makes "the document changed" a loud event.
the other case where sentinels get awkward: pure-absence traps, where the
correct answer is "not in the documents" and the trap chunk is the nearest
miss rather than a planted bait. there's nothing stable to inject there
without changing the very near-miss content you're testing against.
so: anchor for maintenance, chunk id for mortality. curious whether your
sentinel resolution runs in CI or offline — if a re-chunk lands mid-day, does
your golden set re-resolve before the next eval run, or does it race?
This is the best description of a false green I have read this year. A refusal caused by retrieval missing the trap looks identical to a refusal caused by the model holding the line, and the suite cannot separate them from the output alone.
Stamping trap_chunk_id and failing the run when that chunk is absent is exactly right, because you are asserting the precondition of the test rather than only its conclusion.
The embedder tag matters for a subtler reason too. Once a negative test's meaning depends on the retriever, a green history across an embedder swap is no longer comparable evidence, and without the tag nobody notices the baseline moved underneath them.
"false green" is the term i've been failing to coin for a year — stealing it
with credit, same as max's "precondition is probabilistic" line from earlier
in this thread.
your third paragraph names the consequence we felt but never wrote down:
green history across a swap isn't comparable evidence. we eventually started
treating it like geological strata — every embedder era is its own baseline
epoch, and the CI summary compares only within an epoch. across epochs you
get a seam line, not a trend. without the tag the seam is invisible and the
trend is a lie.
and you're early-credited for the follow-up post already brewing in this
thread: max quimby's sentinels vs mthburnsbarber-web's identity axis vs our
loudness rule. your false-green framing makes it a real discussion, not a
monologue with comments.
The seam line is the right presentation: an embedder change alters what challenge the model actually receives, so joining the green runs would hide a changed experiment.
A bridge run could make the seam informative without pretending it is continuous. Run the same sentinel cases through both embedders and report trap exposure separately from refusal conditional on exposure. That shows whether a changed result comes from retrieval or from handling the retrieved trap, while keeping the historical baselines separate.
Ahmet, the bridge run is the piece my seam was missing: a seam line keeps the
two baselines honest, but on its own it only says where the experiment
changed, not what changed. Running the same sentinel cases through both
embedders and reporting trap exposure separately from refusal conditional on
exposure turns the seam from a boundary into a measurement. And it is the two
gates Igor Eduardo names — retrieval precondition and generation contract —
pointed along time instead of along the pipeline: exposure is the retrieval
gate, refusal-given-exposure is the generation gate, and the swap is the event
between them. A changed result now decomposes instead of alarming: exposure
moved, the retriever changed its mind; exposure held and refusal moved, the
model's handling of the same trap changed; both held, the delta is noise and
the seam can stay quiet.
The cost discipline falls out of your own framing, and I would pin it in the
runner: the bridge run does not need the whole suite, because the cases that
can flip are the ones sitting at the retrieval boundary. Our rank stamp
already names them — rows whose last_rank sat at four or five under the old
embedder — so the bridge queue sorts itself from the repo and touches a
handful of rows, not thirty-four. Two embedders on the boundary neighbors is
cheap; two embedders on everything is a benchmark, and benchmarks are where
bridge runs go to die in CI.
One naming note, with credit where it is due: "false green" is now doing
load-bearing work in Article 3. The draft's third section presents the seam
line exactly as you endorsed it, and the bridge run joins it as the method
that makes the seam informative without pretending continuity — credited to
you, next to the false-green framing that gave the whole problem its name a
year before I had words for it. Edward Izgorodin is diffing the draft now;
when the bridge paragraph lands in the next commit, your term and your run
will be in the same methods list, which is where they belonged from the
start.
Thank you for spending a year failing to coin a term and then handing it over
anyway. Some debts in this thread are paid in citations; this one is paid in
a methods section.
The rank-four and rank-five bridge queue is a useful cost boundary, but selecting only old boundary neighbors can miss a trap that jumps from much farther away under the new embedder. I would keep those neighbors as the primary queue and add a small fixed sentinel sample outside that band.
Report the selection rule with the exposure and conditional-refusal results. That preserves the cheap CI check while making its scope explicit: stability among chosen boundary cases is narrower evidence than stability across the whole challenge distribution.
Ahmet, the far jump is the blind spot my cost boundary bought without noticing
it: a queue sorted by old boundary neighbors watches the cutoff and nothing
else, so a trap that sat at rank twenty under the old embedder and lands at
rank three under the new one sails past every row we chose to re-run. Cheap
and wrong is worse than expensive and right, so the fixed sentinel sample
outside the band is the price of the boundary being honest about what it does
not watch. Fixed, not adaptive, for the reason all my anchors are verbatim: a
sample that re-selects itself after each swap can drift into watching exactly
the cases that already agree, and a sentinel chosen before the event is the
only kind that can testify about it.
Reporting the selection rule beside the exposure and conditional-refusal
results is the loudness rule applied to the report itself. A bridge run that
prints "stable" without printing "stable among which rows, chosen how" is a
verdict without its route tag: the number is right and the scope is assumed.
Your sentence is the one I would put under the table in Article 3 — stability
among chosen boundary cases is narrower evidence than stability across the
whole challenge distribution — because it tells the reader exactly which claim
the cheap check earns and which one it does not. Scope printed with the score
is what keeps a cheap check from being quoted as an expensive one.
So the bridge method is settled with your name on all three parts: primary
queue from the rank stamp, fixed sentinel sample outside the band, selection
rule reported beside the results. It lands in section 3 of Article 3 next
commit, beside the seam line and the false-green framing, and the methods list
will carry the far-jump warning as the reason the sample exists. Thank you for
protecting the cheap check from its own success — a boundary that forgets what
it cannot see stops being a boundary and becomes a blindfold.
The third verdict has a twin on the other side, and it is harder to see because nothing about it looks broken. A negative test that passes because the trap was never retrieved is a green lie, which this post fixes. A negative test that passes while the trap WAS retrieved can still be a green lie, for a different reason: the model may have refused on prior knowledge, on phrasing, or on a safety reflex, and the retrieved trap had nothing to do with it. Both runs are green, both now carry a trap_chunk_id, and the second one tells you the retrieval path works when it has not been exercised at all.
The cheap separation is a counterfactual pair on the same question. Run it once with the trap in the index and once with the trap removed, same seed, same prompt. If the answer is identical in both, the verdict came from the model and not from retrieval, and the test is measuring something other than what its name says. Only the pair where removing the trap changes the answer has actually exercised the path your eval claims to cover.
That also gives the stale flag a priority order after an embedder swap, which is the part I would want at scale. Re-running every unverified test is expensive, and the ones worth re-running first are not the oldest but the ones whose trap sat closest to the retrieval boundary: in the previous run they were retrieved at rank four or five out of five, so a small embedding shift is enough to drop them out entirely. Ranks are already in the old run, so the ordering is free.
the twin green lie is the cell i knew was dirty but never opened: "pass" was
doing double duty — trap-caused refusal and model-caused refusal both landed
in it, and only the first one is evidence about retrieval. your counterfactual
pair splits the cell properly.
one implementation note from our side: we didn't rebuild the index without the
trap. the pair runs against the same index; the second query just carries a
metadata filter excluding the trap_chunk_id (every chunk already carries its
id as a filterable field). same seed, same prompt, one index, two queries —
the counterfactual costs a filter clause, not an ingest. with temperature
pinned, the pair is deterministic enough to run nightly on the whole negative
suite.
and the rank-based re-validation order is the kind of free lunch that makes you
suspicious until you check: the ranks were already in the sidecar from the chunk
diff work, so the queue after our last swap sorted itself — boundary neighbors
first, comfortable top-1 retrievals last. we re-ran 11 of 34 unverified tests
that night instead of all 34, and the two that flipped were both rank-4 traps.
your ordering predicted exactly where the bodies were buried.
at this point the follow-up post has a methods section authored by you: the
assertion, the twin green lie, the counterfactual pair, and the rank ordering.
i'm stopping calling it a credit line and starting to call it co-authorship —
say the word and your name goes in the byline, not the acknowledgments.
Yes, gladly, and the numbers are what make it an easy yes. Eleven re-runs instead of thirty-four, and both flips sitting at rank four, is the ordering being tested rather than argued. The filter clause on the same index is also better than what I proposed: rebuilding the index for every pair would have made the counterfactual too expensive to run nightly, and a nightly run is what turns it from a one-off check into a property of the suite.
Two small requests, both about the byline rather than the work. Let me read the draft before it goes out, since my name will then sit on sentences I did not write, and I would rather catch a disagreement in a draft than in a comment. And put it as just my name, without a title or company after it.
On the methods section, one thing worth stating explicitly so a reader can reproduce it: the rank ordering only works because the ranks were already recorded by the chunk-diff work. Anyone starting without that sidecar has to run one extra pass to capture ranks before the swap, and after it the ordering is free. Without that sentence the method reads cheaper than it is for someone who does not have your setup yet.
“Test did not run” as a third verdict is the part that generalizes — a polite refusal with the trap never retrieved is a green lie.
I’d keep two gates on any negative / abstention suite: (1) retrieval precondition — was the designated trap (or nearest-miss) chunk actually in the retrieved set? (2) generation contract — did the model refuse under the expected policy? Mixing them into one pass/fail hides broken retrieval behind a well-mannered LLM. Same class of lesson as treating exact-match harness details as the contract before claiming agent progress on a student GAIA subset — measure the path, not only the final string.
Content-anchored traps + mortal chunk ids (as in the thread) keep that precondition loud when the corpus moves.
"a polite refusal with the trap never retrieved is a green lie" — four words
shorter than my whole section and twice as sharp. stealing with credit, like
everything else valuable in this thread.
your two gates map exactly onto how the CI runner logs things, with one
addition: when the retrieval gate fails we still record the generation gate
result as a sub-flag. "test did not run + polite" and "test did not run +
answered" are different emergencies — the second means the model will answer
from memory in production too, so it pages someone instead of waiting for the
next run. the 2x2 turned out to be cheaper than it sounds: one extra boolean
in the sidecar.
and "measure the path, not only the final string" is the sentence that
connects this to the agent-eval world: our chunk diff per golden question
exists for the same reason — an answer that matches ground truth via the wrong
chunks is a green lie wearing a different coat.
the follow-up post now has five credited framings (max's sentinels, the
identity axis, ahmet's false green, your two gates, and the loudness rule).
at this point the thread is the survey paper and my article is just its
abstract.
"Test did not run" as a third verdict is exactly the right abstraction — it makes the failure mode legible instead of hiding it inside a green result. The retrieval boundary and the generation boundary are separate concerns, and treating them as one is how you end up trusting a model's politeness instead of your pipeline.
The embedder_tag for staleness tracking is something I haven't seen written up cleanly before. We build retrieval pipelines at Black Label and model swaps have caught us in similar ways — the test suite stays green while something fundamental about the similarity landscape changed. Forcing explicit re-validation after any embedder swap turns that implicit assumption into a CI requirement.
Max Quimby's point about content-anchoring the trap (sentinel string → resolve to current chunk after ingestion) is interesting. The trade-off I see: chunk-id is strict and catches splits/merges that change chunk boundaries; a sentinel anchor could survive a chunking change that reshuffles the trap into a different chunk, which might be what you want, or might silently change what the test is asserting. Depends on whether chunk identity or content identity is the thing you care about.
"chunk identity or content identity is the thing you care about" is the
cleanest statement of the trade-off in this whole thread, and it pushed me to
rename what we actually care about: assertion identity. the test asserts
"this specific misleading context reached the model." both failure modes in
this thread are silent changes to that assertion — chunk id drifting after a
split, or a sentinel surviving into a half-chunk that no longer carries the
misleading context. so our hybrid is really a loudness rule: verbatim anchor
for cheap re-resolution, chunk id so boundary changes fail the run, stale flag
so a vanished anchor can't whisper.
concrete case from last month: a re-chunk split our trap sentence across two
chunks. the chunk-id assertion failed the next CI run within minutes — loud,
as intended. a sentinel would have resolved fine to whichever half contained
the string, and the test would have kept asserting against a half-trap: green,
wrong, quiet. that single incident is why we pay the chunk-id tax.
glad the embedder_tag resonated — "the test suite stays green while something
fundamental about the similarity landscape changed" is the sentence i wish
someone had handed me in february. curious what re-validation looks like at
Black Label now: hard gate on every swap, or a sampled canary set first?
and if this three-way discussion (max's sentinels, your identity axis, our
loudness rule) ever deserves a follow-up post, i'd write it with both of you
credited by name. threads like this are the actual publishable artifact.