In FlagEmbedding 1.4.0, from FlagEmbedding import FlagModel is an alias for from .base import BaseEmbedder as FlagModel — the encoder-only base class, whose default pooling_method is "cls" and whose pooling() for that method is one line: return last_hidden_state[:, 0]. BAAI/bge-code-v1 is not an encoder. It is a causal Qwen2 decoder, published with 1_Pooling/config.json declaring pooling_mode_lasttoken: true and pooling_mode_cls_token: false. Position 0 of a causal decoder cannot attend forward. So the CLS vector bge-code was computing depended on exactly one token and nothing else that followed it.
I measured what that means with a randomly initialized Qwen2Model pooled by FlagEmbedding's real pooling(): two sequences identical at token 0 and different in every token after it came back with cosine similarity 1.0 and max absolute difference 0.0 — bitwise identical. The same hidden states through the real last_token_pool gave cosine 0.10; through mean, 0.65. Across 200 resamples of every token after position 0, position 0's output never moved once. And because encode_queries prefixes every query with the same instruction, token 0 is that shared prefix for every query trelix ever sends — meaning every query embedding bge-code produced was identical to every other one, full stop. Five distinct code chunks sharing a first token collapsed into a single vector, against five distinct vectors from last_token_pool on the same input.
This shipped in 28 tagged releases — every tag from v2.0.0 through v3.1.6 contains the commit that introduced the embedder — and stayed green through every one of them because tests/unit/test_embedder_bge.py replaces FlagModel with a MagicMock. A MagicMock answers to any attribute you ask of it. It never once called the real pooling(), so a defect that destroys every query embedding a provider produces never had a chance to fail a test.
v3.1.6 fixed a real, adjacent bug: BGECodeEmbedder.dimension called a method FlagEmbedding has never had, raising AttributeError before an index could even start, and the dimension it fell back to (768) was wrong anyway — the model emits 1536. That fix was correct and it shipped correctly. But removing the raise made a path reachable that had never been reachable before: the pooling defect. The v3.1.6 changelog entry read as though bge-code now worked. It didn't, and one release later, v3.1.7 said so in public, in the same file: "Retracted: bge-code does not 'work now'." I don't read that as an embarrassment. A team that ships a wrong claim and then corrects it in the next release, with the receipts, is doing exactly what a correctness process is for. The alternative — quietly letting the framing stand — is the actual failure mode.
Eight releases, one arc
This is the story of eight tagged releases — v3.1.2, v3.1.3, v3.1.4, v3.1.5, v3.1.6, v3.1.7, v3.2.0, and v3.2.1 — spanning 190 commits and 324 changed files since v3.1.1, where the last article in this series left off. v3.2.1 is the current shipping release: tagged, dated 2026-08-26, live on PyPI, installable today with pip install trelix==3.2.1. The unit suite sits at 4,376 tests collected on this exact checkout. trelix itself is about nine weeks old — the first commit landed 2026-06-25 — which matters for how to read what follows: this isn't a decade of debt being paid down, it's a project auditing itself hard and often, early, while the cost of finding a defect is still one release instead of ten.
Unlike the v3.0.0 span this series already covered — audit trail, OIDC SSO, the VS Code extension, context compression, model-aware budgeting, extended thinking — this span has no comparable feature drop. It is, almost entirely, a correctness and audit arc. I'm not going to apologize for that framing or bury it under a feature list that doesn't fit the material. The interesting thing about these eight releases isn't what got built; it's what got caught, and specifically the shape of what got caught, which repeats often enough across independent parts of the codebase that it deserves to be named as a pattern rather than four unrelated bug reports.
The pattern: a green suite that never touched the defect
Here is the pattern, stated once so the four instances below don't have to keep re-deriving it: a test can pass without ever exercising the behavior it claims to cover. That's not the same claim as "the tests were bad" or "coverage was low." These tests ran. They asserted things. They turned green in CI, release after release. And in every one of the four cases below, the reason they turned green is diagnosable and specific — not vague test debt, but one of four concrete mechanisms.
bge-code's pooling defect survived because a MagicMock answers any attribute asked of it, so no test ever called the real pooling() method that carried the bug. The sparse-vector padding defect, below, survived because every existing fake for the tokenizer returned an all-ones attention mask by construction, which makes masked and unmasked aggregation produce identical numbers regardless of whether the masking code is even present. The intent_hint dispatch bug survived because a unit test asserted the observed, buggy output as the correct value — it had to be deleted, not fixed, once the real behavior was understood. And the federated search-all deduplication bug survived because the existing test happened to use the one input distribution — globally unique row identifiers — that cannot trigger a collision, out of all the distributions that could.
Four independent parts of the codebase — an embedder, a different embedder's aggregation math, a retrieval planner, and a federation layer — produced the same failure mode by four different routes. That repetition is why v3.2.1 exists, and why I'm leading with it rather than a version-by-version changelog walk.
The sparse leg: clean queries scored against contaminated documents
SparseEmbedder.embed tokenizes a batch with padding=True, which pads every sequence in that batch out to the length of its longest member, and then aggregates with torch.log(1 + torch.relu(logits)).max(dim=1)) — a max taken over every sequence position, padding included. A masked-language-model head predicts a real, nonzero logit distribution at pad positions too; it doesn't know they're padding. Nothing in the aggregation multiplied by the attention mask to zero those positions back out. grep -c attention_mask src/trelix/embedder/sparse.py returned 0. A tree-wide grep -rln attention_mask src/trelix/ matched no file in the codebase at all.
I measured it against the real naver/splade-v3-distilbert at top_k=128. A 28-token chunk embedded alone, compared against the identical chunk embedded in a batch alongside a 185-token chunk, gained 28 phantom terms and lost 28 real ones at the cutoff — 22% of the stored vector — with 0.478 max weight drift on the terms that survived both runs. The phantom terms are ordinary English words the chunk never contained: where, store, numbers, text, gene, sequence, phrase, messages. The control that pins the cause is the one that matters most here: the same chunk batched with an equal-length chunk, which needs no padding at all, came back bit-identical to the chunk embedded alone. Batching was never the problem. Padding was.
Two things made this worth a minor release rather than a footnote. sparse_embeddings is a persisted table in store/db.py, so the corruption wasn't transient — reindexing the same repository in a different file order, or with a different TRELIX_SPARSE_BATCH_SIZE, rewrote every row with a different, equally wrong answer, and no index was reproducible against itself. And embed_query routes through embed([text]) — a batch of exactly one, which needs no padding — so every query trelix ever issued against the sparse leg was clean, scored against documents that weren't. No amount of query-side tuning could have surfaced or corrected that asymmetry, because the query side was never where the bug lived.
The existing test_sparse_embedder.py had ten tests covering this embedder, and every one of its fakes returned attention_mask=torch.ones(...). An all-ones mask makes the masked and the unmasked aggregation mathematically identical, so all ten tests passed whether or not the masking code existed at all — which it didn't. The replacement, test_sparse_padding_contamination.py, is nine tests, every one mutation-verified: deliberately dropping the mask multiplication from the aggregation fails five of them while leaving the equal-length control passing, which is exactly the signature that localizes the cause to padding rather than to batching in general.
intent_hint: a test that asserted the bug as the spec
intent_hint is an optional parameter on both the REST /search endpoint and the MCP search_code tool, meant to let a caller skip trelix's own LLM-based intent classification and specify a retrieval strategy directly. The hint builder had a routing bug: it stamped the "direct answer" tier onto all eight recognized intent values while separately setting the correct leg strategy for each — but the executor checks the tier first, so the strategy it computed was never read. Every intent_hint value, regardless of which of the eight intents you actually asked for, took the same direct-answer shortcut.
I measured this against trelix's own codebase: all eight intent values returned byte-identical output — 40 README sections, zero code files, and no overlap at all with the actual correct result set for any of them. intent_hint had been silently disabling retrieval since v2.10.0, over both protocols, for anyone who used it.
The reason it took this long to catch is the plainest of the four: a unit test existed that asserted the broken output was the expected one. That's not a gap in coverage — it's a test that actively encodes the bug as the specification. It had to be deleted and replaced with assertions on the actual retrieval outcome, not patched.
search-all: a test that used the one distribution that can't fail
Federated search-all fans a query out across every registered repository and merges the results. The merge deduplicated on each result's per-database autoincrement row id — and every repository's ids start at 1. Two repositories' top hits collide on id 1 unavoidably; first-seen wins the merge, and "first" was whichever repository's thread finished first in the pool, which is nondeterministic. The consequence: an entire repository's results could vanish from a federated search, silently, while the response reported repos_skipped: 0 — nothing told you a repository had been dropped, because nothing had been skipped; its results had simply lost a dedup race.
The existing test for this path passed because it constructed its fixture repositories with globally distinct identifiers — the one input distribution, out of every distribution the real world produces, under which two repositories' row ids cannot collide. The fix keys deduplication on a globally unique identity instead of the raw row id, and the new test deliberately reuses row ids across repositories to force the collision the old test structurally avoided.
How do you know your check actually checks: the audit trail and the self-audit
Two pieces of this arc aren't part of the four-instance spine above, but they belong in the same conversation because they ask the identical question from a different angle: not "does this test exercise the code," but "does this check actually check the thing it claims to."
v3.1.4 found that trelix audit's list, verify, and export commands — the read path over the hash-chained audit trail introduced in v3.0.0 — used the same store constructor as the writer, which runs CREATE TABLE IF NOT EXISTS on open. Point any of the three read commands at a file that wasn't an audit log, and they added the audit schema to it on the spot, then reported "Audit chain intact" at exit 0, because the chain they had just silently created was empty, and an empty chain is, trivially, consistent. One measured case took an 8 KB, one-table SQLite file to 32 KB and five tables just by being read. A CI integrity gate pointed at the wrong path would pass green while mutating the exact file it was supposed to verify. All three commands now open the file read-only (file:<path>?mode=ro), run no DDL, and exit 2 unless the audit schema is already present. It is the same question as the mutation-testing thesis, asked of an integrity check instead of a unit test: a verification step that can succeed against input it was never supposed to accept isn't verifying anything.
v3.1.2 is the plainest statement of the whole arc's thesis, predating the four-instance spine above by weeks. It's a self-audit: the team indexed trelix's own repository with trelix and checked whether every feature the configuration turned on was actually doing anything. The finding, in the release's own words, was "features that were on and doing nothing." File summaries were enabled and the index had zero of them. PageRank boosting had never once fired. taint_flows was empty on a repository semgrep does find real flows in. The query planner had classified all 219 recorded queries as the same one of its eight intents. None of these failures were visible from outside, because each failure path either logged at DEBUG while the CLI runs at WARNING, or was swallowed by a bare except, or reported a number nobody had reason to doubt. That release's own retrospective on its test suite reads almost like an early draft of this article's thesis: the taint parser was verified against a fixture invented to match its own misreading; every planner test supplied no credentials, so none could notice credentials being silently dropped; the eval metric tests used unique IDs only, so none could notice a repeated ID scoring twice.
Briefly, because it doesn't compete for space with any of this: v3.1.3 also shipped real security fixes — a REST API containment check that validated a caller-supplied path against a root the same caller supplied, and a stored XSS in generated graph HTML from an unescaped symbol name or filename. Both are fixed and both matter, but they're a different kind of bug from everything above, and this article already has its through-line without them.
The fix: mutation testing that measures whether a test can fail
v3.2.1 is a test-infrastructure release. It changes no stored vectors, no retrieval behavior, and no CLI surface — its one production fix is that retry.py's status-code extractor imported every supported LLM provider SDK (anthropic, openai, google.genai, boto3, azure) on every retry decision regardless of which backend actually raised, pulling torch transitively into processes that use no LLM feature at all. Everything else in the release is instrumentation, and it exists because coverage percentage — the metric the previous eight releases had been implicitly relying on — cannot answer the question every instance above turned on: can this test actually fail?
The core of it is a new scoped mutation-testing driver, scripts/mutation.py, wrapping mutmut. It runs against a throwaway git worktree rather than the live tree — unscoped mutant generation writes roughly 143 MB of Python that has no business near a commit — and it ratchets a per-module survivor count in scripts/mutation_baseline.json, deliberately never a survivor ratio: a repo-wide ratio moves whenever an unrelated segfault appears or disappears in a part of the codebase mutation testing can't safely reach, and parts of trelix — anywhere a real torch model gets constructed — are exactly that. First real measurements landed for the parser's per-language extractors (split from one coarse scope key into 23 granular ones), store.db, store.vector, indexing.chunker, compression.extractive, and graph. Closing the gaps mutation testing surfaced meant real new coverage: walker extension-map handling, ContextualChunker boundary conditions, graph community-detection survivor patterns, and operator-environment-variable leak/scrub coverage.
Branch coverage is now on (--cov-branch), with per-package floors enforced separately from the unit run itself — and here the changelog for this release actually gets its own mechanism wrong, so it's worth stating plainly: the floors are not in a file called tests/coverage-floors.json, which doesn't exist anywhere in this repository. They're a FLOORS dict hardcoded directly inside scripts/check_coverage_floors.py, checked in CI against a coverage report the unit job writes to coverage-unit.json (pytest tests/unit/ --cov-report=json:coverage-unit.json, followed by python scripts/check_coverage_floors.py coverage-unit.json). The script's own docstring explains why it's a script and not a test: pytest-cov only writes its report at session finish, after every test has already resolved, so a test can't read its own run's coverage — and a check that can only skip when the report file is missing is a green identical to a check that never ran, which is precisely the defect class this whole release exists to close.
The rest of the hardening follows the same logic. The suite is now hermetic: outbound sockets are banned via pytest-socket, every job is timeout-bounded, and the live-LLM integration tests in tests/integration/test_llm_e2e.py now require an explicit TRELIX_LIVE_LLM_TESTS=1 instead of running whenever a .env file happened to be present — cutting roughly half the suite's wall clock along with the credential exposure that came bundled with it. Sixteen known Java and Rust extractor defects are pinned as xfail(strict=True) rather than left silently uncovered, so each one XPASSes loudly, failing the build, if it's ever fixed without an accompanying changelog entry. A new marker taxonomy means a typo in a -m selector can no longer silently collect and pass the entire suite. None of this catches every possible defect, and it doesn't claim to; the mutation baseline is a ratchet, not a finish line. What it does is make "the test passed but never touched the bug" structurally harder to repeat than it was for 28 releases of bge-code, one release of the sparse leg, and however many releases intent_hint and search-all shipped broken before anyone measured directly instead of trusting the green check.
Top comments (0)