DEV Community: SAI RAM

trelix v3.2.2 to v3.2.5: The Source Tree Was Fine. The Published Package Wasn't.

SAI RAM — Sun, 06 Sep 2026 12:29:30 +0000

Run this against the real, published image and watch it fail:

docker run --rm --entrypoint trelix-mcp ghcr.io/sairam0424/trelix:3.2.1 --version

Exit code 127. Not a crash inside trelix-mcp, not a stack trace, not a permissions error — 127 is the shell's own way of saying the binary you asked for does not exist. And it didn't. The console script trelix-mcp is supposed to install as part of every trelix package was simply absent from the image, on both the slim tag and the -local tag, for the entire life of the 3.2.1 release. Every unit test in the suite was green. Every line of source that builds trelix-mcp was correct. The thing a user would actually get from docker pull did not have the binary its own --version flag implies exists.

This article covers four releases — v3.2.2, v3.2.3, v3.2.4, and v3.2.5 — spanning 173 commits and 88 changed files since v3.2.1, which is where the last article in this series left off. That one was about tests that pass without exercising the code they claim to cover: a MagicMock standing in for a real embedder, an all-ones attention mask that makes masked and unmasked math identical, a unit test that asserted a bug as its own specification. This one, on the heels of the mutation-testing push that closed out that arc, is about a different and in some ways more uncomfortable failure mode: tests that pass while exercising the wrong artifact entirely. A green pytest run against src/ says nothing about whether the wheel on PyPI, the image on GHCR, or the binary on the GitHub Releases page actually does what it claims. Those are three separate build products, built by three separate pipelines, and none of trelix's 4,353 collected unit tests had ever touched any of them directly. v3.2.2 through v3.2.4 is the story of finding that gap and closing it with an actual gate, not a promise to be more careful next time. v3.2.5 is a short postscript proving the discipline stuck.

The Docker image that shipped without its own server

The 127 above wasn't hypothetical or reconstructed after the fact — it's the literal command a human ran by hand against the real published v3.2.1 image, now baked verbatim into scripts/verify_release.py's Docker check with a comment explaining why: "this exact command returned exit 127 on the published 3.2.1 image before trelix-mcp was added to the Dockerfile." Root cause was doubly blocked. The Dockerfile's builder stage never had a COPY packages/trelix-mcp/ line — it only ever copied and installed core trelix. And even if someone had added that line, the repo's .dockerignore had a single bare entry, packages/, that would have silently excluded the whole directory from the build context anyway. Two independent gaps, each individually sufficient to explain the missing binary, both present at once — the kind of thing an ordinary Dockerfile-diff review would not catch, since the second gap lives in a different file entirely. The fix, across commits aabaa14 and 08233f8, rewrites .dockerignore to packages/* followed by !packages/trelix-mcp, and adds the missing COPY packages/trelix-mcp/ packages/trelix-mcp/ plus its install into the same pip install invocation as core. Bundling it costs roughly 84MB on the slim tag by the release's own measurement — trelix-mcp's dependencies are mcp, fastmcp, and trelix itself, nothing that pulls in torch — but that 84MB figure is reported prose in the changelog, not a number any script in this repo computes or asserts. The mechanism that makes it plausible (no ML dependency in trelix-mcp's own pyproject.toml) is verifiable; the exact delta isn't.

The second 3.2.2 bug lived one layer up, in the console script itself. Before the fix, trelix-mcp's main() never inspected sys.argv at all — not incorrectly, not partially, not at all. There was no import argparse, no reference to sys.argv, nothing. Running trelix-mcp --help from a real shell didn't print usage; it silently launched the actual MCP stdio server and sat waiting for JSON-RPC input on stdin, the exact opposite of every CLI convention a --help flag implies. The fix is small enough to quote in full:

def main() -> None:
    """Entry point for the trelix-mcp server (stdio transport).

    Parses argv only for --help/--version/unknown-flag rejection — the normal path (no
    args, launched by an MCP client's server config) falls straight through to running
    the server, unchanged from before this parser existed.
    """
    parser = argparse.ArgumentParser(
        prog="trelix-mcp",
        description="MCP server for trelix — semantic code search over stdio.",
    )
    parser.add_argument("--version", action="version", version=f"trelix-mcp {__version__}")
    parser.parse_args()

Both 3.2.2 bugs share the same shape: something was correct in principle and broken in the specific packaging or invocation path a real user takes. Neither one could have failed a unit test, because no unit test in the suite ever installed a wheel, pulled an image, or ran a console script as a subprocess against sys.argv. They were found by a manual production-verification pass — installing and running the actual shipped PyPI packages and Docker images rather than the source tree — and that pass had, by this point, been run by hand twice: once for 3.2.1, once for 3.2.2. Running it a third time by hand was the thing the rest of this arc exists to stop doing.

The wildcard leak and the response that mattered more than the fix

v3.2.3's actual bug is narrow and specific: Database.bm25_search's path_filter branch, plus three path_filter-scoped queries in src/trelix/retrieval/grep_search.py, built a SQL LIKE pattern directly from a caller-supplied path without escaping it. LIKE treats % and _ as wildcards inside the pattern value itself, which has nothing to do with SQL injection — every one of these queries was already parameterized with ? — and everything to do with LIKE's own semantics. A path_filter of src_auth matches not just the directory src_auth but also, say, srcXauth, because the underscore in the filter is read as "any single character," not a literal underscore. Given how common underscores are in real directory and file names (test_utils.py, my_function), this wasn't an edge case; it was a routine collision waiting for the right two sibling paths to exist in the same repository. The fix lives in escape_like_pattern() at src/trelix/store/db.py:291, whose docstring is worth quoting because it states the distinction precisely:

def escape_like_pattern(value: str) -> str:
    """Escape `%`, `_`, and `\\` in `value` so it is safe as a SQL LIKE pattern segment.

    LIKE treats `%` and `_` as wildcards in the PATTERN VALUE itself — this has nothing
    to do with SQL injection (the callers here already use `?` parameterization) and
    everything to do with LIKE's own semantics.
    """

The sibling method get_index_metadata_with_prefix had already been doing this correctly for a while, pairing escape_like_pattern() with an explicit ESCAPE '\\' clause — so the fix wasn't a novel idiom, it was applying an already-proven pattern to the four call sites that had been missed. It's also not a total fix: grep_search.py's own docstring flags a second, separate, still-open instance of the same defect class on the symbol-name prefix match in _name_search, explicitly out of scope for this release. I'd rather state that than let the fix read as more complete than it is.

What makes v3.2.3 matter for this article isn't the bug — it's what came after it. The team's own reasoning, written into the new test suite's module docstring, is that no existing test, including Click's CliRunner-style in-process mocking and the MCP SDK's own official in-memory Client, is structurally capable of seeing a defect at the real process boundary — the exact class that let trelix-mcp's --help bug ship in 3.2.1. In-memory transports never cross an actual OS process boundary; they can't see a console script that isn't installed, or a subprocess that hangs instead of exiting. So tests/e2e/test_mcp_stdio_e2e.py spawns trelix-mcp as a genuine child process, located via shutil.which("trelix-mcp") — the test fails loudly if the binary isn't on PATH, not just importable from src/ — and talks to it over real stdin/stdout JSON-RPC via the MCP SDK's stdio_client. A companion test, test_pypi_dist_install_e2e.py, builds real wheels for all four published packages, installs each into a fresh venv, and asserts the installed __version__ matches the working tree — never editable, never a live PyPI pull, by explicit design.

That test suite only matters if something forces it to run before a release goes out. In .github/workflows/release.yml, the publish job's dependencies now include a new smoke-test-built-artifacts job that downloads the just-built wheel artifacts, installs them with pip install — never editable — and runs the e2e suite against that install before publish is permitted to execute. It is a real edge in the workflow's dependency graph, not a comment or a convention. Two smaller CI extensions round it out: ci.yml's Docker job now runs docker run --rm --entrypoint trelix-mcp trelix:ci-test --version against every built image, and helm-lint.yml asserts the rendered Helm chart's image tag matches Chart.yaml's appVersion. Both are the kind of cheap check that would have caught a 3.2.1-shaped regression on the next PR instead of the next production audit.

Automating the verification instead of re-typing it

v3.2.4's capstone answers a question the last two releases raised without answering: once you've written the checks, who runs them, and when? Before this release, the answer was a person, watching the GitHub Actions tab for both the Release and Docker Publish workflows to finish, then running scripts/verify_release.py by hand and reading its output. .github/workflows/verify-release.yml replaces that with a workflow that listens for workflow_run completion events from both Release and Docker Publish. Because that event fires twice — once per upstream workflow finishing — the job does a single, non-blocking check via the GitHub CLI (gh run list --workflow "$wf" --branch "$tag") to see whether the other workflow has also gone green for the same tag; if not, it exits cleanly rather than polling, and whichever of the two triggering runs happens to fire second is the one that proceeds. Once both are confirmed green, it runs the same scripts/verify_release.py script a human used to run by hand — PyPI installs into fresh venvs for all four packages, Docker images smoke-tested with the exact --entrypoint trelix-mcp command that returned 127 on 3.2.1, a Helm chart checked out into an isolated worktree and rendered across all three backends, a GitHub Release binary actually executed, and a pip-audit plus wheel-content secret scan — and posts its own pass/fail summary as a workflow run. CONTRIBUTING.md still documents the manual command for ad-hoc re-verification, which matters, because automating the check doesn't mean giving up the ability to run it by hand when you're debugging the check itself.

The other audit: when a symbol's nickname collides with someone else's

Running in parallel with the artifact-verification work, v3.2.4 also shipped a distinct methodology: an audit for silent symbol-collision, not artifact drift. The pattern, once named, shows up in four unrelated corners of the codebase: something that should carry a full, unique address instead answers to a bare nickname, and when two different things share that nickname, one of them silently disappears or gets misattributed.

The clearest instance is in Java. Before the fix, every nested class got a bare qualified_name — a class named Config nested inside ServerConfig and a different class also named Config nested inside ClientConfig both indexed under the identical name Config, with parent_id set to None regardless of which outer class actually enclosed them. Re-indexing one silently left the other's stale row permanently in the database. The fix threads an outer_qualified_name recursively through the class walk in java.py, so a nested class now gets f"{outer_qualified_name}.{name}" and a real parent_id. The Java record-component bug is a different kind of mistake in the same file: the extractor checked for tree-sitter node types record_parameters (the container) and record_component (the item) that the installed grammar simply doesn't emit — it emits formal_parameters and formal_parameter — so every record in the corpus indexed zero fields, silently, because the code was written against a node vocabulary that never existed in the grammar it ran against.

Rust has the identical collision shape one file over. A function defined inside mod inner { } previously got the same bare qualified_name as any other function of the same name elsewhere in the file — the fix in rust.py threads a module_path through the module walk and rewrites the qualified name to inner::nested for anything defined inside it. A separate Rust fix corrected the brace-import flattener, which checked for node types use_tree_list and use_tree that don't exist in the installed grammar either — the real node types are scoped_use_list and use_list — so the dominant Rust import form, use foo::{Alpha, Beta}, fell through to a generic fallback that stored the literal text "{Alpha, Beta}" as a single bogus import name, making every brace-imported symbol invisible to import-graph queries.

The same audit surfaced the collision shape outside the parsers entirely. AgentLoop._do_get_symbol, when asked for an exact qualified name that didn't resolve, used to fall back to an arbitrary bare-name match — silently handing the agent a different symbol's body than the one it asked for, with no error signal. The fix requires exactly one exact match or reports "not found." Federation's make_scip_symbol_id() used to hash a cross-repo symbol identity from just (package, version, qualified_name), omitting file_path — meaning two different files in the same package defining def main() hashed to the identical 16-character id, and INSERT OR IGNORE against that id as a primary key silently dropped the second file's row rather than erroring. It's the most explicit statement of the whole pattern in the release: an identity hash that omitted one required scoping field, and a database that dropped the collision without complaint.

Not every bug the same audit pass caught fits that collision shape, and I'd rather say so than force the frame. The chunker's token_count bug was a stale cached value — computed against the pre-truncation text instead of the actual, truncated chunk that got stored. bm25's stop-word handling had a related but distinct problem: a query made entirely of stop words was supposed to hit the empty-query fallback and return the FTS5 sentinel path, but the stop-word filter ran after that check instead of before it, so a stop-words-only query was treated as a normal, non-empty query instead. The walker's fix normalizes rel_path to NFC via unicodedata.normalize, because a single real file could take two different Unicode-normalization forms across walks (NFD from an HFS+ filesystem versus NFC elsewhere), fracturing one file's identity into what change detection read as a phantom delete-and-add. The graph community-detection fix removed a force-cast to int on node ids that crashed with ValueError on any ticket-linked repo, because ticket node ids are strings by design. And the CLI's update_index command now actually checks result["status"] and exits 1 on failure, instead of printing the JSON error payload and returning exit 0 regardless. Different defect shapes, same audit pass, same release — worth naming honestly as a second act rather than folding into a pattern it doesn't share.

A short coda: the binary that gave advice it couldn't act on

v3.2.5 is one fix, and it's a fitting close because it was caught by the same discipline the verify-release capstone exists to encode: someone read the error message the real, shipped v3.2.4 binary actually prints, in production, rather than the source that built it. The standalone GitHub Release binary's local embedder — used when sentence-transformers isn't available — told users to run pip install 'trelix[local]'. That advice is correct for the pip-installed package. It is actively useless for the frozen PyInstaller binary, which is built with sentence-transformers, torch, and the rest of the ML stack deliberately excluded to keep it small, and which never consults the host's Python or pip environment at all — installing anything on the host has zero effect on a binary that doesn't read it. The fix, in src/trelix/embedder/base.py, checks getattr(sys, "frozen", False) — the standard attribute PyInstaller sets — and gives the frozen binary a message pointing at an API-backed provider or the Python package instead, leaving the pip-installed package's message byte-for-byte unchanged. It's covered by a test that asserts the frozen message never contains the string "pip install." One line of misdirection, found the same way the Docker gap and the console-script gap were found ten weeks and four releases earlier: by running the actual thing that ships, not the code that built it.

trelix v3.1.2 to v3.2.1: The Tests That Passed Without Testing Anything

SAI RAM — Wed, 26 Aug 2026 13:53:41 +0000

In FlagEmbedding 1.4.0, from FlagEmbedding import FlagModel is an alias for from .base import BaseEmbedder as FlagModel — the encoder-only base class, whose default pooling_method is "cls" and whose pooling() for that method is one line: return last_hidden_state[:, 0]. BAAI/bge-code-v1 is not an encoder. It is a causal Qwen2 decoder, published with 1_Pooling/config.json declaring pooling_mode_lasttoken: true and pooling_mode_cls_token: false. Position 0 of a causal decoder cannot attend forward. So the CLS vector bge-code was computing depended on exactly one token and nothing else that followed it.

I measured what that means with a randomly initialized Qwen2Model pooled by FlagEmbedding's real pooling(): two sequences identical at token 0 and different in every token after it came back with cosine similarity 1.0 and max absolute difference 0.0 — bitwise identical. The same hidden states through the real last_token_pool gave cosine 0.10; through mean, 0.65. Across 200 resamples of every token after position 0, position 0's output never moved once. And because encode_queries prefixes every query with the same instruction, token 0 is that shared prefix for every query trelix ever sends — meaning every query embedding bge-code produced was identical to every other one, full stop. Five distinct code chunks sharing a first token collapsed into a single vector, against five distinct vectors from last_token_pool on the same input.

This shipped in 28 tagged releases — every tag from v2.0.0 through v3.1.6 contains the commit that introduced the embedder — and stayed green through every one of them because tests/unit/test_embedder_bge.py replaces FlagModel with a MagicMock. A MagicMock answers to any attribute you ask of it. It never once called the real pooling(), so a defect that destroys every query embedding a provider produces never had a chance to fail a test.

v3.1.6 fixed a real, adjacent bug: BGECodeEmbedder.dimension called a method FlagEmbedding has never had, raising AttributeError before an index could even start, and the dimension it fell back to (768) was wrong anyway — the model emits 1536. That fix was correct and it shipped correctly. But removing the raise made a path reachable that had never been reachable before: the pooling defect. The v3.1.6 changelog entry read as though bge-code now worked. It didn't, and one release later, v3.1.7 said so in public, in the same file: "Retracted: bge-code does not 'work now'." I don't read that as an embarrassment. A team that ships a wrong claim and then corrects it in the next release, with the receipts, is doing exactly what a correctness process is for. The alternative — quietly letting the framing stand — is the actual failure mode.

Eight releases, one arc

This is the story of eight tagged releases — v3.1.2, v3.1.3, v3.1.4, v3.1.5, v3.1.6, v3.1.7, v3.2.0, and v3.2.1 — spanning 190 commits and 324 changed files since v3.1.1, where the last article in this series left off. v3.2.1 is the current shipping release: tagged, dated 2026-08-26, live on PyPI, installable today with pip install trelix==3.2.1. The unit suite sits at 4,376 tests collected on this exact checkout. trelix itself is about nine weeks old — the first commit landed 2026-06-25 — which matters for how to read what follows: this isn't a decade of debt being paid down, it's a project auditing itself hard and often, early, while the cost of finding a defect is still one release instead of ten.

Unlike the v3.0.0 span this series already covered — audit trail, OIDC SSO, the VS Code extension, context compression, model-aware budgeting, extended thinking — this span has no comparable feature drop. It is, almost entirely, a correctness and audit arc. I'm not going to apologize for that framing or bury it under a feature list that doesn't fit the material. The interesting thing about these eight releases isn't what got built; it's what got caught, and specifically the shape of what got caught, which repeats often enough across independent parts of the codebase that it deserves to be named as a pattern rather than four unrelated bug reports.

The pattern: a green suite that never touched the defect

Here is the pattern, stated once so the four instances below don't have to keep re-deriving it: a test can pass without ever exercising the behavior it claims to cover. That's not the same claim as "the tests were bad" or "coverage was low." These tests ran. They asserted things. They turned green in CI, release after release. And in every one of the four cases below, the reason they turned green is diagnosable and specific — not vague test debt, but one of four concrete mechanisms.

bge-code's pooling defect survived because a MagicMock answers any attribute asked of it, so no test ever called the real pooling() method that carried the bug. The sparse-vector padding defect, below, survived because every existing fake for the tokenizer returned an all-ones attention mask by construction, which makes masked and unmasked aggregation produce identical numbers regardless of whether the masking code is even present. The intent_hint dispatch bug survived because a unit test asserted the observed, buggy output as the correct value — it had to be deleted, not fixed, once the real behavior was understood. And the federated search-all deduplication bug survived because the existing test happened to use the one input distribution — globally unique row identifiers — that cannot trigger a collision, out of all the distributions that could.

Four independent parts of the codebase — an embedder, a different embedder's aggregation math, a retrieval planner, and a federation layer — produced the same failure mode by four different routes. That repetition is why v3.2.1 exists, and why I'm leading with it rather than a version-by-version changelog walk.

The sparse leg: clean queries scored against contaminated documents

SparseEmbedder.embed tokenizes a batch with padding=True, which pads every sequence in that batch out to the length of its longest member, and then aggregates with torch.log(1 + torch.relu(logits)).max(dim=1)) — a max taken over every sequence position, padding included. A masked-language-model head predicts a real, nonzero logit distribution at pad positions too; it doesn't know they're padding. Nothing in the aggregation multiplied by the attention mask to zero those positions back out. grep -c attention_mask src/trelix/embedder/sparse.py returned 0. A tree-wide grep -rln attention_mask src/trelix/ matched no file in the codebase at all.

I measured it against the real naver/splade-v3-distilbert at top_k=128. A 28-token chunk embedded alone, compared against the identical chunk embedded in a batch alongside a 185-token chunk, gained 28 phantom terms and lost 28 real ones at the cutoff — 22% of the stored vector — with 0.478 max weight drift on the terms that survived both runs. The phantom terms are ordinary English words the chunk never contained: where, store, numbers, text, gene, sequence, phrase, messages. The control that pins the cause is the one that matters most here: the same chunk batched with an equal-length chunk, which needs no padding at all, came back bit-identical to the chunk embedded alone. Batching was never the problem. Padding was.

Two things made this worth a minor release rather than a footnote. sparse_embeddings is a persisted table in store/db.py, so the corruption wasn't transient — reindexing the same repository in a different file order, or with a different TRELIX_SPARSE_BATCH_SIZE, rewrote every row with a different, equally wrong answer, and no index was reproducible against itself. And embed_query routes through embed([text]) — a batch of exactly one, which needs no padding — so every query trelix ever issued against the sparse leg was clean, scored against documents that weren't. No amount of query-side tuning could have surfaced or corrected that asymmetry, because the query side was never where the bug lived.

The existing test_sparse_embedder.py had ten tests covering this embedder, and every one of its fakes returned attention_mask=torch.ones(...). An all-ones mask makes the masked and the unmasked aggregation mathematically identical, so all ten tests passed whether or not the masking code existed at all — which it didn't. The replacement, test_sparse_padding_contamination.py, is nine tests, every one mutation-verified: deliberately dropping the mask multiplication from the aggregation fails five of them while leaving the equal-length control passing, which is exactly the signature that localizes the cause to padding rather than to batching in general.

intent_hint: a test that asserted the bug as the spec

intent_hint is an optional parameter on both the REST /search endpoint and the MCP search_code tool, meant to let a caller skip trelix's own LLM-based intent classification and specify a retrieval strategy directly. The hint builder had a routing bug: it stamped the "direct answer" tier onto all eight recognized intent values while separately setting the correct leg strategy for each — but the executor checks the tier first, so the strategy it computed was never read. Every intent_hint value, regardless of which of the eight intents you actually asked for, took the same direct-answer shortcut.

I measured this against trelix's own codebase: all eight intent values returned byte-identical output — 40 README sections, zero code files, and no overlap at all with the actual correct result set for any of them. intent_hint had been silently disabling retrieval since v2.10.0, over both protocols, for anyone who used it.

The reason it took this long to catch is the plainest of the four: a unit test existed that asserted the broken output was the expected one. That's not a gap in coverage — it's a test that actively encodes the bug as the specification. It had to be deleted and replaced with assertions on the actual retrieval outcome, not patched.

search-all: a test that used the one distribution that can't fail

Federated search-all fans a query out across every registered repository and merges the results. The merge deduplicated on each result's per-database autoincrement row id — and every repository's ids start at 1. Two repositories' top hits collide on id 1 unavoidably; first-seen wins the merge, and "first" was whichever repository's thread finished first in the pool, which is nondeterministic. The consequence: an entire repository's results could vanish from a federated search, silently, while the response reported repos_skipped: 0 — nothing told you a repository had been dropped, because nothing had been skipped; its results had simply lost a dedup race.

The existing test for this path passed because it constructed its fixture repositories with globally distinct identifiers — the one input distribution, out of every distribution the real world produces, under which two repositories' row ids cannot collide. The fix keys deduplication on a globally unique identity instead of the raw row id, and the new test deliberately reuses row ids across repositories to force the collision the old test structurally avoided.

How do you know your check actually checks: the audit trail and the self-audit

Two pieces of this arc aren't part of the four-instance spine above, but they belong in the same conversation because they ask the identical question from a different angle: not "does this test exercise the code," but "does this check actually check the thing it claims to."

v3.1.4 found that trelix audit's list, verify, and export commands — the read path over the hash-chained audit trail introduced in v3.0.0 — used the same store constructor as the writer, which runs CREATE TABLE IF NOT EXISTS on open. Point any of the three read commands at a file that wasn't an audit log, and they added the audit schema to it on the spot, then reported "Audit chain intact" at exit 0, because the chain they had just silently created was empty, and an empty chain is, trivially, consistent. One measured case took an 8 KB, one-table SQLite file to 32 KB and five tables just by being read. A CI integrity gate pointed at the wrong path would pass green while mutating the exact file it was supposed to verify. All three commands now open the file read-only (file:<path>?mode=ro), run no DDL, and exit 2 unless the audit schema is already present. It is the same question as the mutation-testing thesis, asked of an integrity check instead of a unit test: a verification step that can succeed against input it was never supposed to accept isn't verifying anything.

v3.1.2 is the plainest statement of the whole arc's thesis, predating the four-instance spine above by weeks. It's a self-audit: the team indexed trelix's own repository with trelix and checked whether every feature the configuration turned on was actually doing anything. The finding, in the release's own words, was "features that were on and doing nothing." File summaries were enabled and the index had zero of them. PageRank boosting had never once fired. taint_flows was empty on a repository semgrep does find real flows in. The query planner had classified all 219 recorded queries as the same one of its eight intents. None of these failures were visible from outside, because each failure path either logged at DEBUG while the CLI runs at WARNING, or was swallowed by a bare except, or reported a number nobody had reason to doubt. That release's own retrospective on its test suite reads almost like an early draft of this article's thesis: the taint parser was verified against a fixture invented to match its own misreading; every planner test supplied no credentials, so none could notice credentials being silently dropped; the eval metric tests used unique IDs only, so none could notice a repeated ID scoring twice.

Briefly, because it doesn't compete for space with any of this: v3.1.3 also shipped real security fixes — a REST API containment check that validated a caller-supplied path against a root the same caller supplied, and a stored XSS in generated graph HTML from an unescaped symbol name or filename. Both are fixed and both matter, but they're a different kind of bug from everything above, and this article already has its through-line without them.

The fix: mutation testing that measures whether a test can fail

v3.2.1 is a test-infrastructure release. It changes no stored vectors, no retrieval behavior, and no CLI surface — its one production fix is that retry.py's status-code extractor imported every supported LLM provider SDK (anthropic, openai, google.genai, boto3, azure) on every retry decision regardless of which backend actually raised, pulling torch transitively into processes that use no LLM feature at all. Everything else in the release is instrumentation, and it exists because coverage percentage — the metric the previous eight releases had been implicitly relying on — cannot answer the question every instance above turned on: can this test actually fail?

The core of it is a new scoped mutation-testing driver, scripts/mutation.py, wrapping mutmut. It runs against a throwaway git worktree rather than the live tree — unscoped mutant generation writes roughly 143 MB of Python that has no business near a commit — and it ratchets a per-module survivor count in scripts/mutation_baseline.json, deliberately never a survivor ratio: a repo-wide ratio moves whenever an unrelated segfault appears or disappears in a part of the codebase mutation testing can't safely reach, and parts of trelix — anywhere a real torch model gets constructed — are exactly that. First real measurements landed for the parser's per-language extractors (split from one coarse scope key into 23 granular ones), store.db, store.vector, indexing.chunker, compression.extractive, and graph. Closing the gaps mutation testing surfaced meant real new coverage: walker extension-map handling, ContextualChunker boundary conditions, graph community-detection survivor patterns, and operator-environment-variable leak/scrub coverage.

Branch coverage is now on (--cov-branch), with per-package floors enforced separately from the unit run itself — and here the changelog for this release actually gets its own mechanism wrong, so it's worth stating plainly: the floors are not in a file called tests/coverage-floors.json, which doesn't exist anywhere in this repository. They're a FLOORS dict hardcoded directly inside scripts/check_coverage_floors.py, checked in CI against a coverage report the unit job writes to coverage-unit.json (pytest tests/unit/ --cov-report=json:coverage-unit.json, followed by python scripts/check_coverage_floors.py coverage-unit.json). The script's own docstring explains why it's a script and not a test: pytest-cov only writes its report at session finish, after every test has already resolved, so a test can't read its own run's coverage — and a check that can only skip when the report file is missing is a green identical to a check that never ran, which is precisely the defect class this whole release exists to close.

The rest of the hardening follows the same logic. The suite is now hermetic: outbound sockets are banned via pytest-socket, every job is timeout-bounded, and the live-LLM integration tests in tests/integration/test_llm_e2e.py now require an explicit TRELIX_LIVE_LLM_TESTS=1 instead of running whenever a .env file happened to be present — cutting roughly half the suite's wall clock along with the credential exposure that came bundled with it. Sixteen known Java and Rust extractor defects are pinned as xfail(strict=True) rather than left silently uncovered, so each one XPASSes loudly, failing the build, if it's ever fixed without an accompanying changelog entry. A new marker taxonomy means a typo in a -m selector can no longer silently collect and pass the entire suite. None of this catches every possible defect, and it doesn't claim to; the mutation baseline is a ratchet, not a finish line. What it does is make "the test passed but never touched the bug" structurally harder to repeat than it was for 28 releases of bge-code, one release of the sparse leg, and however many releases intent_hint and search-all shipped broken before anyone measured directly instead of trusting the green check.

trelix v2.11.0 to v3.1.1: Six Feature Areas, Every One of Them Off By Default

SAI RAM — Sat, 15 Aug 2026 15:33:10 +0000

Seed three events into an audit database, then reach past the application and change one row by hand:

$ sqlite3 audit.db "UPDATE audit_log SET principal='attacker' WHERE id=2"
$ trelix audit verify --db audit.db
Audit chain TAMPERED — first divergent entry id: 2
$ echo $?
1

Delete the newest row instead and it still catches it, naming id 3, even though the surviving rows form a perfectly valid chain. Point it at something SQLite cannot open and it exits 2 rather than 0, because "I could not check" and "I checked and it is clean" must never collapse into the same green build.

None of that existed six releases ago. trelix audit verify is one command out of six feature areas that landed in trelix v3.0.0, and it is the one that most changes what the project is for.

What the major bump actually is

The span from v2.11.0 to v3.1.1 is six releases — v2.11.1, v2.12.0, v3.0.0, v3.0.1, v3.1.0 and v3.1.1, the last of them dated 2026-08-15 — 68 commits, 137 files changed, +19,829/-1,211 lines. v2.11.0 closed out the Jira and Linear connector work, which has its own story. Everything after it is a different kind of release.

v3.0.0 carries six new feature areas: Anthropic extended thinking, a model-aware context budget, a VS Code extension that acts instead of merely displaying, a hash-chained append-only audit trail, OIDC SSO, and query-conditioned context compression. Alongside them, an opt-in FTS5 declaration boost for keyword ranking.

It is a major bump because of scope, not breakage. Every one of those six is additive and off by default: TRELIX_AUDIT_ENABLED=false, TRELIX_OIDC_ENABLED=false, TRELIX_LLM_THINKING_ENABLED=false, TRELIX_RETRIEVAL_COMPRESSION=false, declaration_boost_enabled False, and context_token_budget still the exact 12_000 integer it was in v2.12.0. A default v3.0.0 install assembles context byte-identically to a default v2.12.0 install, and there is a test that proves it rather than a release note that asserts it.

An audit trail you can hand to somebody else

The trail is a separate SQLite file. AuditConfig.resolved_db_path defaults to <cwd>/.trelix/audit.db, never the index database, for one blunt reason: the index is disposable, and an audit trail that disappears on reindex is not an audit trail.

Each row's entry_hash is sha256(prev_hash || canonical_json(content)), canonicalised in ten lines of src/trelix/audit/store.py with sort_keys=True and separators=(",", ":"). Exactly eleven columns are hashed, in a fixed _CONTENT_COLUMNS tuple, and the DB-assigned id is deliberately not among them: a writer cannot hash a value it does not have until after the INSERT.

That exclusion is why row ordering needs a second defence, and it explains the deleted-tail catch from the opening: a hash chain is structurally blind to truncation. An audit_meta table carries a running count and head_hash, upserted inside the same with self._conn: transaction as the insert, and verify_chain checks both against the chain it just walked. audit_log.id is INTEGER PRIMARY KEY AUTOINCREMENT rather than a rowid alias, so a delete-then-refill cannot close the gap it made.

export TRELIX_AUDIT_ENABLED=true turns the trail on and trelix serve ./my-repo starts recording (TRELIX_AUDIT_DB_PATH moves the file if you want it off <cwd>/.trelix/audit.db), trelix audit list -n 50 reads it, and trelix audit export --format ndjson writes one JSON object per line so a SIEM can be fed by Filebeat or Vector. All of it is stdlib sqlite3, 54 tests cover the surface, and the store is fail-open by default (AuditConfig.fail_closed: bool = False) so a full disk is not an outage.

SSO is the identity half, and it is deliberately narrow: trelix is a resource server, not an OIDC client. No /login, no /callback, no code exchange, no refresh. It verifies tokens callers already hold. Three variables stand it up — TRELIX_OIDC_ENABLED=true, TRELIX_OIDC_ISSUER and TRELIX_OIDC_AUDIENCE — on top of pip install 'trelix[sso]'; TRELIX_OIDC_ALGORITHMS defaults to ["RS256", "ES256"] and is asymmetric-only, enforced in OidcVerifier.__init__, again in authenticate against the unverified JWS header before any key resolution, and a third time in jwt.decode. The test that matters skips trelix's own header gate entirely: it signs a token with HS256 using a real 2048-bit public key's PEM as the HMAC secret — the canonical algorithm-confusion forgery — and asserts jwt.decode alone still refuses it. Principal.principal_id is f"{self.subject}@{self.issuer}", never derived from email, because email-keyed identity is an account-takeover primitive, and the JWKS fetch is HTTPS-only, host-pinned and capped at 1 MiB — not hypothetical, since the pre-fix response.read() let a hostile issuer drive RSS from 59 MB to about 662 MB in 0.2 seconds.

The limits are in the docs rather than the marketing. The trail is tamper-evident, not tamper-proof: the chain and the anchor that checks it live in the same file, so anyone with write access can rewrite a row, recompute every subsequent hash and update the anchor in one transaction. There is sha256 and no key. Only the HTTP surface is audited, so an MCP-only deployment produces an empty trail and the agent loop's per-turn calls are invisible. TRELIX_AUDIT_RETENTION_DAYS is declarative only — nothing prunes — and /health is audited, so liveness probes will dominate the row count. This is authentication, not authorization — OIDC groups are captured and stored but enforced nowhere — and there is no SAML, with no plan for it; put a broker in front.

Thinking, and a budget that knows which model you are talking to

Extended thinking is a request parameter, not a model. AnthropicBackend._thinking_kwargs() returns one dict, merged into both messages.create and messages.stream:

return {
    "thinking": {
        "type": "enabled",
        "budget_tokens": self._config.thinking_budget_tokens,
    }
}

No model string in trelix changes. TRELIX_LLM_THINKING_ENABLED defaults False and TRELIX_LLM_THINKING_BUDGET_TOKENS to 4096.

Only one component opts in — RetrievalSynthesizer, at two call sites. The index-time callers, ContextualChunker and FileSummarizer, do not pass the keyword at all; a single global flag would have turned an indexing run into roughly five to ten times the LLM cost to produce reasoning nobody reads.

One consequence is worth stating plainly: enabling thinking forces temperature to 1.0 in both complete() and stream(), overriding whatever the caller passed — and the synthesizer does pass 0.0 and 0.2, so a flag named thinking_enabled silently ends near-deterministic synthesis. Thinking also bills as output tokens with no separate counter, so ChatResponse.output_tokens mixes answer and reasoning.

The model-aware budget is opt-in via a value rather than a flag. context_token_budget: int | None still defaults to 12_000; set it to null, none, auto or empty and Retriever._resolve_effective_budget() returns int(window * context_window_fraction), fraction defaulting to 0.5. resolve_window() is a first-match scan over a hand-ordered 37-entry table in src/trelix/llm/context_windows.py, so gpt-4o-2024-11-20 resolves to 128000. Provider-prefixed Bedrock ids do not resolve: us.anthropic.claude-sonnet-4-20250514-v1:0 returns None and falls back to 12,000 with a WARNING — exactly the default a Bedrock user setting auto was trying to escape.

Raising the budget alone changes very little, and the config docstring says so: rerank_top_n defaults to 15 and top_k_vector/top_k_bm25 to 20/20, which cap the candidate pool before the packer ever sees a budget. TRELIX_RETRIEVAL_SCALE_TOP_K_TO_BUDGET is the explicit opt-in that multiplies top_k_vector and rerank_top_n — but not top_k_bm25 — by effective_budget / 12_000, taking top_k_vector from 20 to 106 on gpt-4o's 64,000. It stays off because it raises per-query cost on three axes at once.

Compression that cannot cost you a result

pack_compressed runs two waves. Wave 1 is the exact pre-existing uncompressed pack, with the compressor never called. Wave 2 walks the same candidate pool and skips anything already selected, so it only ever gets a second look at candidates wave 1 could not fit — the ones the old packer silently dropped. Accepted entries are appended, never substituted, then re-sorted into ranking order. The compressed selection is a strict superset of the uncompressed one, so Recall, MRR and nDCG cannot regress by construction. That is a property you read; no ranking experiment required. It cannot blow the ceiling either: remaining is recomputed per candidate, and anything that will not fit even at _FLOOR_RATIO = 0.01 lands in stats["skipped"].

Citation fidelity is structural. format_compressed_blocks() emits one [Lines a-b] <qualified_name> header per kept span with an explicit # ... N lines elided ... marker in every gap, and each header's text is sliced from the body by the same arithmetic that produced the header. Spans are clamped where they are created and re-filtered again at render time — two independent gates on one invariant, which matters because roughly 35 extractor sites store a truncated body while keeping the full AST span. A header claiming lines its own text does not contain manufactures a confident, wrong citation in a prompt that will quote it verbatim.

ExtractiveCompressor, the default provider, does zero query-time inference — it reads sub-chunk vectors that already exist from index time. The tripwire that lets me claim the off path is unchanged is tests/unit/test_assembler_backcompat_golden.py, which loads the pre-compression assembler out of git show v2.12.0:src/trelix/retrieval/assembler.py and diffs the new one across 8 intents — 177 assertions, including [id(r) for r in new_ctx.results] == [id(r) for r in old_ctx.results], so no reordering hides behind equal text.

The headline "query-conditioned" path is by default lexically conditioned: sub-chunk rows require TRELIX_CHUNKER_MULTI_GRANULARITY=true (default false), are Python-only even then, and need the query embedding already resident in the embedder's LRU. The changelog's performance claim keeps its hedge: roughly 30-60% fewer synthesis input tokens and 15-35% lower latency on a network-API synthesis path, on the compressible intents. That is an expectation, not a benchmark.

The editor surface, and one ranking flag

The VS Code extension is the most visible feature of the release and it needed no server work, because the MCP surface already was the API: it spawns trelix-mcp over stdio and builds all four features from search_code, get_symbol, ask_agent and blast_radius.

Code lenses re-fire on every keystroke, so the performance contract is enforced in the provider rather than documented. provideCodeLenses makes zero MCP calls — its only external call is vscode.executeDocumentSymbolProvider — and only the count-bearing lens reaches MCP, through resolveCodeLens, for lenses VS Code actually paints, keyed on ${docUri}@${docVersion}::${symbolName}. The version component is the difference between a cache and a correctness bug: any edit must miss rather than show a stale dependent count.

Chat became possible because of a bug fix that is the same change as the feature. TrelixMcpClient.ask() used to call getPrompt({ name: "trelix-search" }) and render the joined result as the answer — but getPrompt returns a filled-in prompt template, so what users saw was scaffolding telling them to use the search_code tool. No type checker could have caught it, because both calls return well-formed data and only one of them is an answer. The fix calls ask_agent, which runs the multi-turn ReAct loop and returns {answer, session_id, turn_count}; that session_id is what makes follow-up questions work, so multi-turn chat was structurally impossible until ask() moved. The regression test asserts getPromptCalls === 0.

The limitations sit next to the constants that cause them. const THREAD_KEY = "default" exists because the 1.95 chat API gives the handler no stable thread id, so two chat threads open at once share one agent session. The /explain, /search and /impact slash commands are stateless one-shot calls carrying no session at all. The extension is not on the Marketplace: cd workspace-vscode && npm install && npm run build && npm run package gets you a .vsix. Its manifest is still at 0.3.0, so the file name does not tell you which feature set it contains.

The declaration boost is the smallest feature in the release and the easiest to demonstrate, because trelix's own search engine could not find its own search engine. Query the live self-index for bm25_search and the method that implements it — Database.bm25_search in src/trelix/store/db.py — comes back at rank 34 of 90 matches under default unweighted FTS5, beaten by sixteen test functions and four documentation headings that merely mention the term. With top_k_bm25 defaulting to 20 that is a recall failure, not a ranking nuisance: it is out of the candidate pool before fusion or reranking runs. declaration_boost_weight=5.0 moves it to rank 9. The mechanism is a reweighted bm25(symbols_fts, ?, ?, 1.0, 1.0, 1.0) call applying the weight to name and qualified_name only — no schema change, no reindex — and it is one-directional: rows with no name match score byte-identically at both weights. It is gated twice, declaration_boost_enabled False and declaration_boost_weight 1.0, with the default no-op pinned by an exact-equality test over the returned score pairs. To reproduce the rank move: export TRELIX_RETRIEVAL_DECLARATION_BOOST=true and export TRELIX_RETRIEVAL_DECLARATION_BOOST_WEIGHT=5.0.

Then three releases proving the surface was true

A big feature drop earns a specific obligation, and v3.0.1, v3.1.0 and v3.1.1 are what paying it looks like.

v3.0.1 is the consequential one. PythonParser.parse() recorded local indices into its symbols list during the walk, then ran symbols.insert(0, <module>) afterwards whenever the module had a docstring — shifting every recorded index by one. Symbol.parent_id, CallEdge.caller_id and TypeEdge.from_symbol_id all pointed into that list, so one off-by-one corrupted the call graph, the symbol hierarchy and the type graph at once. It never raised and never produced a NULL, because Indexer builds local_to_db from the final symbols list, so a pre-insert index still resolved to a valid row. Just the wrong one. A validity check passes; only a correctness check catches this. The fix reserves symbols[0] before the walk. The release-time measurement was 8,815 of 8,815 index references wrong across 139 source files, which is arithmetic rather than sampling: all 2,179 symbols lived in the 131 docstring-bearing files.

v3.0.1 requires a reindex. v3.0.0's release note said upgrading from v2.12.0 needed no reindex and no migration — true of the schema, false of the graph. The graph the old parser wrote is wrong on disk and no code path corrects it in place. If you upgraded from v2.12.0 or v3.0.0 and did not reindex, your call graph is still wrong: run trelix index ./my-repo again. v3.1.0 and v3.1.1 need no reindex and no schema change.

v3.1.0 is a rendering-correctness release, and its best proof needs no attacker at all: src/trelix/indexing/parser/extractors/rust.py:1035 contains re.sub(r"^//[/!]?\s?", ...), which Rich reads as an unmatched closing tag, so trelix ask against trelix's own repository died with MarkupError instead of showing results. The silent mode is worse: a balanced-looking pair is swallowed and the command exits 0 having dropped characters. A directory named deep[ then raised the same MarkupError from inside the except block meant to skip one file, discarding the whole index run and hiding its own cause.

The machine-readable side needed the opposite fix: _print_json() leaves the payload byte-identical and disables the renderer instead. soft_wrap=True is the part that mattered, because Rich hard-wraps at console width and a wrap landing inside a JSON string injects a newline that json.loads rejects — which the existing short-payload contract tests were too short to catch. Under FORCE_COLOR=1, routinely set in CI, trelix graph --json | jq produced garbage at exit 0 because the spinner wrote to stdout. 51 new regression tests, all but five demonstrated failing against v3.0.1 — the exceptions are negative and byte-identity controls that must pass on both sides — and the markup tests each assert the payload's literal characters appear in the output rather than merely that nothing raised.

v3.1.1's headline is a safety guarantee that was never true. From v2.x through v3.1.0, SECURITY.md said trelix "does not follow symlinks outside the repo boundary." FileWalker had no symlink handling at all: _iter_files used entry.is_dir()/entry.is_file(), both of which follow, and rel_path is computed on the unresolved path, so an out-of-tree file was indexed and reported as sitting inside the repo. TRELIX_WALKER_FOLLOW_SYMLINKS=false now makes the boundary real by comparing resolved paths on both sides, because Path.is_relative_to is lexical and an unresolved comparison would let repo/link -> /etc straight through. It is opt-in for a non-security reason: confining by default would silently drop files from any repository that symlinks to vendored directories.

The unit suite sits at 2,457 collected on this branch, from 2,341 at v3.0.0 and 2,445 at v3.1.0. One fact about that span is more telling than the count: src/trelix/audit/ and src/trelix/auth/ have not been touched since the commit that introduced them. The integrity core and the token verifier needed zero source changes across three releases; the code that renders them needed three. And the audit verify guard that started this article shipped with a test that only ever passed a directory — a path SQLite genuinely cannot open — so it exercised the branch that already worked. Covering the working half of a two-branch guard buys confidence rather than earning it, which is exactly what three releases of looking for the other halves were for.

What the 3.x line is for

If you are deciding whether to build on trelix, the useful summary is not the feature list. It is that the 3.x line has a verifiable trail with a CI-gateable exit code, a real OIDC resource-server gate with a forged-token test rather than a config assertion, a context packer whose quality floor is provable by reading it, and an editor integration built entirely out of the MCP surface. Each is off until you turn it on, and each has its boundaries written next to the code that draws them: tamper-evident and HTTP-only, authentication without authorization, lexical compression by default, one agent session across two chat threads.

pip install 'trelix[serve,sso]' at 3.0.0 or later gets you the audit and SSO surface. v3.1.1 is cut and dated but not yet tagged or published; when it lands it will change nothing at default settings. The one instruction that carries real consequences is the oldest in the span: if you came from v2.12.0 or v3.0.0, reindex.

Trelix v2.11.0: Jira and Linear Now Live Inside the Code Graph

SAI RAM — Sun, 02 Aug 2026 10:13:29 +0000

I've had the same conversation with trelix a hundred times: "what does _verify_auth do." It answers instantly, correctly, and completely uselessly for the question I actually had, which was why does this function exist. Git blame gets you partway there — a commit message, maybe a PR description if you're lucky. But the real answer, the actual requirement, almost always lives somewhere else entirely: a Jira ticket, a Linear issue, a bug report. Code and "the work" have always lived in two systems that never talk to each other. trelix v2.11.0, shipped 2026-08-02, is the release where that stops being true.

The headline feature is two new connectors — Jira and Linear — joining the Xray and TestRail connectors trelix already had for test-management platforms, with all four now sharing one interface. Sync a project's tickets and by the time the command returns, those tickets are already wired into the code graph and already influencing search ranking. Not a CSV dump, not a webhook you have to build your own consumer for. Real typed integrations, with each platform's actual authentication quirks handled correctly, sitting behind one shared contract.

One interface, four connectors

Every connector — JiraConnector, LinearConnector, XrayConnector, TestRailConnector — implements the same ArtifactSource abstract base class (src/trelix/indexing/connectors/base.py). Two methods: validate_config() and fetch(). That's it.

validate_config() exists specifically so a missing credential fails loud and fast. Look at what JiraConnector.validate_config() actually checks:

missing = [
    name
    for name, val in (
        ("TRELIX_JIRA_BASE_URL", self._config.base_url),
        ("TRELIX_JIRA_EMAIL", self._config.email),
        ("TRELIX_JIRA_API_TOKEN", self._config.api_token),
        ("TRELIX_JIRA_PROJECT_KEY", self._config.project_key),
    )
    if not val
]
if missing:
    raise ValueError(f"JiraConnector is missing required config: {', '.join(missing)}")

If you forgot TRELIX_JIRA_API_TOKEN, you get told that directly, before a single HTTP request goes out — not a 401 three requests deep into a paginated search, not a silent empty result you have to debug later. fetch() is the other half of the contract: paginate internally, however the target platform does pagination, and hand back a fully materialized list[Artifact]. Callers never see a page token or a cursor.

That uniform shape is the whole reason adding Linear as the fourth ArtifactSource implementation was tractable inside this release cycle instead of a future one. It didn't have to invent its own calling convention — it just had to satisfy the same two methods over a completely different platform reality underneath.

The part that actually matters for what you get out of this, though, is sync(), which lives once on the base class and every connector inherits unchanged:

def sync(
    self,
    db_writer: ArtifactWriter,
    linker: ArtifactLinker | None = None,
) -> ConnectorSyncResult:
    self.validate_config()
    fetched = self.fetch()
    written = 0
    errors = 0
    edges_linked = 0
    for artifact in fetched:
        try:
            db_writer.upsert_artifact(artifact)
            written += 1
        except Exception:
            errors += 1
            continue
        if linker is not None:
            try:
                edges_linked += linker.link_one(artifact.source_ref)
            except Exception:
                pass
    return ConnectorSyncResult(
        artifacts_fetched=len(fetched),
        artifacts_written=written,
        errors=errors,
        edges_linked=edges_linked,
    )

Read that loop carefully: when a linker is supplied, every successfully-written artifact gets passed through linker.link_one() in the same call. Not queued for a later batch job. Immediately, synchronously, before sync() returns. Run trelix connector sync ./repo jira and the output isn't "downloaded 84 tickets into a table" — it's Synced jira: fetched 84, wrote 84, errors 0, linked 79 edge(s). Those 79 edges already exist in generic_edges and are already visible to PageRank the moment the process exits. There's a standalone trelix link-artifacts command if you want a manual full re-link pass — useful after a --no-link bulk sync, or after a schema change surfaces new matches against artifacts you already synced — but it's an option, not a requirement. The default path gives you a fully linked graph in one command.

Jira and Linear: same contract, opposite platforms

This is the part I find genuinely interesting to build, because Jira and Linear satisfy the identical ArtifactSource contract while being about as different as two ticket-tracker APIs can be.

Jira Cloud is REST, authenticated with HTTP Basic — email plus API token, Jira's own recommended scheme for this kind of integration, no OAuth dance required. Pagination is a cursor token (nextPageToken), not offset/limit. The interesting wrinkle is the description field: Jira Cloud's v3 API returns description as Atlassian Document Format, a nested JSON tree, on every single ticket, unconditionally — never a plain string. That's not a rare shape you might hit on a rich-text ticket; it's the only shape. So the connector ships a real renderer, _adf_to_text(), that walks the tree — paragraphs, headings, bullet and ordered lists, code blocks, inline code, blockquotes, panels, expand sections, rules, links, embedded cards — and produces plain text good enough for ArtifactLinker's regex matching downstream. It's not a full ADF renderer (no tables, no user-mention name resolution), but it covers everything actually observed on a live Jira Cloud site.

Linear is a different animal entirely. There's no REST surface at all — it's GraphQL-only, a single POST endpoint (https://api.linear.app/graphql), and there's no official Python SDK, only a TypeScript one. Pagination is Relay-style: first/after arguments, pageInfo { hasNextPage, endCursor }. The loop has to terminate on hasNextPage, not on a short page — Relay pagination can legitimately hand back fewer results than requested while hasNextPage is still true, so checking len(results) < page_size the way TestRail and Xray do would end the sync early and silently drop issues.

The auth detail is worth calling out by name, because it's the exact kind of thing that breaks silently if you're not paying attention: Linear's personal API key goes into the Authorization header with no Bearer prefix. Just the raw key. Every other connector in this codebase uses Bearer <token> — Xray's connector literally does f"Bearer {self._token}" — so copy-pasting that pattern for Linear produces a request that authenticates against nothing and fails with a 401 whose message doesn't obviously point at the missing prefix. The module docstring in linear.py calls this out explicitly as "the one detail most likely to get corrected back to Bearer by someone skimming Xray's connector — don't."

Linear also runs a real query-complexity budget: 10,000 points per request, where a connection multiplies its children's cost by its own first argument. This connector's field selection costs roughly 7.7 points per issue, so at page_size=100 that's about 1 + 770 ≈ 771 points — comfortably under the cap, with real headroom for schema drift. And Linear signals rate-limiting differently from everything else in this codebase: HTTP 400, not 429, carrying a GraphQL body error with extensions.code == "RATELIMITED", and no Retry-After header at all. Reset timing has to come from reading X-RateLimit-Requests-Reset/X-RateLimit-Complexity-Reset response headers directly, so _fetch_issues_page() runs its own small, connector-local retry loop for exactly this case rather than teaching the shared retry contract a Linear-specific response shape.

What actually makes a synced ticket useful

Downloading a ticket into a table doesn't do anything by itself. The piece that turns a synced artifact into something that changes search results is ArtifactLinker (src/trelix/indexing/artifact_linker.py), and it's worth walking through because the design has a real precision problem baked into it that's easy to get wrong.

Stage one is regex extraction: _IDENTIFIER_RE finds identifier-shaped tokens in an artifact's title and body — function/class/variable-looking strings — and checks each one, casefolded, against an index of every symbol name and qualified name in the codebase. A hit becomes a GenericEdge with edge_kind="references_artifact" and weight=1.0 — a real graph edge connecting that ticket directly to that code symbol.

The obvious failure mode here is that plenty of function names are also ordinary English words. A ticket titled "the test suite failed to run and update the process" would, on pure token overlap, spuriously link to functions literally named run, test, update, and process — none of which have anything to do with that ticket. ArtifactLinker handles this with _COMMON_WORD_STOPLIST, roughly 40 common English/programming words (get, set, run, test, process, update, data, file, main, and so on) excluded specifically from the bare-name side of the index. It's a real precision guard, not an afterthought, and it's scoped carefully: it only gates the bare name key, not qualified_name, so a genuinely qualified match like auth.run still links correctly. Only the literal stoplisted string on its own gets excluded.

Stage two is an opt-in embedding-similarity fallback, gated behind ArtifactLinkerConfig.embedding_fallback_enabled, and it only runs for artifacts where stage one found zero matches. Those fallback edges get weight=0.5 — deliberately half the weight of a regex hit — so a fuzzy semantic match never outranks an exact identifier match for PageRank purposes. Exact reference wins; embedding similarity is a second-chance net, not a peer signal.

Xray and TestRail: proving the abstraction generalizes

If the ArtifactSource contract only worked for issue trackers, I'd be less confident in it. Xray and TestRail are both test-management platforms, and they slot into the exact same shape.

TestRail is the simpler of the two: HTTP Basic auth (username plus API key), offset/limit pagination capped at TestRail's own 250-per-page ceiling. Xray Cloud is odder — its tests are Jira issues under the hood, but test-specific content (steps, expected results) has no Jira-native equivalent, so a single GraphQL getTests query pulls both the Jira fields and the Xray-specific step data in one call, with no separate REST round-trip needed. Auth is its own thing too: a client_id/client_secret pair, distinct from a personal Jira API token, exchanged via POST /api/v2/authenticate for a short-lived bearer JWT attached to every subsequent request. Both connectors write artifact_kind="test_case" records, and both flow through the exact same sync() → ArtifactLinker path as Jira and Linear's tickets. Same registry lookup, same CLI surface, same auto-link behavior. The abstraction holds.

The CLI is identical across all four: trelix connector sync ./repo <jira|testrail|xray|linear>, with a --link/--no-link flag defaulting to on, resolved through one factory — get_artifact_source("jira"|"testrail"|"xray"|"linear") in registry.py — that mirrors the same match-statement pattern trelix already uses for embedder and vector-store selection.

The plumbing underneath

None of the above would mean much if a sync silently swallowed failures or nobody could tell why a request hung. This release also landed a unified retry/backoff contract (core/retry.py, built on tenacity) shared across every LLM backend, embedder, and connector — full-jitter exponential backoff on 429s and 5xx responses, honoring a server's Retry-After header when present. Structured JSON logging shipped alongside it, with trace-ID correlation when OpenTelemetry is enabled. And Personalized PageRank landed as an opt-in flag (TRELIX_RETRIEVAL_PAGERANK_PERSONALIZATION) that concentrates teleport mass on ticket-linked symbols instead of spreading it uniformly — its own docs flag a real interaction risk: combined with pagerank_boost_enabled on a repo with few ticket-linked symbols, it can invert the ranking get_top_central_symbols() produces. Opt-in for a reason.

None of this was validated against mocks alone, either. Live-testing the Jira connector against a real production Jira Cloud site turned up two real bugs — ADF descriptions silently rendering as empty bodies on every ticket, and bad credentials silently reporting success because /rest/api/3/search/jql returns an identical empty result set for a bad token, no token, and a genuinely empty project. Both got fixed this cycle: a real _adf_to_text() renderer for the first, and a pre-flight auth check against /rest/api/3/myself for the second, since that endpoint correctly returns a real 401 where the search endpoint doesn't.

What this actually unlocks

The full unit suite sits at 1,909 passing tests as of this release, up from 1,643 at v2.9.0 — not because coverage was thin before, but because four connectors, a linker, a retry contract, and Personalized PageRank all needed their own test surface.

But the number that matters isn't the test count. It's that trelix can now sit in front of a question like "why does this function exist" and actually answer it, because the ticket that requested the change, or the bug report that prompted the fix, is sitting in the graph one edge away from the symbol itself. Ask "what's supposed to happen when I touch this code" and the answer isn't just call-graph traversal anymore — it's the test case that verifies it, linked in through the same mechanism, the same weight-1.0 edge, the same ArtifactLinker pass. Before this release, trelix understood code. Now it understands why the code is there.

trelix v2.7 to v2.9: The Release Where the Pipeline Itself Became the Product

SAI RAM — Fri, 24 Jul 2026 16:44:32 +0000

On 2026-07-09 I shipped trelix v2.7.0. The architecture felt done — seven retrieval legs, a knowledge graph, an agentic loop. Then I opened the GitHub Release page and counted two binary assets where there should have been three. release.yml built the macOS and Linux PyInstaller binaries both as dist/trelix, and softprops/action-gh-release uploads assets by basename — two files with the same name collide into one asset, with no metadata telling you which OS survived. I genuinely could not tell, from the published release, whether the surviving binary was macOS or Linux. That's not a bug in trelix's retrieval logic; it's a bug in the thing that puts trelix in front of users, and it shipped because nobody checked the page after the workflow went green.

That bug is why this article exists. v2.7.1 through v2.9.0 — six releases over two weeks — spend a surprising amount of energy on parts of the project that aren't retrieval quality at all: the release pipeline, the concurrency model, a language-parser migration, the deployment story, and a VS Code extension and GitHub App with real, embarrassing problems. I'm grouping by theme instead of walking the changelog top to bottom.

Shipping infra is its own product surface, and it broke

The binary collision got fixed in v2.7.1 (2026-07-10) by renaming each binary uniquely before upload. Fresh eyes on the pipeline turned up company: the PR-time CI workflow had never built a Linux binary, even though the release workflow built one at tag time. I added the missing matrix entry and a verify step.

The most embarrassing find: trelix-mcp's own test suite had never run in CI. It let a real regression sit undetected — a test asserted the MCP server had "exactly 6 tools" when it had actually registered 8 since two subscription tools shipped in v2.5.0. A wrong-but-passing test is worse than no test. I wired the tests into CI and fixed the assertion.

Smaller mistakes, same release: three companion packages' dependency floors on trelix core had been bumped in v2.7.0 on the assumption they used new APIs. None did, so I reverted them. The changelog had also rotted: an entry was defined twice with conflicting URLs, silently resolving to the last definition, so the first link was dead. Rebuilt the footer from the actual git tags.

Five concurrency bugs, found only once I wrote real stress tests

v2.7.2 (2026-07-12) reads like the payoff of finally taking concurrency seriously instead of assuming check_same_thread=False meant "safe." It shipped real scale features — Qdrant Cloud readiness, incremental per-symbol embedding on partial re-index, an opt-in parallel BM25 read pool — but the five bugs stress-testing surfaced matter more.

A TOCTOU race in the sparse embedder's lazy-load checked whether the model was loaded before acquiring its lock, so two racing threads could both see "not loaded" and both start loading; fixed with double-checked locking. An MCP stdout write race let concurrent notification writes interleave partial JSON-RPC lines, corrupting a client's output; fixed with a lock around the write-and-flush pair. And unbounded subscription-registry growth let a misbehaving client grow the registry forever, fixed with a max-subscriber cap and a TTL sweep.

The one I find genuinely unsettling in hindsight: silent foreign-key corruption on partial re-index. The parent-symbol, call-callee, and type-edge columns are all set to null on delete, which sounds safe until deleting a changed symbol's old row silently nulls those links on every row that referenced it — including unchanged rows with nothing to do with the edit. No error, just quietly severed graph edges accumulating as files change. I added snapshot-and-repoint helpers so the indexer captures stale links before the delete and repoints them.

And the BM25 lock was incomplete even after I thought I'd handled it. The shared database connection is opened with check_same_thread=False, which I'd treated as a green light for concurrent use — it isn't; that flag disables SQLite's thread-affinity check, not concurrent-statement safety. Grep, sparse, and vector legs all hydrate through that connection from sibling worker threads. I added a real lock everywhere it's touched from a worker thread, verified with a 60-thread by 10-iteration by 3-leg stress test and zero errors — the only way I'd trust this, given the previous fix had looked equally trustworthy.

Same release: a Qdrant client API migration to keep pace with an upstream deprecation, pinned so a future major bump can't break it again. One honest non-win: Windows ARM64 binaries were briefly added to the build matrices, then reverted — two core dependencies publish no wheel for that target. Linux ARM64 shipped; Windows ARM64 didn't.

v2.7.3 was a pure documentation release. A full README audit fixed 15+ factual bugs: wrong env var names, fabricated pip extras, a broken Homebrew tap, a config value that crashes on use, wrong REST method and table names. The architecture diagram got redrawn to show all 7 retrieval legs instead of 3. I backfilled the changelog's empty [2.2.0] entry, which had shipped 5 real features — agentic ReAct loop, data-flow analysis, taint analysis, sparse+dense hybrid retrieval, multi-granularity indexing — never documented anywhere. The README shrank by roughly a third by consolidating duplicated API sections into pointers. None of it changed behavior; it changed whether docs matched reality.

Becoming genuinely deployable, not just runnable

v2.9.0 is where trelix stopped being "clone it, pip install, run the CLI" and became something you could put in front of other services. Four pieces went in, and the order was the point.

Typed REST API response models came first — every route now declares a real Pydantic model instead of an untyped object in the OpenAPI schema, groundwork for the next item. Cursor pagination on the search endpoint came second, the one deliberately narrow breaking change in an otherwise additive release: the endpoint now returns a results-plus-cursor envelope instead of a bare list, matching the MCP tool's existing pagination contract, done so the new SDK wouldn't lock in against the old shape.

The TypeScript SDK came third — a hand-written HTTP client with generated types covering every route, with /ask's streaming response getting its own async generator because token-plus-terminator-plus-error-frame semantics don't fit the same shape as everything else. Fourth, OpenTelemetry tracing — opt-in, off by default, emitting one span per retrieval leg plus spans for each pipeline stage. Worth explaining: OTel's context propagation doesn't cross a thread-pool boundary automatically, and trelix's parallel sub-query execution runs inside exactly that kind of pool, so naive instrumentation produces disconnected, orphaned spans. The fix wraps the traced function to carry the current context across the pool boundary — the docs note the underlying span-naming conventions are still "Development," not "Stable."

The deployment story rounds out with an official multi-arch Docker image in two variants — a slim, API-embedder-only build and a -local build bundling the offline embedding stack — running as a non-root user, with the entrypoint overriding the CLI's loopback-only default to listen on all interfaces (a loopback bind is a silent dead-end in a container unless overridden). Plus a Helm chart modeling the server's real behavior: every route re-derives its config from the request's own repo parameter, so one Deployment is already multi-repo-capable — but the chart's persistent volume is a shared data directory across every repo you serve through it, called out loudly in the docs. Ingress defaults to disabled: the server ships with zero auth middleware, and I'd rather state that directly than bury it.

Building this surfaced the same documentation-rot pattern again: docs referenced an embedder env var that was actually a silent no-op, plus a nonexistent CLI flag and Docker examples with the wrong port — docs drift silently unless something forces someone to run the commands in them.

Python 3.13, and how one dependency swap turned into six bugs

The Python 3.13 item sounds like the least interesting line in the whole changelog: bump the minimum version, done. The actual blocker was tree-sitter-languages, abandoned upstream with no wheels for the new interpreter. The fix was a single swap to the actively-maintained tree-sitter-language-pack, behind trelix's one chokepoint for grammar loading — that's the whole diff at the chokepoint. It is not the whole blast radius: the new library exposes different AST node names and shapes, which propagated silently into nearly every per-language extractor, because each had been written against the old grammar's node vocabulary without anyone realizing how much was implicit knowledge rather than a documented contract.

Six separate bugs came out of this: C#'s grammar name changed, silently tolerated before this release only because the old library accepted both names; Kotlin's extractor had to be rewritten entirely after its field-lookup API stopped returning anything, breaking class, interface, enum, function, and property extraction at once; Python's docstring extraction broke because the new grammar drops a wrapper node; Go's interface-method node was renamed; TypeScript's interface-body node got a new name; and C#'s using alias imports lost their wrapper node.

Every one of these was catchable only because per-language extractor tests already existed before the migration — without them, most would have shipped as silent failures. A seventh casualty I almost missed: the PyInstaller build spec still imported the retired package, breaking every binary build with a flat import error, because that build runs in its own workflow, outside the test suite and linters. Dropped the stale entries, added the new package as a hidden import, verified with a local build plus a smoke test.

One behavior change worth knowing: grammar loading is now network-on-first-use with local caching, not bundled in the wheel — fine on a laptop, mildly alarming for an air-gapped job. A prefetch function warms the cache during image builds; CI runs it automatically now.

The VS Code extension, the GitHub App, and an admission I'm not going to soften

The search command became a debounced (250ms) search-as-you-type picker with live snippet preview, instead of the old one-shot input-then-static-list flow. The preview runs through a virtual document provider that gets real syntax highlighting for free by keeping the real file's extension on its virtual URI. The state machine lives in its own testable class, verified with a fake-timer harness proving the debounce actually debounces.

The security fix here is direct: the ask-panel's Webview interpolated the raw, unescaped LLM answer string straight into HTML, with no content-security policy and no script restrictions. A crafted or adversarial answer — not a hypothetical when you're piping retrieval results through a model — could execute arbitrary script inside the Webview's context. Fixed with HTML-escaping plus disabled scripts and an explicit deny-all CSP; it's an XSS vulnerability, and it's fixed now. Separately: search results had been silently mis-parsed the entire time the extension existed — it read the wrong field names off each result, confirmed against the actual server source, so those fields were always empty strings and clicking a result opened a broken, empty file URI.

And then the line I'm quoting close to verbatim because paraphrasing softens it: the trelix Code Review Check has never posted a single real annotation since the workflow shipped. Status messages ran unconditionally to stdout even in JSON output mode, and combined with the workflow's output redirect, the annotation-posting step's JSON parse had been throwing on every run, silently swallowed since the day this workflow shipped. Every PR ever reviewed by this Check got nothing. Fixed by routing status output to stderr and narrowing the redirect to stdout only. Even after that fix, the mapping logic still wouldn't have worked — wrong response keys, lowercase severity strings compared against real uppercase values. New regression tests were verified against the pre-fix code first — most failed with the exact parse error this bug produces — before trusting they passed for the right reason. The GitHub App also reached GA-readiness: real installation-token minting behind an expiry-aware cache, webhook signature verification via constant-time comparison, a request-body size cap matching GitHub's own limit, and a subprocess timeout on the review shell-out.

Multi-repo federation, and the security fix found before it hit anyone

v2.8.0 and v2.8.1 shipped the same day, 2026-07-20, and the second was a direct response to auditing the first before letting it sit. v2.8.0 exposed the existing federation infrastructure to MCP clients through four new tools, and added a CLI command apparently missing despite the underlying method already existing. Persistent agent memory landed for the agentic loop too — a follow-up call can now resume a prior conversation with full context, sessions auto-evicting after a week of inactivity.

Building this surfaced two real, previously invisible bugs: federated search had silently lost repo provenance in an earlier refactor, so the search command's repo column had been blank with no test catching it; and a per-repo weighting setting — settable, stored, documented — had never actually been forwarded into the fusion math, silently doing nothing since it was added.

v2.8.1 is where the real security finding lives. All four federation MCP tools passed a caller-supplied config path straight into the registry's load/save calls with zero validation, meaning an MCP client — including a prompt-injected agent, exactly the threat model MCP has to take seriously — could point registry I/O at an arbitrary filesystem path. I found this in a pre-push audit of v2.8.0, before it reached anyone running the released version. The fix confines the path to one of two known-safe directories via a proper containment check, not a naive string-prefix check, which would also incorrectly match a similarly-named sibling directory. Same release: repo-count and fan-out caps so a runaway add-repo loop can't scale every search linearly against an unbounded repo count, plus a pagination fix for a per-repo candidate pool that had been widening as the cursor grew, letting later pages get fused from a differently-shaped pool than earlier ones.

Tombstone v1.3-v1.4: Resilience Was the Easy Layer

SAI RAM — Thu, 09 Jul 2026 19:57:31 +0000

I removed four || true statements from a GitHub Actions workflow on July 9th and watched CI go red in four different ways within the same run. Not one failure. Four — a Python pytest install that had never actually finished, a ruff lint violation nobody had looked at, a Ruby require path that resolved to nothing, and a Java Gradle wrapper that didn't exist in the repo. All four had been "passing" for who knows how long, because the test steps were configured to succeed no matter what came back.

That's the theme of this release window. v1.2 was about making the running system survive failure — retries, circuit breakers, idempotency keys, DLQs. This one is about making the layers above the running system — the Helm chart, the SDKs, the GitOps pipeline, the CI config — tell the truth about their own state. Resilience isn't a feature you ship, it's a property you discover you're missing.

The Helm chart was only deploying two of five services

Tombstone has five application services — flag-api, gateway, evaluator, intelligence, marketplace — plus the operator. Until v1.3.0, the Helm chart (infra/helm/flagmind) only had Deployment templates for two of them. This wasn't a secret; it was written down in COMPATIBILITY.md under a section literally titled "Known Gap." Run helm install in a fresh cluster and you got flag-api and gateway, nothing else. Anyone deploying evaluator, intelligence, or marketplace was hand-rolling manifests or copy-pasting the two existing templates and hoping the env vars lined up.

v1.3.0 closes that gap: deployment-evaluator.yaml, deployment-intelligence.yaml, and deployment-marketplace.yaml now ship in the chart. evaluator gets an optional HPA behind evaluator.autoscaling.*. intelligence exposes IS_PRIMARY_REGION straight from values.yaml, which matters for the multi-region setup where secondary regions run in read-only relay mode.

Here's the footgun worth flagging, because it isn't obvious until it bites you: every one of those templates uses tombstone.selectorLabels in spec.selector.matchLabels, not the separate tombstone.labels helper, which includes a version label. It's tempting to use the one "labels" helper everywhere for consistency. Don't. Kubernetes Deployment selectors are immutable once the object exists. If your selector helper includes a label that changes on every release — like a chart version — the second helm upgrade you ever run will fail outright, because the new selector no longer matches the old one. tombstone.selectorLabels is a narrower, stable subset — name and component, nothing that changes across releases — specifically so matchLabels never drifts. One line in a template, and it's the difference between a chart that upgrades cleanly forever and one that works exactly once.

SDK parity is a correctness bug, not a feature request

The TypeScript SDK (@tombstone/core) has had the full 5-step evaluation pipeline since v2.0 — preliminary checks, prerequisites, individual targeting, rule matching, fallthrough — with the complete operator set and semver comparisons. The Python SDK didn't. It had basic targeting but was missing large chunks of the operator surface and all of the prerequisite evaluation logic. That's not a nice-to-have gap: a flag with a semver_gte rule or a prerequisite on another flag could evaluate to true in a Node service and silently fall through to the default in a Python service, evaluating the same flag against the same user. Two SDKs, two answers, same input — the kind of bug that looks like a targeting mistake in the dashboard when it's actually a parity gap in the client.

v1.3.0 closes it. packages/sdks/flagmind-python/tombstone/matching.py now implements the full operator set — eq/neq/in/nin/contains/startsWith/endsWith, the four numeric comparisons, and all five semver operators (semver_gt/gte/lt/lte/eq) — plus date_before/date_after. The semver comparison is hand-rolled: a _padded_version() helper left-pads each numeric segment to 5 characters and appends a ~ sentinel for 3-part releases, so 1.0.0-beta sorts below 1.0.0 using pure string comparison. It's the same GrowthBook paddedVersionString() pattern the TypeScript SDK already used, with zero new runtime dependencies — no semver package, no extra install footprint.

Prerequisite evaluation was the other half. evaluation.py now threads an evaluation_cache: dict[str, bool] through the recursive prerequisite check, so a flag with three prerequisites sharing a common ancestor doesn't re-evaluate that ancestor three times. Circular chains are rejected via a _seen_keys tracking set rather than recursing forever. The SDK also distinguishes two failure modes with dedicated exception types: InconclusiveMatchError means a targeting condition couldn't be evaluated locally — missing attribute, type mismatch — and the caller should move on to the next rule. RequiresServerEvaluation means the evaluation genuinely needs data the local cache doesn't have, and the client falls back to a REST round-trip instead of silently returning a wrong default. Conflating the two used to mean callers couldn't tell "skip this rule" apart from "call the server."

Making the API impossible to not find

The previous article's two worst bugs — the Slack kill switch sending environment as a query param when the handler read it from the JSON body, and the four-eyes approval routes that were built, reviewed, and merged but never registered in flag-api/cmd/main.go — share a root cause that has nothing to do with the code itself. Nobody had an easy way to look at what flag-api actually exposed versus what the proto files said it should expose. The OpenAPI spec existed; nobody was looking at it, because there was nowhere convenient to look.

v1.3.0 adds a Redoc explorer at GET /api/v1/docs, embedded via go-redoc rather than pulled from a CDN — it reads the existing grpc-gateway OpenAPI spec at /api/v1/openapi.json, so there's no second source of truth to keep in sync. The implementation detail that made this take longer than expected: the plan referenced a chi adapter for go-redoc that doesn't exist in any published version — the library only ships gin/fiber/echo adapters. The fix was to call goredoc.Redoc{SpecPath: specURL}.Body() directly for pre-rendered HTML and wrap it in a plain http.HandlerFunc. Small feature, but it's the direct answer to how those two bugs happened in the first place: the API surface wasn't something anyone could casually glance at.

GitOps: ordering discipline, then a second controller on purpose

v1.4.1 brought Flux CD v2.3+ into gitops/, structured as three layered Kustomizations — infrastructure, apps, flags — each declaring dependsOn against the one before it. The infrastructure layer also sets healthChecks against the tombstone-operator HelmRelease, which matters more than it sounds: dependsOn alone only blocks applying a Kustomization until the upstream has been applied — it does not wait for the upstream to actually be healthy. Without healthChecks, Flux considers the operator's CRDs "done" the moment the HelmRelease manifest hits the API server, before the CRDs are actually established, and the apps layer — which deploys FeatureFlag CRs via the flagmind chart — could start reconciling against CRDs that technically exist but aren't ready yet. Pairing dependsOn with healthChecks on the upstream is what actually guarantees ordering — this is exactly the kind of GitOps bug that works fine in every test until the day the operator pod is slow to come up.

The more interesting decision is v1.4.2's addition of Argo CD v2.11 — not as a Flux replacement, as a second controller with a deliberately split job. Flux keeps infrastructure: the operator's CRDs, image update automation across the tombstone container images. Argo CD takes over the flagmind chart plus the FeatureFlag and FlagPolicy custom resources themselves. The reason to split: FeatureFlag CRs have a rolloutPct field that the intelligence service's ML rollout recommendations mutate live in the cluster, independent of Git. A naive GitOps setup would see that mutation as drift and revert it on the next reconcile — the platform's own ML-driven rollout logic fought and undone by its own deployment tooling every sync interval.

The fix lives in gitops/clusters/production/argocd/apps.yaml: an ignoreDifferences block excluding /spec/environments/production/rolloutPct and /spec/environments/staging/rolloutPct from diff detection, paired with RespectIgnoreDifferences=true in syncOptions. Both are required — ignoreDifferences alone only suppresses the OutOfSync display; without the sync-options flag, Argo CD still overwrites the field back to the Git value on every sync. Miss either half and the ML rollout percentage gets silently stomped on a fixed interval, a nasty class of bug to chase down because nothing in the intelligence service's logs would look wrong.

Argo CD also needed Lua health checks for Tombstone's own CRDs — FeatureFlag (Pending to Progressing, Synced to Healthy, Error to Degraded) and FlagPolicy (Compliant to Healthy, Violation to Degraded) — because its generic health rollup doesn't understand custom CRD status fields out of the box. Without them, the root Application would just show every FeatureFlag as permanently "Unknown" rather than reflecting real reconciliation state.

Blast radius, but as a deployment gate now

This is the part I like best, because it closes a loop back to the v1.0 launch: blast-radius scoring — the LOW/MEDIUM/HIGH/BLOCKED classification the evaluator computes for flag changes — now gates Kubernetes deployments, not just flag rollouts. v1.4.1 adds an Argo Rollouts AnalysisTemplate named tombstone-blast-radius that polls GET /api/v1/blast-radius?flag_key=<key> on the evaluator service during a canary step. gitops/apps/production/tombstone/rollout-analysis.yaml wires this into flag-api's own Rollout: step to 20% traffic, run the analysis template, promote to 100% only if the result is LOW or MEDIUM, abort immediately on HIGH or BLOCKED (failureLimit: 1, so the first bad reading stops it, no averaging across a window). The same scoring engine that decides whether a flag change is safe to ship now also decides whether a code deployment is safe to ship — the same signal doing double duty at two layers of the stack.

Argo CD Notifications routes sync-failed events into the existing marketplace Slack endpoint (marketplace.tombstone.svc:8086/api/v1/marketplace/slack/actions) rather than standing up a second webhook — one less integration surface to maintain.

Removing the safety net

Back to where I started. The || true removal in .github/workflows/ci.yml (commit 9c9bff9) touched test steps across the Python intelligence service, the TypeScript SDK build, the Python SDK, the Ruby SDK, and the Java SDK — every one configured to report green regardless of outcome. The very next commit (031d041) is titled, accurately, "fix pre-existing test failures surfaced by removing || true," and it fixes four of them:

The Python SDK's pytest step installed mmh3 and the package itself but never installed pytest — the runner didn't exist in the environment, so python -m pytest had presumably been failing at the shell level the whole time.
ruff check . --ignore F401 in the intelligence service flagged unused local variables (now_ts, model) a real lint gate should have caught the moment they were written.
packages/sdks/flagmind-ruby/lib/tombstone.rb didn't exist at all — spec files require "tombstone" but the gem's actual entry point is flagmind.rb, a leftover from the gem's rename. I added a one-line alias file.
The Java SDK's step ran ./gradlew test, but no gradlew wrapper is committed to the repo — the step was failing to even start.

Fixing the Java one turned into its own small saga across four more commits, because each fix uncovered the next problem: swapping to gradle/actions/setup-gradle failed because the pinned SHA wasn't resolvable; falling back to apt-get install gradle got Gradle installed, but the Ubuntu-runner package is old enough that it chokes on useJUnitPlatform(); installing Gradle 8.7 directly from services.gradle.org fixed that but exposed that build.gradle declared sourceCompatibility = JavaVersion.VERSION_21, an enum literal the older Gradle parser doesn't accept, which needed to become the plain string '21'. Even after all that, the Java tests still fail, but for a real reason: the source files declare package io.tombstone.* while living under io/flagmind/ directories, the same rename leftover that broke the Ruby require. That's now continue-on-error: true with a comment pointing at v1.5.0, not a silent || true — the failure is visible in the CI UI and tracked, instead of invisible and untracked.

Four bugs weren't introduced in this release. They'd been there for a while, hiding under a shell operator that made "it ran" indistinguishable from "it passed." The same commit mirrors a fix v1.2.1 made earlier, adding pytest-asyncio back for a similar reason — a test suite that can fail silently isn't testing the thing it claims to test, it's testing that the CI runner can execute a command.

Supply chain and the honest caveat

Alongside the CI honesty pass, all remaining GitHub Actions workflow files got pinned to immutable commit SHAs instead of mutable tags like @v4 — nine files in one commit (bfb167f), closing out the supply-chain hardening that earlier workflows (flux-bootstrap.yml) had already established as the convention. Two other release blockers landed in the same commit: an ImageUpdateAutomation resource still on the wrong beta API version, and a structural fix to the production rollouts kustomization that was silently missing a resource reference.

Everything above — the layered Flux Kustomizations, the Argo CD split, the blast-radius AnalysisTemplate — is validated against a local k3d test cluster: every kustomize build passes, Flux bootstraps cleanly, Argo CD installs and reconciles. What it is not yet validated against is the actual production target, Oracle Cloud Kubernetes, blocked on an Oracle Cloud account signup that hasn't happened yet. The operator Helm chart is already published to ghcr.io/sairam0424/charts/tombstone-operator at v0.1.0, ready for the day the cluster exists. Better to say that plainly than let "GitOps shipped" imply more than k3d has actually proven.

The throughline across v1.2 through v1.4 is the same lesson at three different altitudes. v1.2 was runtime resilience — the system surviving its own dependencies failing. v1.3 was correctness at the API and SDK boundary — two clients agreeing on what a flag evaluates to. v1.4 is deployment and CI honesty — the pipeline that ships the system telling the truth about its own state, in the right order, without silently eating failures. None of these layers were broken in an obvious way. They were all quietly returning something other than the truth, and the only way to find that out was to stop letting them.

trelix v1.0 to v2.7: When "It Works" Meets "It Scales"

SAI RAM — Thu, 09 Jul 2026 19:39:19 +0000

Twelve days after I shipped trelix v1.0.0, I was staring at a RetrievalConfig object with two conflicting sets of values and no idea which one was actually running. I'd built AdaptiveRouter to accept a retrieval_config parameter so callers could override the environment-variable defaults programmatically. Except it didn't. The constructor took the parameter, and then quietly ignored it and built its own instance from env vars anyway. Nobody had wired the plumbing from Retriever through QueryPlanner down to AdaptiveRouter.__init__. It's the kind of bug that doesn't throw — it just makes your carefully-set config a decoy.

That fix landed in v2.7.0, PR #55, thirteen days and seven minor releases after launch. In between, trelix went from "search my repo well" to something closer to a platform: a knowledge graph, seven fused retrieval legs, an agentic loop, federated multi-repo search, and a GitHub Actions bot that reviews your PRs. Here's what actually shipped, grouped by what it was trying to solve rather than by version number.

From flat search to a knowledge graph

v1.0 already had hybrid BM25 + vector + call-graph search. What it didn't have was any notion of the codebase as a system — which files cluster into modules, which symbols sit at the center of the import graph, which concepts a human would use to describe an architecture. v2.0.0 (2026-06-28) and v2.1.0 (2026-06-30) fixed that.

The new trelix/graph/ module builds a CodeGraph as a NetworkX MultiDiGraph, unifying call, import, and type edges into one traversable structure. On top of that, Louvain community detection clusters the graph into architectural modules — run trelix graph ./repo and you get the top communities, not just a flat symbol list. ConceptExtractor layers an LLM on top of symbol batches to name those communities in plain English, and it's built to fail quietly: any extraction error returns [] rather than crashing the pipeline. GraphVisualizer.export_html() renders the whole thing as an interactive Pyvis HTML page with community coloring, gated behind pip install trelix[knowledge-graph] so the base install doesn't inherit the dependency weight.

Graph search became a first-class retrieval leg — graph_search_enabled=True runs a CodeGraph BFS as a fourth leg after RRF fusion — and pagerank_boost_enabled uses import-graph centrality to boost symbols that sit at architectural chokepoints. None of this is static: GraphUpdater.update_file() is wired into trelix watch, so the graph and its communities update incrementally as files change, instead of requiring a full rebuild.

This came with the release's one deliberate breaking change: trelix graph — which used to mean "show me callers and callees of this symbol" — got renamed to trelix call-graph. The name trelix graph now means "build the knowledge graph." I made the call that a growing surface area needed the more intuitive name reserved for the bigger feature, and documented the rename explicitly in the changelog rather than let people discover it by trial and error.

Seven legs, one fusion function

v1.0 had three retrieval legs. By v2.2.0 it had seven, and every one of them is grounded in a specific paper rather than a hunch.

The fourth leg was the graph BFS above. The fifth is RAPTOR-style (arXiv:2401.18059) file-level summarization — file_summary_leg_enabled, gated behind TRELIX_FILE_SUMMARIES_ENABLED=true at index time — which lets trelix answer "explain this codebase" questions that no symbol-level chunk could answer alone. The sixth is HyDE (arXiv:2212.10496): instead of embedding your raw natural-language query, hyde_fallback_enabled generates a synthetic code snippet and embeds that, closing the semantic gap between "how do I validate a JWT" and the actual token-validation code. The seventh is multi-query expansion, which decomposes one query into N variants and RRF-fuses the independent retrievals for broader recall.

Layered over all seven is FLARE (arXiv:2305.06983) — a confidence-gated re-retrieval loop that watches synthesis output for uncertainty phrases and triggers another retrieval pass when it finds them, rather than committing to a possibly-wrong first answer.

None of this is worth shipping without a way to measure whether it's actually better, so v2.1.0 added a CoIR-format eval harness (ACL 2025, arXiv:2407.02883) — trelix eval --golden <file> reports nDCG@10, Recall@10, and MRR, implemented as pure-Python trelix.eval.ndcg with zero pandas dependency. Every query now also writes a row to a query_telemetry SQLite table — latency, intent classification, result count — surfaced through trelix telemetry.

Here's the detail that actually matters: in v2.1.0, MultiQueryExpander existed as a class but nothing called it. It took until v2.3.0 (2026-07-02) for it to get wired into _retrieve_standard, and even then I had to be careful about one specific line — variants[1:] is used, not variants[:], so the original query never runs twice through the fusion. It's a one-character difference between "seven legs" and "seven legs, one of them redundant."

Teaching it to act, not just retrieve

v2.2.0 (2026-07-01) shipped across four parallel feature branches — PRs #29 through #32, merged via release PR #33 — and it's the release where trelix stopped being purely a retrieval system.

The agentic loop (trelix/agent/) is CodeAct-style ReAct: instead of one retrieval-then-synthesize pass, the agent can decide it needs another lookup, run it, and fold the result back into its reasoning before answering. Alongside it, trelix/analysis/taint.py and defuse.py added real data-flow and taint analysis — tracing how a value flows from a source to a sink across function boundaries, which is a different kind of question than "what code is semantically similar to this query."

The other two branches were retrieval-quality work: SPLADE-Code sparse retrieval (trelix/embedder/sparse.py, trelix/store/sparse_store.py) gives trelix a learned sparse representation to sit alongside BM25 and dense vectors, and multi-granularity indexing (trelix/indexing/multi_granularity.py, MGS3-style) means the index isn't forced to choose one chunk size — function-level, class-level, and file-level granularities can all be retrieved against.

Production hardening: guards built before the bugs happened

The most interesting engineering in this window isn't a feature — it's a class of failure I preempted instead of debugging in production. DimensionGuard, added at Retriever.__init__ in v2.3.0, checks embedding provider and dimension at startup and raises DimensionMismatchError with the exact recovery command (trelix migrate-vectors --reset) if they don't match what's on disk. Without it, switching from an Azure embedder (3072-dim) to a local model (384-dim) doesn't error — it silently returns wrong results, because cosine similarity between mismatched-dimension vectors still computes a number, just not a meaningful one. That's the worst kind of bug: no stack trace, no crash, just quietly bad answers. v2.5.0 (2026-07-06) extended the same guard to FileWatcher.__init__, so a provider mismatch fails fast at watch startup instead of at query time, days later, when nobody remembers which embedder was configured when.

The MCP surface grew up in the same window. v2.3.0 added MCP Resources (trelix://index/stats, trelix://repo/{path}/manifest, trelix://repo/{path}/symbols/{name}) and MCP Prompts (trelix-search, trelix-explain, trelix-blast-radius) — reusable, application-addressable primitives instead of one-off tool calls. v2.5.0 went further: trelix-mcp now advertises resources.subscribe=True, and a thread-safe SubscriptionRegistry tracks who's watching which URI, so notify_file_changed() can fire notifications/resources/updated the moment watchfiles detects a change. That notification path had a gap of its own until v2.7.0 Phase 1 (PR #55): FileWatcher._do_reindex only fired the notification on hash-identical skips, never on an actual successful re-index — the one case where a subscriber genuinely needed to know. The same release added idx_files_rel_path as an index on files.rel_path, eliminating a full table scan that GraphUpdater.update_file() was silently paying on every single file-change event.

Federation, and finding your code's twin

DiffReviewer and trelix review <repo> [--diff] [--base] [--head] (v2.3.0) turned trelix into something you point at a diff, not just a repo — it parses git diffs via DiffParser.from_git() (subprocess with shell=False, no injection surface), turns each hunk into a retrieval query, and generates review comments that are crash-safe by construction: DiffReviewer.review() never raises. v2.4.0 (2026-07-04) connected that to GitHub directly — GitHubPRClient plus trelix review --pr owner/repo#N --post-comments, authenticating only via GITHUB_TOKEN, handling all seven GitHub file-status values, and warning past a 3,000-file truncation limit. v2.7.0 Phase 3 (PR #57) closed the loop with .github/workflows/trelix-review.yml, which runs that same review command on every PR and posts findings as GitHub Check annotations with file and line references — continue-on-error: true on the indexing step, because CI runners without local embedding models shouldn't fail the whole workflow. The same phase shipped workspace-vscode/, a VS Code extension scaffold with trelix.search and trelix.ask commands, talking to the existing trelix-mcp package over stdio — no new backend, just a new front door.

The other axis of growth was going multi-repo. RepoRegistry (v2.3.0) manages ~/.config/trelix/repos.json, and FederatedRetriever fans a query out across every registered repo in parallel, RRF-merges the results, and dedupes by (file_path, symbol_id) — crash-safe, returning [] if every repo fails rather than propagating one bad repo's exception. v2.4.0 added a SHA-256-keyed TTL cache (cache_ttl=120.0) tuned for the query patterns of an actual debugging session, where you ask variations of the same question five times in ten minutes. v2.7.0 Phase 2 (PR #56) pushed federation further with make_scip_symbol_id() — stable, SCIP-style cross-repo symbol IDs, sha256-truncated and pipe-separated so scoped npm packages like @scope/pkg resolve unambiguously — and DiffEmbedder, a CCRep-style (arXiv:2302.03924) before/after body-pair encoder for PR diff hunks. search_similar_diffs() finds historically similar changes via cosine similarity, with a NaN guard and dimension-mismatch protection baked in from day one, because I'd already been burned once by silent dimension corruption.

Scale work, and holding the frontier to a real bar

v2.6.0 (2026-07-08) tackled the two things that get expensive as a codebase grows: recomputing the whole community graph on every file save, and blocking on a full index pass before you can search anything. The DF Louvain frontier heuristic (compute_affected_frontier(), detect_communities_incremental(), arXiv:2404.19634) reprocesses only the seed nodes, their neighbors, and their existing community members — falling back to a full recompute only when the affected frontier exceeds 50% of the graph. TRELIX_INDEXER_STREAMING=true (v2.7.0 Phase 2) makes indexing itself lazy — _iter_files() yields files one at a time into a bounded Queue(maxsize=64), with a try/finally guarantee that the producer sentinel always gets sent even on an exception. It's off by default, and I mean that literally: zero behavior change on the path everyone is actually running.

I also shipped two things I'm not willing to oversell. The XTR late-interaction reranker (NeurIPS 2023, arXiv:2304.01982) is cheaper than ColBERT/PLAID by reusing tokens you already retrieved instead of reloading every document's full token set — that's a genuinely good idea. But it's explicitly marked EXPERIMENTAL in the changelog, it emits a UserWarning on first use, and it has not been benchmarked against CoIR or CoREB on code-specific retrieval. PLAID stays the production-validated default. Same discipline applies to the GroUSE-inspired synthesis harness (arXiv:2409.06595, COLING 2025) — SynthesisEvalHarness scores hallucination, completeness, and faithfulness across seven failure modes, because I'd been leaning on "does GPT-4 think this answer sounds right" as an implicit quality bar, and that correlation is not a substitute for actually checking whether the citations are real.

Test count tells the same story as the changelog: 929 unit tests at the v1.0.0 baseline, 1,467 unit plus 41 MCP tests — 1,508 total — by v2.7.0. pip install "trelix[local]" still gets you a fully offline setup, and the index is still one SQLite file. Growing the surface area from three retrieval legs to seven, plus a knowledge graph, an agentic loop, and federation, didn't require growing the infrastructure footprint at all — every new leg, every graph feature, every federation layer is opt-in behind a config flag that defaults to off. That was a deliberate constraint, not an accident, and it's the one I'm least willing to relax as this keeps growing.

I Built trelix Because I Was Tired of Grepping My Way Through Codebases

SAI RAM — Sun, 05 Jul 2026 11:38:03 +0000

I spent my most of day's on a new team grepping through 80,000 lines of code trying to find where authentication worked.

Four hours. Three teammates interrupted. Twelve dead ends across files I didn't understand. The code was fine — it was well-written, well-organized, reasonably documented. The tooling was the problem. I was using grep to understand something that wasn't a text search problem. Code has structure: call edges, import chains, type hierarchies, AST relationships. Grep ignores all of it.

That day stuck with me. I kept running into the same pattern on different teams, different codebases, different languages. Every time I joined something new or came back to a project after six months away, the first few days were archaeology. Tracing calls manually. Reconstructing context that should have been queryable.

I built trelix to fix this. It's an open-source code intelligence engine that indexes any repository with Tree-sitter, embeds every symbol, and answers natural-language questions using hybrid BM25 + vector + call-graph search. It works offline. No API key needed. Zero infrastructure.

pip install "trelix[local]"
trelix index ./my-repo
trelix ask ./my-repo "how does the authentication middleware work?"

The Problem With Code Search

The tools we have for understanding code — editors, grep, ctags, language servers — were designed for writing code, not for understanding it at scale. They're excellent at navigating to a known destination. They're poor at answering questions like "how does the request lifecycle work end-to-end?" or "what calls this function, and what does that caller depend on?" when you don't already know the answer.

The fundamental limitation of grep is that it treats your codebase as a document corpus. It finds strings. Code isn't a document corpus — it's a graph. Functions call other functions. Modules import other modules. Classes extend other classes. When you ask "how does authentication work?", the answer isn't a file or even a few files. It's a traversal of that graph, starting from a semantic entry point and following edges to collect the relevant context.

Vector search solves part of this — semantic similarity gets you closer to the right files without knowing the exact tokens. But pure vector search misses structural relationships. It doesn't know that UserRepository.get_by_token() is always called by AuthMiddleware.verify() which is called by every protected route handler. That's call-graph knowledge, not embedding knowledge.

trelix uses both.

What trelix Does

trelix indexes any repository into a single SQLite file (.trelix/index.db) and then answers questions about it.

The index contains: every symbol extracted via Tree-sitter (functions, classes, methods, their bodies and line spans); call edges and import edges between symbols and files; a hybrid search index combining sqlite-vec HNSW vectors with FTS5 BM25; and since v2.1.0, a Code Property Graph that unifies all of the above into a traversable NetworkX graph.

A query like trelix ask ./repo "explain how authentication works" goes through a 3-tier adaptive router:

Tier 1 (Direct) — for simple factual patterns like "what is X" or "define X", trelix skips retrieval entirely and answers from the LLM directly. No unnecessary round-trips.

Tier 2 (8-intent) — for most code queries, it classifies the intent into one of eight categories (symbol_lookup, feature_flow, dependency_map, blast_radius, etc.) and runs the appropriate retrieval strategy.

Tier 3 (Multi-step) — for complex queries like "walk me through the request lifecycle end-to-end", it decomposes the question into 2-3 sub-queries, runs each independently, and merges the results.

Results from all active retrieval legs are fused via Reciprocal Rank Fusion (k=60) before being assembled into the context window for LLM synthesis.

How It Actually Works

The indexing pipeline runs in four phases:

Phase 1 (Parse) — Tree-sitter walks every file and extracts symbols with their source, line spans, and AST structure. Runs in parallel via ThreadPoolExecutor.

Phase 2 (Write) — Symbols and chunks are written to SQLite. Cross-file parent_id relationships are resolved.

Phase 3 (Embed) — Every chunk is embedded asynchronously in batches of 4 concurrent API calls. With the local provider (sentence-transformers, no API key), this runs entirely offline.

Phase 4 (Resolve) — Cross-file call edges are resolved with a 3-priority strategy: qualified name first, then type_hint+name, then name-only fallback. This gives about 40% fewer false-positive cross-file edges compared to name-only matching.

The result is a single .trelix/index.db file that contains everything: vectors, BM25, call graph, import graph, symbols, file hashes for incremental updates.

Zero Infrastructure, Full Power

This was a deliberate design decision and one I keep coming back to.

Most code intelligence tools require running a vector database, a relational database, and often a separate API server. That's a lot of infrastructure to maintain for what is fundamentally a local developer tool. trelix's default is a single SQLite file using sqlite-vec for HNSW vector search and FTS5 for BM25. Zero external infrastructure. Works on a laptop with no internet connection.

When you need to scale: LanceDB backend for 100k+ chunks (3-5× faster vector insert on ARM/Apple Silicon), Qdrant for 500k+ chunk deployments with multi-repo shared collections. But the default handles most codebases and most developers will never need to switch.

# Default (sqlite) — up to ~100k chunks
trelix index ./my-repo

# LanceDB — 100k+ chunks
TRELIX_STORE_BACKEND=lance trelix index ./my-repo

# Qdrant — 500k+ chunks
TRELIX_STORE_BACKEND=qdrant trelix index ./my-repo

Beast Mode: Seven Retrieval Legs

The default setup (BM25 + vector + grep + call graph) handles most questions well. But trelix has five additional retrieval legs that you can enable when you need higher recall or more sophisticated query handling:

Leg 5: File-summary semantic search — RAPTOR-style (arXiv:2401.18059). At index time, trelix generates LLM summaries of every file and embeds those summaries separately. This surface is especially good for "explain this codebase" or "what files deal with payment processing?" queries — questions where the answer is at the file level, not the symbol level.

Leg 6: SPLADE-Code — sparse+dense hybrid via learned sparse retrieval. SPLADE encodes queries into sparse high-dimensional token vectors, expanding vocabulary beyond exact matches in a way that complements both BM25 and dense vector search.

Leg 7: Multi-granularity — indexes code at block AND statement level simultaneously. Some queries are better answered by a full function body; others are better answered by a single statement. Having both granularities in the index improves recall on precise questions.

Plus query-side enhancements: HyDE (generates a hypothetical code answer as the ANN query vector, improving recall on abstract questions), FLARE (confidence-gated re-retrieval — when synthesis spans show uncertainty, trelix re-queries before finalizing the answer), and since v2.2.0, an agentic ReAct loop that does multi-turn retrieve→observe→re-retrieve with self-correction.

# Enable everything
TRELIX_RETRIEVAL_AGENTIC=true \
TRELIX_GRAPH_SEARCH_ENABLED=true \
TRELIX_RETRIEVAL_FILE_SUMMARY_LEG=true \
TRELIX_RETRIEVAL_HYDE_FALLBACK=true \
TRELIX_RETRIEVAL_FLARE=true \
TRELIX_RETRIEVAL_SPARSE=true \
TRELIX_CHUNKER_MULTI_GRANULARITY=true \
trelix ask ./my-repo "explain the full request lifecycle"

The Features I'm Most Proud Of

GitHub PR review. This is v2.4.0 and it's become one of the most-used features. trelix review --pr owner/repo#42 fetches the PR diff from GitHub, retrieves codebase context for each changed hunk, runs an LLM review, and can post findings back as a single batched review comment with --post-comments. The key insight is that reviewing a diff without understanding the surrounding codebase is like proofreading a sentence you've never read before.

trelix review --pr sairam0424/trelix#42
trelix review --pr sairam0424/trelix#42 --post-comments

Federated search. trelix search-all "query" fans out across all registered repos in parallel via ThreadPoolExecutor and RRF-merges the results. With trelix watch-all, a single watchfiles.awatch() call watches all registered repos simultaneously. The TTL cache on FederatedRetriever gives about 90% hit rate for typical debugging-session query patterns.

trelix federation add api ./services/api
trelix federation add web ./services/web
trelix search-all "JWT validation"
trelix watch-all

MCP integration. One command and trelix is available inside Claude Code, Cursor, Windsurf, and Continue.dev:

pip install trelix-mcp
claude mcp add trelix -- trelix-mcp

Then inside Claude Code: "index my repo at /path/to/repo, then find how authentication works".

What Surprised Me Building This

I expected the hardest part to be the embedding and retrieval architecture. It wasn't. The hardest part was making the system opinionated enough to be useful without being so opinionated that it broke on unusual codebases.

The call-graph resolver was the most representative example. My first version used name-only matching for cross-file call edges — login() in file A calls login() in file B. This produced a dense, noisy graph with maybe 40% false-positive edges. The fix was a 3-priority resolution strategy: try qualified name first (most precise, lowest recall), then type hint + name (moderate precision), then name-only as fallback. That reduced false positives significantly while maintaining recall on codebases that don't have full type annotations.

The other thing that surprised me was how much value came from the structural metadata rather than the semantic embeddings. The call graph, import graph, and type hierarchy are what make trelix's answers qualitatively different from a vector search over code files. Semantic similarity gets you to the right neighborhood. Graph traversal gets you to the right answer.

What I'm Still Uncertain About

The 3-tier query router works well for the queries I've tested it on. I'm less confident about it on very large codebases (millions of lines) where the graph becomes expensive to traverse. The current implementation caps BFS depth at 2, which is usually right but occasionally misses important connections. I'm still figuring out the right heuristics for adaptive depth.

I'm also still calibrating the GraphRAG map-reduce threshold. The current default (activate at >20 results or >8k tokens) is conservative. For some query types it activates too eagerly; for others, not eagerly enough. This is the main retrieval parameter I'm watching in practice.

Try It

# Offline — no API key
pip install "trelix[local]"
trelix index ./your-repo
trelix ask ./your-repo "how does your main feature work?"

# With LLM synthesis
pip install trelix
export OPENAI_API_KEY=sk-...
trelix ask ./your-repo "explain the request lifecycle end-to-end"

# MCP in Claude Code
pip install trelix-mcp
claude mcp add trelix -- trelix-mcp

# Review a PR
trelix review --pr owner/repo#42 --post-comments

Everything is MIT licensed, on PyPI, and at github.com/sairam0424/trelix. The full documentation is in the repo README including the beast-mode activation block if you want all seven retrieval legs at once.

What's the longest you've spent trying to understand a piece of code you didn't write? I've had 4-hour archaeology sessions on codebases with good documentation. I'd like to know how much of that time you think was the code being genuinely complex versus the tooling failing you.

Tombstone vs Unleash vs Flagsmith vs Flipt vs GrowthBook: Feature Flag Platforms Compared (2026)

SAI RAM — Sun, 28 Jun 2026 06:27:57 +0000

I've been building with feature flags for a long time, and I've used most of the major tools. This comparison is written from the perspective of someone who eventually built their own — not because the others are bad, but because none of them answered the question I kept asking: which of my 5,000 active flags is responsible for what's happening in production right now?

Most comparisons you'll find online are either outdated, vendor-written, or only compare the "how do I deliver a flag value?" dimension. That dimension matters. But at scale, it's not the dimension that keeps you up at night.

The Comparison Matrix

Capability	Tombstone	Unleash	Flagsmith	Flipt	GrowthBook
Flag CRUD + targeting rules	✅	✅	✅	✅	✅
Real-time streaming to SDKs	✅ SSE	✅ SSE	✅ SSE	✅ SSE	✅ SSE
Approval workflows (four-eyes)	✅	✅ paid	✅ paid	❌	❌
GitOps YAML sync	✅	❌	❌	✅	❌
Circuit-breaker auto-rollback	✅	❌	❌	❌	❌
Blast radius pre-check	✅	❌	❌	❌	❌
Causal dependency graph	✅	❌	❌	❌	❌
"What Changed?" incident query	✅	❌	❌	❌	❌
Tombstoning (permanent key archival)	✅	❌	partial	❌	❌
ML rollout recommendations	✅	❌	❌	❌	❌
CUPED variance reduction	✅	❌	❌	❌	partial
mSPRT sequential testing	✅	❌	❌	❌	❌
Merkle-chained audit trail	✅	❌	❌	❌	❌
OpenFeature compliance	✅	✅	✅	✅	✅
Kubernetes operator + CRDs	✅	partial	❌	✅	❌
WASM zero-dependency eval engine	✅	❌	❌	❌	❌
Self-hosted, fully open-source	✅ MIT	✅	✅	✅	✅
Cloud managed option	planned v1.1	✅	✅	✅	✅
OPA policy-as-code RBAC	✅	partial	❌	❌	❌
Polyglot SDK support	6 languages	5+	5+	5+	3

When to Choose Each Tool

Unleash is the best-established open-source flag platform. It has a large community, solid documentation, and a hosted cloud option. I'd reach for Unleash when I want a proven, well-documented system with minimal operational risk — especially for teams that are new to feature flags. Its weakness: it's purely a delivery system. It doesn't tell you anything about what your flags are doing to production.

Flagsmith has a great developer experience and a clean SDK. The hosted cloud option is reasonably priced. I've found it particularly good for teams that want feature flags and remote config in one system. Like Unleash, it stops at delivery — there's no safety layer.

Flipt is the choice for teams that care deeply about GitOps. It's the only other tool that takes YAML-as-code seriously, and it has good Kubernetes integration. Flipt's evaluation model is clean and well-documented. I'd choose Flipt over Tombstone for teams that specifically need a GitOps-native flag system without the operational overhead of Tombstone's additional services.

GrowthBook is genuinely excellent for experimentation. If your primary use case is A/B testing and you don't need the production safety features, GrowthBook's stats engine (Bayesian + frequentist) is the most sophisticated in the OSS space. Its feature flag delivery is functional but secondary.

Tombstone is the right choice when you're running 500+ flags across multiple services and production reliability matters as much as experiment velocity. The circuit-breaker auto-rollback, blast-radius scoring, and causal incident correlation are not features you'll find anywhere else in OSS. The tradeoff is complexity — 8 services is a real operational commitment.

The Capabilities That Don't Exist Elsewhere

Circuit-breaker auto-rollback is the one I care most about. The implementation: SDKs report evaluation events with flag key + outcome to the evaluator service. Per-flag error rates are tracked in rolling windows in Redis (5% errors over 100 requests in 10 seconds = trip). When the breaker trips, an OnTrip callback executes, disabling the flag and writing to the audit log. Recovery: 5-minute HALF_OPEN window. The whole cycle — flag changes, errors spike, flag disabled — can happen in under 30 seconds without a human involved.

I think about the Knight Capital incident a lot when I work on this. 45 minutes, $440M, entirely from a feature flag that should have been disabled. A circuit breaker with a 10-second window trips in 30 seconds. That's the delta.

Blast radius scoring answers the question before you change anything. Four tiers: BLOCKED (>50% traffic + >5% historical error delta), HIGH (>25% traffic or 5+ dependent flags), MEDIUM, LOW. BLOCKED changes require a 10-character minimum justification — long enough to be intentional, short enough to not be bureaucratic. I found that threshold through trial and error.

"What Changed?" incident correlation is the thing that saves the first 10 minutes of every production incident. Given a timestamp, it queries the causal dependency graph and returns flags that changed in the preceding window, ranked by blast radius. One API call instead of 20 minutes of log archaeology.

Quick Start Comparison

Tombstone:

git clone https://github.com/sairam0424/Tombstone
cp infra/.env.example infra/.env
make dev  # all 8 services + dashboard at localhost:3000

Unleash:

git clone https://github.com/Unleash/unleash
docker compose up -d  # dashboard at localhost:4242

Flagsmith:

git clone https://github.com/Flagsmith/flagsmith
docker compose up  # dashboard at localhost:8000

Flipt:

docker run -p 8080:8080 flipt/flipt:latest
# dashboard at localhost:8080

GrowthBook:

git clone https://github.com/growthbook/growthbook
docker compose up -d  # dashboard at localhost:3000

Honest Caveats About Tombstone

The stack is complex. 8 services, PostgreSQL, Redis, and Kafka is not something you want to manage if you're a team of 3. For small teams, start with Unleash or Flipt.

The ML layer needs data. Thompson Sampling requires ≥50 observations per flag before making rollout recommendations. For new flags, the intelligence service says "insufficient data" and steps aside. That's the right behavior, but it means you won't see ML recommendations for the first few days.

The intelligence service bundles a 400MB embedding model (BAAI/bge-m3) for NLP flag search. First build takes 3–5 minutes. Every build after that is seconds.

Cloud hosting is planned for v1.1. Right now, Tombstone is self-hosted only.

Conclusion

If you're evaluating feature flag platforms in 2026, the decision comes down to what you're optimizing for:

Production reliability at scale → Tombstone
Proven, well-documented OSS → Unleash
Developer experience + remote config → Flagsmith
GitOps-native → Flipt
Experimentation-first → GrowthBook

The honest summary: Tombstone adds capabilities that don't exist anywhere else in open source, but it asks for more operationally. The question is whether the circuit-breaker and blast-radius features are worth the additional complexity for your team. For teams managing thousands of flags across multiple services, I'd say yes. For everyone else, Unleash or Flipt will serve you well.

Tombstone is MIT licensed and self-hosted. GitHub: https://github.com/sairam0424/Tombstone

I Built a Self-Hosted Feature Flag Platform That Auto-Rolls Back Bad Flags — Here's Why

SAI RAM — Sat, 27 Jun 2026 20:08:00 +0000

After reading about the Knight Capital incident one too many times, I got frustrated with every feature flag tool I'd used. They all answer the same question well: "what's the value of this flag?" None of them answer the question I actually need during a 3am incident: "which of my 5,000 flags is causing this?"

So I built Tombstone.

What it does differently

Every OSS flag platform I've seen is a delivery system. Flag state in, evaluation result out. Tombstone adds a safety layer on top of that:

Circuit-breaker auto-rollback. When a flag causes >5% errors over 100 requests in a 10-second window, it disables automatically. No pager. No runbook. MTTR goes from "however long it takes your on-call to wake up" to ~30 seconds.

Blast radius scoring. Before you change a flag, you see its tier: BLOCKED, HIGH, MEDIUM, or LOW. BLOCKED changes (flags touching >50% of traffic with a poor error history) require a written justification before you can proceed. Deliberate friction for high-risk changes.

"What Changed?" incident query. Given an incident timestamp, returns flags that changed in the preceding window ranked by blast radius. One API call instead of 20 minutes of log archaeology.

Quick start

git clone https://github.com/sairam0424/Tombstone
cp infra/.env.example infra/.env  # zero changes needed
make dev                           # dashboard at localhost:3000

All 8 services start in one command. PostgreSQL, Redis, Kafka, everything included.

The honest tradeoffs

It's a complex stack. 8 services isn't right for every team. For small teams, Unleash or Flipt are better choices. Tombstone earns its complexity when you're managing 500+ flags across multiple services and production reliability is a first-class concern.

The ML rollout recommendations (Thompson Sampling + LinUCB contextual bandit) need ~50 observations per flag before they kick in. New flags get "insufficient data" and the system steps aside — the right behavior, but worth knowing.

Stack: Go (flag-api, gateway, evaluator) + Python 3.12 (intelligence/ML) + TypeScript (SDKs, dashboard). MIT licensed.

GitHub: https://github.com/sairam0424/Tombstone

How I Built Tombstone: A Self-Hosted Feature Flag Intelligence Platform to Prevent the Next Knight Capital

SAI RAM — Sat, 27 Jun 2026 13:46:34 +0000

The 2am Dashboard That Started Everything

It was 2:47am when I opened our feature flag dashboard and realized I had no idea what had changed. P99 latency on our payments service had spiked to 4.2 seconds about 20 minutes earlier, and the on-call playbook said to check recent flag changes first. We had LaunchDarkly for flag evaluation, Jira for change tickets, and a Notion doc that was supposed to track active experiment flags. The Notion doc hadn't been touched in six weeks. The Slack channel that was nominally our audit log had 340 unread messages from the previous day's deploy sprint.

The actual question I needed to answer — which flags changed in the last 30 minutes across all services — had no answer. Not a slow answer, not an approximate answer. No answer.

That's a knowledge management failure, not an infrastructure failure. We had three systems that each held a partial slice of production state and shared exactly zero causal model between them. LaunchDarkly knew flag evaluation counts. Jira knew someone opened a ticket. Notion knew whatever someone remembered to type. None of them knew that a flag flip in service A at 2:31am might be causally related to the latency spike in service B at 2:33am.

Knight Capital in 2012 is the canonical proof that this failure mode is existentially dangerous. They lost $440 million in 45 minutes — not because their trading system was buggy, but because the POWER_PHLX flag key was reactivated on only one of eight servers during a deployment. That reactivation woke up dormant RLP (Repurposing Liquidity Provider) code that had been dead for eight years. No system tracked key provenance. No system blocked reuse of a key that had previously controlled live trading logic. The blast radius was uncontained because the organization treated flag state as ephemeral configuration rather than durable history.

Atlassian published a post about hitting 4,000+ active feature flags at scale. At that volume, on-call engineers can no longer reason about which flags are safe to flip during an active incident. The flags become load-bearing in ways nobody documented, and the institutional knowledge of which key controls what behavior lives entirely in the heads of engineers who may not be on the incident bridge.

We were at 200+ flags across 12 services when I hit my 2am wall. Nowhere near Atlassian scale, but already past the threshold where a Notion doc and a Slack channel constitutes an audit trail.

Tombstone is the system I wish had existed that night — and understanding why the existing toolchain fundamentally can't be patched into something safe requires starting with how flags actually fail in production.

Tombstoning: The Single Most Important Safety Property

Every feature flag platform I've used treats flag deletion as a soft operation — mark the row inactive, maybe hide it from the UI, but leave the key string available for reuse. This is catastrophically wrong, and Knight Capital proves it.

In 2003, Knight deprecated their RLP (Repurpose Liquidity Provider) functionality. The POWER_PHLX flag key that controlled it sat dormant. Nine years later, during a routine deployment, eight of nine servers had SMARS code installed; the ninth still ran the old code path gated by that same key. When the flag was "reactivated," the ninth server interpreted it with 2003 semantics while the rest used 2012 semantics. $440 million, 45 minutes. If POWER_PHLX had been tombstoned after RLP deprecation, the 2012 reactivation attempt would have been rejected at the control plane — before a single byte reached a trading server.

The core invariant in Tombstone is this: a flag key is a permanent identifier, not a reusable string. Once archived, a key is cryptographically retired. This isn't a policy; it's a database constraint.

CREATE TABLE tombstones (
    id           UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    flag_key     TEXT NOT NULL,
    archived_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    archived_by  UUID NOT NULL REFERENCES users(id),
    final_audit_entry_id UUID NOT NULL REFERENCES audit_log(id),
    merkle_hash  BYTEA NOT NULL,
    CONSTRAINT uq_tombstones_flag_key UNIQUE (flag_key)
);

-- Enforced via trigger on flags INSERT:
CREATE CONSTRAINT TRIGGER prevent_tombstoned_key_reuse
    AFTER INSERT ON flags
    DEFERRABLE INITIALLY IMMEDIATE
    FOR EACH ROW EXECUTE FUNCTION reject_if_tombstoned();

Any INSERT into flags with a tombstoned key raises a constraint violation at the database layer — the service layer never even makes the decision. This dual enforcement matters: application bugs don't open a gap.

The tombstone record itself is append-only and Merkle-linked to the final audit log entry at time of archival. You cannot update a tombstone row. You cannot delete it. The merkle_hash chains each tombstone to its predecessor, so tampering with the archive history changes the hash and becomes detectable. The operation is designed to be irreversible by construction, not by convention.

The operational consequence is intentional friction. Engineers cannot recycle keys — they must create a new key with a new name for new behavior. I've found this forcing function has secondary benefits: teams start naming flags with lifecycle semantics baked in (checkout_v2_stripe_migration rather than new_checkout), and the tombstone audit trail becomes an accurate historical record of what that key meant, not just what it does now.

This append-only archive is what gives the audit chain its integrity guarantees — which turns out to be load-bearing for incident response.

The Flag Lifecycle: From Draft to Tombstone

Every flag moves through six stages. Understanding the lifecycle is how you avoid the Knight Capital failure mode in the first place:

DRAFT — Flag exists in the database. No users affected. Configure type, description, and safe default here. Test in development.
ACTIVE — Code references the flag. Deployed to production. Flag still disabled (0% rollout) — this is the dark launch phase. Ship the code first, release the feature when you decide.
ROLLING OUT — Flag enabled in production at 1–99%. Real users are seeing the feature. The circuit breaker is watching. Ramp gradually: 1% (30 min) → 10% (1 hour) → 50% (2 hours) → 100%
FULL ROLLOUT — All users at 100%. Flag still in the codebase. Monitor for 7+ days before scheduling cleanup.
CLEANUP — Run ast-rewriter to remove dead code references. Open a PR. The "enabled" branch stays; the else branch is removed.
TOMBSTONED — Flag key permanently archived. Can never be reused. Appears in /tombstones. The Knight Capital failure mode is now impossible for this key.

The gap between FULL ROLLOUT and TOMBSTONED is where most teams fail. The flag hits 100%, the team moves on, and six months later nobody remembers what dark_launch_v2 controls. Tombstone's flag-cleanup domain loop detects flags at 100% for 30+ days and creates a cleanup signal automatically.

Causal Incident Correlation: Turning a 3-Hour Post-Mortem Into a 10-Second Report

The post-mortem ritual I was stuck in before Tombstone looked like this: PagerDuty fires, I acknowledge, then I spend the next hour manually cross-referencing Slack messages, Jira tickets, and a feature flag dashboard that shows current state with no temporal depth. The actual causal flag change might be sitting right there in plain sight — I just had no tooling to surface it.

The query model I built is deliberately simple. On PagerDuty webhook receipt, Tombstone's correlation service scans the append-only audit log for every flag state change within a configurable lookback window (default 30 minutes, tunable per environment). The append-only constraint matters here — I'm not reconstructing state from a mutable table, I'm replaying a ledger. Every write is a new row with an immutable timestamp, actor, flag key, previous value, and new value. The query is a bounded range scan, not a diff computation.

What makes the output useful rather than just noisy is the scoring layer. Raw recency isn't enough — if three flags changed in the same deploy window, listing them alphabetically is useless. I apply exponential recency decay: a change 2 minutes before the alert timestamp scores dramatically higher than one 28 minutes prior. The ranking answers not just what changed but what changed in a way that is temporally suspicious.

The output contract is fixed: top 3 correlation candidates, each containing actor, delta_seconds_before_alert, flag key, previous value, new value, and a pre-signed rollback link that's valid for 15 minutes. The pre-signed link is load-bearing — it means the on-call engineer can execute a rollback without navigating any UI, without elevated permissions at 3am, and without touching the flag's current live state directly.

{
  "candidate_rank": 1,
  "flag_key": "pricing.v2_calculation_engine",
  "actor": "deploy-bot@internal",
  "previous_value": false,
  "new_value": true,
  "delta_seconds_before_alert": 247,
  "rollback_url": "https://tombstone.internal/rollback/pre-signed/abc123"
}

The real-world validation came during a payment error spike. A flag enabling a new pricing calculation had been toggled 4 minutes before the error rate climbed. Tombstone surfaced it as candidate #1. The on-call engineer clicked the rollback link without opening a single log query or Kibana tab. Total time from page to rollback: under 90 seconds.

The failure mode I spent the most time on was false positives from scheduled changes. A cron job toggling a flag at 3am scores high on recency decay but is completely unrelated to an unconnected incident. Tombstone now cross-references a pre-approval registry — any scheduled change with a corresponding approval record gets annotated as "scheduled": true, "pre_approved": true in the correlation output, which visually de-prioritizes it for the on-call engineer without removing it from the candidate list entirely.

Clock skew between services was the other edge case worth addressing explicitly — and it's where the circuit-breaker integration earns its keep.

Circuit Breaker Auto-Rollback: The Safety Net That Fires Before You Ack the Page

The evaluator service runs a sliding window over per-flag error rates — 100 requests minimum sample, 5% error threshold. When a flag's error rate crosses that line, the evaluator doesn't wait for a human. It rolls back to the flag's declared safe_default, writes an audit entry, and hands off to the incident correlation pipeline. The whole sequence completes in under 200ms. By the time PagerDuty has routed the page to your phone, the blast radius is already contained.

This is the core advantage of a kill switch over a traditional deploy rollback:

Kill switch: 10 seconds, zero risk of new bugs.
Deploy rollback: 20+ minutes, CI pipeline required, risk of introducing new bugs in the rollback commit.

Here's what that audit entry looks like when the circuit fires:

{
  "event": "auto_rollback",
  "flag_key": "checkout_flow_v2",
  "triggered_by": "system:evaluator",
  "threshold": { "error_rate": 0.05, "window_requests": 100 },
  "sample_request_id": "req_7f3a92c",
  "crossed_at_request": 104,
  "rolled_back_to": "control",
  "timestamp": "2024-11-14T02:47:33.812Z"
}

That's not just a log line — it's a first-class audit event, Merkle-linked into the same append-only chain as every human-initiated change. The rollback is attributable, reproducible, and queryable. triggered_by: system:evaluator is a real actor in the system, not a null field.

I designed this around a concrete scenario: a checkout_flow_v2 flag rolls to 10% of traffic. Latency spikes 800ms. Error rate crosses 5% at request #104. The evaluator rolls back, the causal correlation pipeline attaches the latency/error timeline to the incident, and the PagerDuty alert arrives with the rollback confirmation and the causal report pre-attached. The on-call engineer reads a complete picture, not a blank canvas.

Blast-radius scoring gates whether that automation is even allowed to fire. A flag scored BLOCKED — one that touches payments, auth, or any dependency marked critical — cannot be auto-rolled back. It requires four-eyes sign-off before the rollback executes. HIGH flags get auto-rollback but with immediate escalation. MEDIUM and LOW roll back silently and generate a low-priority ticket.

The known gap I haven't fully closed: flags controlling background jobs that fail silently. If errors don't surface as HTTP 5xx responses, the sliding window never sees them. A worker that swallows exceptions and logs to nowhere keeps the error rate at zero while the job queue backs up. I've found this is a signal registration problem — the evaluator exposes a health signal API, but it's opt-in, and teams running background jobs rarely think to wire it up until something burns.

That silent failure mode is exactly why blast-radius scoring alone isn't sufficient — it scores the flag's potential impact, but it can't compensate for missing telemetry.

The Architecture: 8 Services, One Causal Model

The core architectural decision I made early was a hard separation between the control plane and the data plane — and I mean hard, not "they talk to different database schemas" hard.

flag-api (:8081) owns all writes. Every flag mutation, every rollout percentage change, every tombstone — nothing lands in the system without going through flag-api. It maintains the append-only Merkle-linked audit log, where each entry is structured as:

{
  "entry_id": "01HX...",
  "payload_hash": "sha256(current_payload)",
  "prev_hash": "sha256(prev_entry.payload_hash + prev_entry.prev_hash)",
  "timestamp": "2024-11-03T02:47:13Z",
  "actor": "svc:gitops-sync",
  "change": { "flag": "dark-launch-v2", "op": "rollout_update", "pct": 15 }
}

Tamper detection is an O(n) chain walk — you rehash each entry against its predecessor. On startup and on every export request, flag-api verifies the full chain. It's not blockchain theater; it's the minimum viable guarantee that a Jira ticket edit didn't quietly retrograde your audit history.

gateway (:8080) owns all reads. SDKs never talk to flag-api. Gateway streams flag state changes via SSE, backed by Redis Streams consumer groups. This is where I diverged from a naïve polling architecture.

With polling, a restarting SDK instance loses the delta between its last poll and reconnect. With consumer groups, each connected SDK instance registers a named consumer in Redis Streams:

XREADGROUP GROUP tombstone-sdks sdk-instance-{uuid}
  COUNT 100 BLOCK 0 STREAMS tombstone:flag-changes >

On reconnect after a rolling deploy, the consumer resumes from its last acknowledged offset — > becomes the last unacknowledged ID. No change is skipped. Flag updates reach SDK in-process caches in under 10 milliseconds under normal load, not because I did anything clever, but because SSE over a persistent connection and Redis Streams at localhost latency are just fast.

evaluator (:8082) is the piece I'm most deliberate about. It sits in the data path conceptually — it observes the evaluation stream — but it is explicitly not in the hot path. Blast-radius scoring and circuit-breaker logic run async against a mirrored evaluation event stream. Flag resolution itself never blocks waiting for the evaluator. When the evaluator detects a threshold breach, it writes a rollback command back through flag-api. The latency budget for flag resolution stays in the microseconds; the evaluator can take 50ms to compute a blast radius score and nothing degrades.

The lifecycle bookends are gitops-sync (:8084) and ast-rewriter (:8085). Flags enter the system as YAML-as-code via Git PRs — gitops-sync watches the repo, validates schema, and calls flag-api on merge. Flags exit via ast-rewriter, which runs dead-code analysis against the TypeScript and Python SDKs' call sites and opens automated PRs to remove stale references. The tombstone mechanism is what makes ast-rewriter trustworthy: a key can't be rewritten out of the codebase while it's still receiving non-zero evaluation traffic.

The remaining three services — the OPA policy enforcer, the MCP server, and the OpenTelemetry collector sidecar — round out the platform. Each declares clear responsibilities within this topology, though the causal correlation model depends most critically on trace propagation being correct end-to-end.

Blast Radius Gate and Four-Eyes Approval: Enforcing Change Discipline at the Control Plane

The evaluator service computes a blast-radius score on every flag write — not just at creation. Touch a targeting rule and the score recalculates immediately based on which service paths evaluate that flag. BLOCKED flags sit on authentication or payment codepaths and require two approvals before any change ships. HIGH flags require one. MEDIUM and LOW are self-serve. The tiers aren't static labels you set once and forget; they're derived from actual evaluation telemetry, so a flag that started as MEDIUM quietly becomes BLOCKED the moment your payment service starts evaluating it.

The four-eyes enforcement lives at the service layer in flag-api, not in the UI. That distinction matters. I've seen too many "approval workflows" that are really just frontend validation — one direct API call and the gate evaporates. In Tombstone, the control plane rejects the activation request if the approval count doesn't meet the threshold for that blast-radius tier, full stop. Self-approval is also rejected at the service layer: the approver's identity is checked against the requester's identity on every activation, so a solo engineer can't route around the requirement by approving their own pending change.

That specific edge case surfaced a real bug. A team tried to push a BLOCKED flag change at 11pm the night before a launch. No second approver was available, so they reached for the break-glass path — a signed token any engineer can generate to override the approval gate. The override works, but it fires an immediate Slack and PagerDuty notification to the on-call lead with the justification string the engineer provided. In this case, the engineering lead reviewed the alert, pulled up the diff, and caught a targeting rule that would have enabled a payment flow for 100% of users instead of the intended 1% canary cohort. The break-glass path is explicitly designed to be used; it's not a trap. But it makes the override visible and synchronous enough that a second set of eyes usually happens anyway.

Scheduled changes are a first-class primitive with cryptographic binding. An engineer authors a change now, a second engineer approves it, and the system executes at a future timestamp. The approval hash covers both the flag payload hash and the scheduled timestamp — if either is modified after approval, the scheduled execution is blocked and the original approver receives a notification. This closes the subtle attack surface where someone approves a change, then quietly updates the payload before it fires.

That binding property turns out to be essential once you start reasoning about the audit log as a causal record rather than a changelog.

The ML Layer: Anomaly Detection, Stale Flag Hygiene, and Contextual Bandit Rollouts

The intelligence service (Python 3.12, :8083) runs a three-model ensemble because no single anomaly detector handles the full range of failure modes I care about.

Z-score handles baseline deviation against a rolling historical window — fast to compute, easy to reason about. But I burned myself on it during a Black Friday load test: evaluation volume spiked 8x on schedule, and Z-score lit up every flag touching the checkout path as anomalous. Seventeen alerts, all noise, exactly when I needed signal. EWMA solved that specific problem. It adapts the baseline dynamically, so a traffic ramp that follows the expected curve stays quiet. Genuine deviations — a flag evaluation rate that diverges from the trend rather than just exceeds a threshold — surface clearly. Z-score still runs; it catches sudden step changes that EWMA's decay factor would smooth over. The third model is Isolation Forest, operating across the multivariate space of correlated flags. A single flag's metrics looking normal doesn't mean the system is healthy — I've seen cases where two interdependent flags each showed marginal anomaly scores that combined into a real incident. Isolation Forest ingests the joint feature vector across all active flags sharing a prerequisite or targeting overlap and scores the ensemble, not the individual.

Stale flag hygiene is a problem that accumulates silently. The intelligence service queries flag evaluation counts over a configurable window (default 30 days). Flags with zero evaluations and no scheduled changes are surfaced as cleanup candidates — but I don't trust that signal alone. Before a flag is marked safe to archive, the ast-rewriter runs a static analysis pass across the codebase, resolving all key references. If checkout_v2_redesign still appears in a dead branch nobody merged or a feature spec file that imports the SDK, it doesn't get flagged for deletion. Only when the AST walk returns zero live references does the UI surface the "safe to archive" indicator.

The rollout recommendation engine uses LinUCB, a contextual bandit that treats rollout percentage as an arm selection problem. Context dimensions are geo, device class, and plan tier. Reward signals are conversion rate, error rate delta, and p95 latency. In production, a flag for a new recommendation algorithm started at 2% globally. The bandit observed that mobile users in the EU cohort were converting at 40% higher rates on the new algorithm — and autonomously recommended accelerating that arm to 15%, while holding the desktop cohort flat pending more data. Static percentage rollouts would have averaged that signal away entirely.

Semantic search rounds out the layer: flag descriptions are embedded at write time via pgvector, so engineers can query "find all flags related to checkout latency" and surface semantically related keys rather than hunting by prefix — which matters more than you'd expect once your flag count crosses a few hundred.

The intelligence service feeds back into the control plane, but that loop introduces its own consistency challenges.

The 5-Step Evaluation Pipeline and Domain Loops

Every flag evaluation in Tombstone passes through the same five-gate pipeline — no shortcuts, no bypasses, even for internal callers.

1. Key existence + tombstone check        → reject unknown or tombstoned keys immediately
2. Prerequisite graph resolution          → recursively evaluate dependencies, depth-first
3. Targeting rule match (context input)   → first-match wins, rules ordered by priority
4. Variation assignment (consistent hash) → stable bucketing via MurmurHash3 on user_id + flag_key
5. Circuit-breaker gate                   → abort and return fallback if breaker is open

The prerequisite graph is the piece that surprised most reviewers when I first proposed it. Flags can declare hard dependencies — flag B requires flag A to be on before B will ever serve anything other than its default variation. Evaluation of B recursively resolves A first. The constraint I care about most is at the write path: cycles are detected via DFS at commit time and the write is rejected with a full cycle path in the error body. A team on our experimentation squad accidentally wired two experiment flags into a mutual dependency — A requires B, B requires A. The flag-api returned a 409 with cycle_path: ["flag_A", "flag_B", "flag_A"] and the write never landed. Without that check, evaluation would spin indefinitely.

The intelligence service runs three persistent domain loops — stale detection, anomaly scanning, and bandit reward collection — completely off the request path. They write recommendations back to the flag-api via authenticated internal POST; the evaluator never touches ML inference directly. This keeps P99 evaluation latency deterministic.

The marketplace service (:8086) is the integration fabric. Slack, Datadog, PagerDuty, OpsGenie, Jira, Linear, and OpenTelemetry adapters are registered plugins. Each declares an event subscription and a payload transform — no hardcoded webhooks. The OpenTelemetry adapter is the one I'd highlight: every evaluation emits a span carrying flag key, matched variation, targeting rule ID, and evaluation latency, which plugs directly into existing Datadog or Honeycomb dashboards with zero custom instrumentation.

That pipeline composability is what makes the next operational challenge — deploying this across environments without configuration drift — the real stress test.

What v2.2.0 Ships, What I Learned, and Why It's Named Tombstone

v2.2.0 (Dashboard v1.0.0) ships all eight services runnable locally with make dev: full Merkle-verified audit trail, causal incident correlation, circuit-breaker rollback, the three-model ML ensemble, and the React dashboard. It's the first stable, self-hosted release — production-ready for deployment today.

The name is operational vocabulary, not branding. A tombstone is what you place on something permanently ended — the flag key is dead, its history is preserved, and nothing else can ever wear its identity. Knight Capital's ghost flag had no tombstone. That's the entire point.

The deepest lesson: the hard problem in feature flag infrastructure isn't evaluation performance — consistent hashing solves that in microseconds. It's knowledge continuity across personnel and time. A flag created in 2019 by an engineer who left in 2021 is still evaluating 50,000 times per day in 2024, and nobody knows what it gates or whether removing it will cause an incident. Tombstone's NLP search and stale detection surface it; the tombstone record preserves the full history permanently after archival.

What I'd change in v2: the Python intelligence service creates a language boundary that complicates deployment. The Z-score and EWMA models belong in the Go evaluator; only the Isolation Forest and contextual bandit justify the Python boundary. That change would collapse the service count from 8 to 6 and eliminate a failure surface that's bitten us twice in production.

Feature Flags at Scale: Designing a Distributed Control System for Production Behavior

SAI RAM — Sat, 20 Jun 2026 18:02:32 +0000

The Counterintuitive Truth: Feature Flags Are Not Config Files

Most engineers first encounter feature flags as a simple abstraction: a key-value lookup that returns true or false. That mental model works fine for a single service handling a few hundred requests per minute. It becomes actively dangerous at scale.

A mature feature flag system isn't a config file with an API wrapper — it's a distributed control plane. The distinction matters architecturally. A control plane manages the real-time behavior of a running system across many nodes simultaneously, with its own consistency guarantees, failure semantics, and propagation latency. That's a fundamentally different design problem than reading a YAML file on startup.

One constraint drives every downstream decision: user traffic must never block on a remote flag service call. If evaluation requires a synchronous RPC, you've coupled your request path to the availability and latency of an external system. Netflix's Archaius library enforces this by evaluating flags entirely in-process against a locally-cached configuration snapshot. A network round-trip per evaluation injects 10–50ms of tail latency at p99 — catastrophic when you're competing on streaming start times measured in hundreds of milliseconds. Google, Meta, and Netflix collectively evaluate flags against millions of requests per second with sub-millisecond overhead. That figure is only achievable through local evaluation backed by an async synchronization layer, not RPC.

The other failure mode engineers underestimate is flag sprawl. Systems accumulate flags the way codebases accumulate dead functions — gradually, then all at once. I've seen services carrying thousands of flags where fewer than 10% were actively managed. The operational weight alone becomes a liability: which flags are safe to remove? Which ones are kill switches for production behavior that no one documented?

Knight Capital's $440M loss in 45 minutes in 2012 remains the canonical cautionary tale. A stale feature flag inadvertently activated dormant trading code during a deployment, and the blast radius was immediate and irreversible. Flag lifecycle management — creation, ownership, expiration — isn't operational housekeeping; it's a correctness property of your system.

Understanding why local evaluation is non-negotiable sets up the architectural pattern that makes it possible: the flag state replication pipeline.

Requirements: What 'Feature Flags at Scale' Actually Demands

The functional surface alone surprises most engineers. A production flag system isn't serving booleans — it's serving typed values (integers, strings, arbitrary JSON), kill switches with hard fail-closed semantics, percentage rollout gates, canary targets scoped to specific infrastructure segments, and versioned snapshots that let you replay what the system believed at a given point in time. Targeting rules compound this quickly: at Uber, a single flag evaluation might need to resolve against user_id, region, device_type, tenant, and experiment_group simultaneously. A naive if-else chain works at 10 rules. At 50, it becomes a maintenance liability. At 200, it's a correctness hazard. You need a rule engine with a well-defined evaluation order, conflict resolution, and deterministic behavior under partial attribute sets.

The non-functional requirements are where the real architecture lives. Sub-millisecond evaluation latency isn't aspirational — it's a hard constraint once flags sit in the hot path of request handling. At millions of evaluations per second, any synchronous network call to a central store is a non-starter. Availability needs to clear 99.99%, which means the evaluation path must degrade gracefully when the control plane is unreachable, either failing closed (deny by default) or failing open (permit by default) based on the flag's declared safety policy. These aren't interchangeable decisions, and conflating them causes incidents.

The consistency model is the architectural insight that most designs get wrong by trying to make it uniform. The control plane — authoring, validation, audit — requires strong consistency. A flag misconfiguration that half your fleet sees and half doesn't is strictly worse than a brief write delay. The data plane, by contrast, intentionally tolerates eventual consistency. Meta's Gatekeeper system operates with a 30–60 second propagation window across its evaluation tier, accepting that staleness is acceptable, but staleness-during-outage is not. Local evaluation against a cached snapshot is the entire point.

Observability isn't an afterthought here — it's a first-class requirement. Flag exposure tracking, per-evaluation audit logs, and rollout telemetry are the mechanism by which you prove a flag change caused a regression rather than merely correlating with one. Without them, rollback decisions are guesswork.

These requirements shape every layer of the system, starting with the data model that carries all of it.

High-Level Architecture: Control Plane vs. Data Plane

Here's the counterintuitive part: a flag system optimized for evaluation speed looks almost nothing like a flag system optimized for safe flag management. Those are fundamentally different problems, and conflating them is the root cause of most flag infrastructure failures I've seen in production.

The solution is a clean separation into two planes with explicitly different contracts.

The Control Plane owns authoring, validation, and rollout orchestration. A flag change flows through a UI or API → a validation engine (targeting rule schema checks, mutual-exclusion guardrails, kill-switch constraints) → a strongly-consistent store — Spanner if you need globally-serialized writes, Postgres if you're regionally scoped — → a distribution service that fans changes out to consumers. This path is slow by design. Write latency of hundreds of milliseconds is acceptable; a misconfigured targeting rule that crashes a canary population is not. The control plane is write-optimized and correctness-prioritized.

The Data Plane is the exact opposite. It's an embedded SDK running inside every service instance — a JVM agent, a Go library, a sidecar — holding a complete in-memory snapshot of all flag configurations. Evaluation is a pure function: deterministic rule engine, no network I/O, no locks on the hot path. At a million evaluations per second, even a 1ms P99 latency on flag lookup is catastrophic. The data plane pays that cost once at startup and on incremental updates, then amortizes it across every request indefinitely.

The distribution service is the bridge. It maintains a persistent watch on the config store — a Postgres LISTEN/NOTIFY channel, a Spanner change stream, or a custom CDC pipeline — and pushes config diffs to registered service caches as changes land. The critical word is pushes.

Pull-based polling is an anti-pattern at scale, and the reasoning is straightforward: flip a high-traffic flag and every service instance's poll timer fires within the same jitter window. You've just created a thundering herd directly on your config store, exactly when the system is under change-induced stress.

This push-from-source architecture is proven at hyperscaler scale in adjacent systems. Envoy's xDS protocol uses an identical model — a management server pushes config diffs to data plane proxies rather than having proxies poll. The Kubernetes controller pattern applies the same principle: controllers watch for state changes and reconcile, rather than continuously re-fetching the entire desired state. Thousands of flag SDK instances refreshing simultaneously after a topology change isn't a hypothetical; it's the default failure mode of naive polling designs.

The consistency requirements across these planes diverge sharply — and that divergence shapes every caching and propagation decision downstream.

Flag Data Model: Beyond the Boolean

The mental model of a flag as a key-value pair breaks down the moment you need to answer: "Which version of this rule is in production, who owns it, and when does it expire?" A production flag is a versioned rule tree — a structured document carrying type metadata, an ordered list of targeting rules with predicates, a default value, ownership metadata, and an expiry timestamp. That last field is chronically undervalued; stale flags accumulate into a slow-moving operational hazard that eventually bites you during an incident.

Rule evaluation is ordered and short-circuits. The canonical sequence is: kill switch overrides first, then explicit targeting rules, then percentage rollout buckets, then the global default. That ordering is load-bearing. Consider a checkout flag from a real e-commerce scenario:

{
  "flag": "new_checkout_v2",
  "version": 14,
  "type": "boolean",
  "owner": "payments-team",
  "expires_at": "2024-09-01T00:00:00Z",
  "rules": [
    { "if": "region == EU", "value": false },
    { "if": "user_percent < 5", "value": true }
  ],
  "default": false
}

The EU rule precedes the rollout rule deliberately — GDPR compliance is a hard override, not a population sample. Reversing that order silently ships non-compliant behavior to a subset of European users.

Targeting predicates support compound expressions: region == EU AND user_tier == premium AND hash(user_id) % 100 < 5. The hash function here isn't an implementation detail — it must be stable and deterministic across services and restarts. A non-deterministic hash means the same user evaluates into different buckets across requests, producing the kind of experience flapping that's nearly impossible to reproduce in staging.

More sophisticated systems take a Zanzibar-influenced approach where rule predicates reference relationship tuples — user is_member_of beta_group — rather than raw attribute values. This decouples group membership from the flag definition itself; adding a user to a beta cohort updates the authorization graph, not the flag document, enabling dynamic targeting without a flag redeployment cycle.

JSON-typed flags deserve special attention. A flag that returns {"timeout_ms": 3000, "retry_count": 2} is no longer feature gating — it's remote configuration. At this point, the flag system's data model starts pulling double duty, and the boundary between "flags" and "dynamic config" dissolves entirely, with real implications for how you think about consistency guarantees.

Flag Evaluation Engine: O(1) on the Hot Path

The counterintuitive part of flag evaluation performance isn't the algorithm — it's when the work happens. Engineers typically assume that fast evaluation means a fast lookup at runtime. The real optimization is eliminating runtime work entirely by front-loading it at cache-load time.

When the SDK receives a flag payload from the data plane, it doesn't store the raw rule list. It pre-compiles it: rules are indexed by flag key into a structure that supports O(1) key lookup, with rule traversal deferred to evaluation time but bounded by rule count, not user count. LaunchDarkly's SDK does exactly this — at initialization, it converts the incoming rule list into a key-indexed map so that every evaluation starts with a single hashtable lookup, followed by linear traversal over a typically small, finite rule set. Evaluation complexity is O(1) amortized across the flag key space; the linear component is a constant you control by limiting rule depth.

Per-request memoization eliminates a second class of waste. In a non-trivial service, a single flag like new_checkout_v2 may be evaluated a dozen times across middleware, service logic, and rendering layers within one request. Without memoization, each call re-traverses the rule tree and re-computes targeting. With it, the first evaluation populates a request-scoped cache keyed on (flag_key, evaluation_context_hash); subsequent calls return the cached variant directly. Twelve evaluations become one rule traversal plus eleven map reads.

Determinism is non-negotiable. Percentage rollouts computed as hash(user_id) % 100 must produce identical results across every service instance and every SDK version deployed simultaneously. I once watched this go wrong in production: at a fintech running a gradual checkout rollout, two SDK versions in parallel deployment used different hash seeds. The result was roughly 3% of users seeing alternating UI states on page refresh — the new checkout one request, the old checkout the next. The bug was invisible in logs until flag exposure tracking revealed that the same user_id was receiving different variant assignments. Diagnosis took three days; the fix was a one-line seed normalization.

Evaluation context is a snapshot. The SDK captures the flag ruleset version at request start. Mid-request flag updates — which happen continuously in a live system — do not mutate in-flight evaluations. Consistency within a request is strict; consistency across requests is eventual.

The evaluation engine's correctness guarantees only hold if the data feeding it stays fresh and coherent, which brings cache invalidation and update propagation into focus.

Distribution Model and Failure Modes

The most expensive mistake I've seen in flag infrastructure is treating distribution as a read-through cache problem. It isn't. At scale, distribution is a consistency problem — and the failure modes from getting it wrong are subtle enough to evade your staging environment entirely.

Push over pull, always. The thundering herd case makes this obvious: when 10,000 service instances poll on a 30-second interval and a flag update lands, you get a coordinated spike against your flag store roughly every polling cycle. But the latency argument is equally compelling — push-based systems propagate changes in seconds; pull-based systems propagate changes in up to one polling interval, which is the wrong answer when that flag is a kill switch. Practically, this means your flag store should maintain persistent connections to subscribers (SSE, gRPC streaming, or WebSocket), pushing diffs on change rather than waiting for clients to ask.

Fail-closed vs. fail-open is a per-flag contract, not a system default. A kill switch for a payment processor that disables a fraud-detection bypass should fail-closed: if the flag store is unreachable, the conservative behavior is to assume the kill switch is active and disable the feature. A UI experiment showing a new checkout button layout should fail-open: the safe default is the existing experience, not a hard failure. This policy belongs in the flag definition itself, not in application code that will inevitably diverge across services.

Version pinning and atomic snapshot application. Applying a partial diff is worse than applying nothing. Consider a coordinated update that activates a kill switch and raises a rate limit ceiling to compensate — applying only the kill switch activation causes a correctness regression. Services should maintain a monotonic version counter and only commit a new snapshot if the full version is received. If a diff is incomplete or arrives out of order, hold the previous version.

The cold start problem deserves specific treatment. A freshly launched instance has no local cache. Two options: block on a synchronous fetch before accepting traffic, or start with hardcoded defaults and accept a divergence window. Envoy xDS makes the correct trade-off for safety-critical config — it blocks listener activation until the initial config push is received, meaning no traffic is served until the full snapshot is loaded. AWS AppConfig takes a complementary approach at the distribution layer itself: config pushes include a bake time window, with CloudWatch alarms monitored during rollout and automatic rollback triggered if error rates spike. That's the right abstraction boundary — rollback logic in the distribution infrastructure, not scattered across application code.

The evaluation engine is only as correct as the snapshot it's working from, which means the consistency guarantees of your distribution layer directly constrain the safety properties of every flag in your system.

Kill Switches, Canaries, and Progressive Rollouts

Kill switches occupy a special tier in the evaluation order — they're evaluated before any targeting predicate runs. The implementation consequence is significant: a kill switch cannot depend on user context, because at the moment you need it, you may not have a valid user object, a working database, or a functioning auth service. It's a boolean override, period. The system checks it first, returns the override value if set, and never touches the targeting rules. This is what makes Uber's surge pricing kill switch work: during a major incident, on-call engineers flip a single flag that disables surge pricing globally within 30 seconds across all regions. That response window is only achievable because evaluation requires no network call — the flag state is resident in every process's local cache, and propagation uses the fan-out push model covered in the previous section. A synchronous network call per evaluation would make a 30-second global rollback physically impossible at their request volume.

Flag-based canaries differ from infrastructure canaries in a subtle but operationally important way: the new code path runs in the same binary as the existing path. There's no separate deployment, no second fleet to drain. Activating a flag canary takes seconds; rolling it back takes the same. The tradeoff is that you can't isolate resource contention between paths, but for pure logic changes it's strictly faster.

The critical implementation detail in percentage rollouts is that the percentage is not random per-request — it's hash(user_id) % 100. This ensures a given user sees a consistent experience across every service instance and across the entire duration of the rollout. Without this, a user mid-checkout could alternate between old and new behavior on sequential requests, producing both bad UX and uninterpretable metrics.

Modern systems go further by coupling rollout percentage to real-time metric feedback. Meta's Gatekeeper ramp feature starts a flag at 0.1% of users and automatically increments by 0.1% every 30 minutes if no metric regression is detected — error rates, p99 latency, business KPIs. If a regression surfaces during a 5% canary window, the system rolls back automatically and pages on-call. A complete 0%→100% ramp can finish overnight with zero engineer involvement.

The automated feedback loop depends on one thing the flag system itself can't provide: a reliable, low-latency signal from your observability stack — which shapes how the control plane and metrics pipeline need to be coupled.

Flag Lifecycle Management: The Failure Mode Nobody Plans For

Flag sprawl is the failure mode that hits you slowly, then all at once. You don't notice the first 500 flags. You barely notice the first 1,000. At 4,000+, Atlassian's engineering team discovered that on-call engineers could no longer reason about which flags were safe to flip during an active incident. Their response: mandatory 90-day expiry on every flag, with automated JIRA ticket creation when expiry approached. The alternative — an on-call rotation paralyzed by combinatorial state uncertainty — was untenable.

The underlying problem is a combinatorial explosion. Ten independent boolean flags produce 1,024 possible system states. Fifty flags produce more states than atoms in the observable universe. You cannot test that. You cannot reason about it under pressure at 2am.

Every flag needs three things enforced by automation, not convention: an owner, a creation timestamp, and an expiry date. Flags without expiry dates are tech debt with a fuse. When the flag reaches 100% rollout, automated tooling should open a PR to remove the call sites — the flag is now dead code that still burns CPU in your evaluation engine and adds cognitive overhead to every engineer who reads that branch.

The Knight Capital incident in 2012 remains a stark reminder of lifecycle failure. The SMARS "Power Peg" flag was never cleaned up after deprecation. A new deployment accidentally reactivated it, routing live orders through dead code. $440 million in losses in 45 minutes.

Flag dependencies compound this risk significantly. If Flag B's rollout assumes Flag A is enabled, that dependency must be explicit in your data model — an implicit dependency discovered during an incident rollback is a production outage waiting to happen. A simple depends_on field in the flag schema, validated at write time, catches these relationships before they become archaeology problems at 3am.

The data model carrying this metadata sets the foundation for the operational tooling that makes cleanup tractable at scale.

Performance Optimizations, Observability, and Big Tech Patterns

The work you do at evaluation time should be close to zero. That's the design principle driving every meaningful performance optimization in mature flag systems.

Rule compilation is where the real work happens. At SDK initialization and at every cache refresh, raw flag rule trees are compiled into optimized decision structures — typically sorted arrays of targeting predicates with precomputed hash ranges and attribute extractors resolved to direct field offsets. A flag that requires parsing a JSON rule on every evaluation is already broken at scale. After compilation, evaluation reduces to a sequential scan of an in-memory structure with no deserialization, no regex compilation, no string splitting. This amortizes all parsing cost once per refresh cycle across every subsequent evaluation per second.

Flag exposure tracking is the observability primitive everything else depends on. Every evaluation should emit a structured event: {flag_key, variant, user_id, user_context_hash, sdk_version, timestamp}. This isn't logging for debugging — it's the foundational data primitive for experiment analysis, regression detection, and audit compliance. Google's flag exposure pipeline feeds directly into ABACUS, their experimentation platform; exposure events are the join key between user actions and flag variants, making causal inference possible without any manual instrumentation at the product layer. Miss an exposure event, and your experiment data is uninterpretable.

The convergence is striking. Google, Meta (Gatekeeper), Netflix (Trebuchet), and Uber (Flipr) all independently arrived at the same architecture: local evaluation SDK, push-based distribution, kill-switch priority, lifecycle enforcement. Netflix goes a step further — Trebuchet evaluates flags at the API gateway layer for A/B testing on the homepage, attaches the evaluation result to the request context, and propagates variant assignments through all downstream services. This ensures consistent variant assignment within a session and, critically, enables kill switches that stop traffic before it reaches application logic rather than short-circuiting inside it.

That boundary — edge evaluation versus in-process evaluation — is where flag system design intersects directly with your traffic management strategy.

How Big Tech Does It

Netflix — Trebuchet:

Evaluates flags at the API gateway layer for A/B testing on the homepage. Attaches variant assignments to the request context. Propagates through all downstream services. Kill switches stop traffic before it reaches application logic. Homepage experiments run on tens of millions of users simultaneously.

Meta — Gatekeeper:

30–60 second propagation window across the evaluation tier is intentional. Staleness is acceptable. Staleness-during-outage is not. Incremental rollouts auto-ramped with business metric feedback. Thousands of simultaneous experiments.

Google — internal flag systems:

Flag exposure events feed directly into ABACUS, their experimentation platform. Exposure events are the join key between user actions and flag variants. Without them, experiment data is uninterpretable. Every evaluation emits a structured event: {flag_key, variant, user_id, context_hash, sdk_version, timestamp}.

Uber — Flipr:

Region-aware kill switches. A single flag can disable surge pricing across all regions in 30 seconds. Driver matching, dispatch logic, routing algorithms — all gated. City-by-city control granularity.

Key takeaways — the checklist:

✅ Local evaluation only — no RPC on the request path, ever

✅ Push-based distribution — pull creates thundering herds at scale

✅ Kill switches evaluated first — before any targeting rule, before any user context

✅ Per-flag fail policy — fail-closed or fail-open declared at creation, not at runtime

✅ Deterministic rollouts — hash(flagKey + userId) % 100, same seed everywhere

✅ Per-request memoization — one traversal per flag per request, not one per call site

✅ Exposure events at every evaluation — the foundation of experiment analysis

✅ Owner + expiry date required at creation — enforced by automation, not convention

✅ Automated cleanup PR when flag hits 100% stable — dead code doesn't survive on inertia

✅ Explicit depends_on in the schema — implicit flag dependencies are 3am archaeology problems

The test: Can your on-call engineer disable a production feature globally in under 60 seconds without touching code or config files? If yes — you have a kill switch. If no — you have a boolean in a YAML file.