DEV Community: SAI RAM

trelix v3.2.2 to v3.3.8: A GitHub App Already Hardened and Running in Production, and a Connector That Never Touches a Pixel

SAI RAM — Sat, 26 Sep 2026 11:08:08 +0000

docker run --entrypoint trelix-mcp ghcr.io/sairam0424/trelix:3.2.1 --help returns exit code 127: command not found. Not a typo, not a stale tag — the published 3.2.1 image never contained trelix-mcp at all. The builder stage copied core trelix and nothing else, and .dockerignore's blanket packages/ exclusion would have blocked the MCP package even if someone had remembered the COPY line. A second, unrelated defect shipped in the same release: the published trelix-mcp console script ignored every flag you gave it — --help, -h, --version, anything — and silently launched the real MCP stdio server instead. Both were found the only way they could have been found: by installing the actual published artifacts and running them, not by reading a source tree that had never been wrong.

That's v3.2.2, and it's where this span starts. This article covers the thirteen tagged releases from v3.2.2 through v3.3.8 — 2026-08-26 through 2026-09-25 — picking up the day after the last article in this series ended at v3.2.1. v3.3.8 is the current release: tagged, pushed, marked "Latest," and — I checked this directly rather than trusting the changelog — live on PyPI right now.

Thirteen releases, one throughline

The prior span's throughline was a green test suite that never touched the bug it claimed to cover — a MagicMock standing in for an embedder, an all-ones attention mask making masked and unmasked math identical, a unit test that asserted a bug as its own specification. That defect class is still present in this span (more on it below), but it's no longer the dominant story. Two things sit ahead of it now.

The first is that the response to catching defects graduated from "write a regression test" to "build a system that makes this class of defect structurally unable to reach a tag again." The second is quieter and, I think, more interesting: almost everything genuinely new in this span — a diagram connector, an MCP protocol upgrade, a retriever-caching fix, a compression provider, an audit-log pruning mechanism — turned out to be sound at the code level when I went and checked it against source. What kept drifting was the layer just above the code: a renamed class the CHANGELOG never caught up to, a performance number with no artifact behind it, a guarantee that overstates what the code actually does. If the last article's angle was "the code lied to the tests," this one's second angle is closer to "the code is right, and the notes about the code keep being slightly wrong" — which is a more mature failure mode, and one that only shows up once the cruder failure mode has already been closed off.

The origin: a manual audit that automated itself

v3.2.2's fixes were correct but ordinary. What matters is what happened next. v3.2.3, two releases later, turned the exact manual "install the real published artifacts and run them" pass that had just caught two live defects into permanent CI: a real-subprocess E2E suite that spawns trelix-mcp as an actual OS process talking real stdio JSON-RPC, fresh-venv installs of all four published packages, a smoke-test-built-artifacts job that release.yml's publish job is now hard-gated on, a Docker check that trelix-mcp is actually present in the built image, and a Helm-lint check that the rendered image tag matches Chart.yaml. The same release fixed a real, separate bug along the way — a LIKE-wildcard escaping gap in path_filter-scoped BM25/grep queries, where unescaped _ and % in ordinary directory names were being read as SQL wildcards. Not injection (the queries were already parameterized), just a distinct semantics gap, closed with a shared escape_like_pattern() helper and an ESCAPE '\' clause.

The centerpiece is scripts/verify_release.py, a 742-line script with five independent check functions that each return a list of failures and never raise — so one category's bug can't hide another's results. check_pypi_installs does a fresh venv per package and a real pip install <pkg>==<version>, then a real console-script and import smoke test against the version string. check_docker_images pulls both the plain and -local tags and runs the exact --entrypoint trelix-mcp --version command that returned 127 on the unpatched 3.2.1 image — this check exists specifically because that command failed once, silently, in production. check_helm_chart checks out v<version> into a throwaway git worktree, never the caller's own working tree, and runs helm lint/helm template across all three store backends. check_github_release_binaries downloads and file(1)-checks all four platform assets and executes whichever one matches the current host, printing rather than silently skipping when it can't run a foreign-arch binary. check_security_audit runs pip-audit --format json on all four packages in a fresh venv and separately scans every wheel for .env*, .git-path, or credential/secret-named files that should never ship.

v3.2.4 wired this permanently into CI via .github/workflows/verify-release.yml: it waits for both release.yml and docker-publish.yml to go green for a pushed tag, then runs verify_release.py automatically and posts a PASS/FAIL comment, with the manual command still available for ad-hoc re-runs. I read the 99-line workflow directly — it uses env: passthrough rather than ${{ }} shell interpolation, matching docker-publish.yml's own injection-avoidance convention. v3.2.5, the same day, fixed a smaller instance of the identical class of gap: the frozen PyInstaller binary's embedder error message told users to pip install 'trelix[local]', which does nothing at all for a binary that never touches the host's Python or pip.

I ran the one check in this whole dossier that didn't rely on reading source: curl https://pypi.org/pypi/trelix/json. info.version is 3.3.8. The release history includes 3.2.2 through 3.2.5, then 3.3.0, 3.3.5, 3.3.6, 3.3.7, 3.3.8 — 3.3.1 through 3.3.4 are conspicuously absent from PyPI's own list, which I take to mean they were likely never tagged as standalone PyPI releases rather than any kind of cover-up; I didn't dig further and I'm flagging it rather than resolving it. trelix-mcp is confirmed in lockstep at 3.3.8 too, wheel and sdist both uploaded 2026-09-25T09:54 UTC. That's a directly executed, non-paraphrased result — the single strongest claim in this whole article, because it's the one number here that isn't source-reading, it's a live check against the real index.

v3.3.0: the migration that broke things on purpose

Every other release in this span is a patch. v3.3.0 is a deliberate minor bump with four real breaking changes, which is the honest way to signal "this will require you to change something," and I want to give it credit for that framing even where its own migration doc undersells it.

The four changes: the Python floor moved to >=3.12 across all three packages; the TRELIX_RETRIEVAL_FLARE_MAX_ITER env alias was removed outright, with zero remaining references anywhere outside docs; AWS_REGION is now hard-required for the Bedrock backend, raising ValueError where it used to silently default to us-east-1; and the Anthropic backend now silently no-ops on a temperature argument instead of forwarding it, logging a one-time warning instead. That last pair — a required env var and a silently dropped parameter — are exactly the two changes most likely to actually break a deployment on upgrade, and they are, by my own read of docs/migration/v3.2-to-v3.3.md, the two the migration guide doesn't mention at all. The doc covers three of the six real changes and its own closing section states the LLM-provider and MCP changes are "not yet released" — they shipped the same day as the doc, under the same version tag. If you're upgrading past v3.3.0, read the CHANGELOG, not the migration guide.

The release also introduces a reasoning-content abstraction across both cloud backends — and here's a naming error worth stating plainly because it'll cost you time if you don't know about it: the CHANGELOG calls this type ReasoningBlock. The actual class, which I confirmed directly in src/trelix/llm/client.py (line 29), is named ThinkingBlock. A repo-wide grep for ReasoningBlock across src/, packages/, and tests/ returns zero matches. ChatResponse.thinking_blocks: list[ThinkingBlock] plus a back-compat ChatResponse.thinking: str | None is what actually ships, and both the Anthropic and Bedrock backends build ThinkingBlock instances correctly, including the previously-dropped redacted_thinking variant. If you grep the codebase for the CHANGELOG's own name, you'll find nothing — use ThinkingBlock.

Separately, MCP gained SEP-2322 support: AgentResult.needs_input, set only on an explicit CLARIFY action from the agent loop, lets the ask_agent MCP tool wrap an ElicitRequest inside an InputRequiredResult when the agent genuinely needs clarification, resuming later via a dedicated answer-extraction path. Non-elicitation callers are unaffected — it's additive. This is the most solidly verified claim in the release, and also the one place the CHANGELOG's own claim doesn't hold all the way through: Bedrock's stream() method got a fix for dropped reasoning deltas, and the changelog describes it as fixing dropped reasoning and tool-use deltas together — "now logged at debug level instead of vanishing untraced." Reading the actual diff, only the reasoningContent half shipped; there is no branch at all for toolUse deltas, and a contentBlockDelta carrying incremental tool-call argument fragments still falls through with zero logging, exactly as it did before the fix. It's a small gap, but it's the kind of gap that only shows up when you read the code instead of the note describing it.

The parser-correctness class, one level deeper

The prior article's four-instance spine — a green suite that never exercised the defect — recurs here too, but this time it's the product of a deliberate audit rather than an accident someone happened to trip over: a correctness sweep hunting one specific bug pattern, qualified-name collisions, that pulled 19 independently confirmed defects across seven subsystems in v3.2.4 alone.

Two of them are worth walking through because they're the same shape at different layers. The installed tree-sitter-java grammar emits formal_parameters/formal_parameter for a Java record's component list, not the record_parameters/record_component node types the extractor had assumed — every record's component list had been rendering empty. The fix shares a single _extract_type_list_edges() helper across six call sites rather than patching the one spot that got noticed. Rust had the mirror problem: use foo::{Alpha, Beta} — the dominant real-world Rust import form — fell through to a generic single-name fallback because the code checked for use_tree_list/use_tree node types tree-sitter-rust doesn't actually emit; the real types are scoped_use_list/use_list. Pre-fix, that path would have produced the literal string "{Alpha, Beta}" as a bogus import name.

One layer up, in the federation retriever, make_scip_symbol_id() hashed only (package, version, qualified_name) — so two files sharing a qualified name, like two def main() entry points in different modules, collided, and the second was silently dropped via INSERT OR IGNORE. The fix folds file_path into the hash. And a fix in v3.3.4's chunker — where oversized symbols had their excess body silently discarded and replaced with a truncation marker, unrecoverable by retrieval — created a new collision surface one release later: multiple chunks could now share one symbol_id, and the fusion/dedup layer's key didn't yet account for that. v3.3.5's own fix docstring is candid about this being the same defect class recurring one abstraction level down from a prior bug, in code the same engineers had just touched days earlier. That's the honest version of "we fixed it" — not zero recurrence, but recognizing the recurrence immediately and naming it as such rather than treating each instance as a surprise.

A separate 94-agent chunking-strategy study, read in full, found trelix's one-symbol-per-chunk strategy underperforming alternative strategies by 3.57–5.64 percentage points of exact-match accuracy against a real published benchmark — and then rated its own recommendation "LOW-MEDIUM confidence a full redesign is worth building right now," because a redesign would break the 1-chunk-to-1-symbol cardinality assumption baked into the reranker and graph-context code — "the exact class of bug this session already found three separate times in adjacent code this same week." That's a research doc citing its own team's bug history as the reason to be cautious about its own conclusion, which is a genuinely unusual thing to see in a changelog-adjacent artifact, and I think it's a better outcome than either shipping the redesign on a hunch or ignoring the study.

trelix Code Review: not a Marketplace app, and one open question

There are two real, shipped ways to get trelix's review on a PR, and both post a check literally named "trelix Code Review." The first is a GitHub Actions workflow that trusts nothing beyond the repo's own GITHUB_TOKEN. The second is a standalone GitHub App — TypeScript/Express, webhook-driven, zero workflow YAML required in the installing repo, confirmed live right now on Railway (production environment, deployment from commit ac94e1c on main, instance status RUNNING). The tradeoff between the two is exactly what it looks like: the Actions path trusts nobody, the App path is more convenient but grants a real third party real permissions on your repo.

It is not, and should not be described as, a Marketplace app. infra/github-app/README.md says this outright, in words I'm quoting rather than paraphrasing: "Not claimed: GitHub Marketplace listing. This App is installable and hardened, not Marketplace-verified." docs/ROADMAP.md lists the Marketplace listing as an explicit open backlog item, gated on needing roughly 100 installations before GitHub will even review it for the listing — an adoption gate, not an engineering one.

The security posture underneath, which I read at the file level rather than taking on faith: webhook signature verification uses timingSafeEqual via @octokit/webhooks-methods and rejects unauthorized requests before the payload body is even read; installation tokens are minted and cached per-config in a WeakMap, with JWT-mode auth correctly not cached; PR checkouts clone into a temp workspace and fetch refs/pull/<n>/head, which is the correct ref for fork PRs, with the token passed only via a GIT_ASKPASS environment variable — never in a URL, argv, or git config, where it could leak into logs or process listings; and there's a 25MB payload cap applied before signature verification, which is the only defense against an oversized unsigned body. As of v3.3.8, a review that times out or crashes now always posts a completed Check run instead of leaving the PR with no status at all — timed_out conclusion when the process was SIGTERM'd, neutral otherwise. That's a real, shipped fix for a real silence-on-failure gap, and it's a good instinct for a system whose entire value proposition is a status check people trust.

What's still open, and what I want to be careful not to resolve in either direction just because it would make a tidier ending: there's a documented, not-yet-confirmed-resolved concern that the App's own review of trelix's own large codebase has been getting SIGKILLed before indexing finishes, suspected to trace back to review-runner.ts's hardcoded 5-minute INDEX_TIMEOUT_MS/REVIEW_TIMEOUT_MS. I checked those constants directly — they're unchanged at 5 minutes, in this span and since. But I also checked every CHANGELOG line, every ROADMAP line, and the git log for any commit connecting those two constants to a SIGKILL-during-self-review incident, and found nothing. The only SIGKILL mentions anywhere in the App's code concern a completely different scenario — OOM-killed processes leaving stale temp workspaces behind, which a sweepStaleWorkspaces() cleanup at boot handles. So: the timeout constants are exactly as short as the concern says, but nothing in the repository substantiates that they're actually the cause of a self-review failure, or that the failure is still happening at all. I'm stating this as an open question because it is one, not because I want to end this section on a cliffhanger. It should not be confused with a different, already-closed issue in the same App: a missing TRELIX_EMBEDDER_PROVIDER=azure env var on Railway, which caused indexing to silently fall back to a never-installed local embedder and fail — that was fixed on 2026-09-22 via a deployment config change, not a code change, and it's fully resolved. These are two unrelated bugs at two unrelated layers; only one of them is closed.

The .drawio connector: diagrams as text

v3.3.6 shipped a fifth connector alongside Jira, TestRail, Xray, and Linear: trelix connector sync <repo> diagram, for local .drawio (diagrams.net) files. The mechanism is more mundane than it sounds, in the best way — every shape in a drawio file's mxGraph XML carries its label in a plain value="..." attribute, and every edge is an explicit <mxCell edge="1" source=... target=...>, so the whole diagram is fully text-describable without any vision model. The connector truncates the XML to 20,000 characters and sends it through the same text-only chat client every other LLM call site in trelix already uses. Raster images — PNG, JPG — are explicitly out of scope; there's no glob or handling for them anywhere.

v3.3.7 fixed a real bug in the fallback path: when captioning failed, the connector used to discard the diagram's real content entirely and report just a character count. Now it regex-extracts actual mxCell value="..." shape labels, HTML-unescapes before stripping inline tags (the order matters, since drawio escapes attribute values, so entities have to decode before tags can be safely stripped), de-duplicates in first-seen order, caps at 50 labels, and only falls back to the old length-only message when literally zero labels exist. It's a small fix but a real one — the difference between "the diagram failed to caption, here's 4,200 characters" and "the diagram failed to caption, here are the 12 real shape labels it contains" is the difference between a useless fallback and a usable one. There's no diagram-specific config or env var; it rides whichever LLM provider is already configured, and the artifact linker treats a diagram exactly like a Jira ticket — same edge shape, same matching logic, no special-cased code path anywhere.

Performance: real numbers, and one number that isn't

v3.3.8 fixed a real inefficiency: trelix-mcp, the LangChain retriever, and the LlamaIndex retriever all used to rebuild a fresh Retriever object on every single call. They're cached now — module-level with a lock in the MCP server, invalidated correctly on reindex, per-instance in both framework adapters. The mechanism is sound and I'd recommend the fix without reservation. What I can't recommend repeating is the specific number attached to it: the CHANGELOG cites a "24.5x–52.7x measured speedup," and that figure appears nowhere except the CHANGELOG's own prose — not in the commit message I pulled directly, not in the PR, not in any test. The commit itself says only that three consecutive calls in one session took 13-16 seconds, then 5, then 5, with zero warm-up benefit — no multiplier, and no recorded post-fix warm baseline anywhere to derive one from. The fix is real; the multiplier isn't backed by anything outside the CHANGELOG's own unsupported prose, so I'm not repeating it as measured.

The rest of this theme holds up better. trelix-mcp's and the graph community-detection code's use of nx.pagerank now catches ImportError and degrades to a uniform-score fallback instead of crashing, because scipy is only imported lazily on first call and the frozen binary deliberately excludes it — a real crash class closed with two small, defensive catches. Qdrant gained int8/binary quantization support, with one small flag: the commit claims verification against qdrant-client==1.19.0, but the lockfile currently pins 1.18.0. Not a contradiction — a dev-venv version never captured in the lockfile is the more likely explanation than anything adversarial — but not independently confirmable either, so treat "verified against 1.19.0" as unconfirmed rather than false. A direct Cohere embedder landed using cohere.ClientV2 rather than routing through Bedrock, with a 96-item batch limit. And an abstractive compression provider shipped with a real accuracy gap between what the CHANGELOG claims and what the code guarantees: the CHANGELOG says the signature and docstring line are "always kept byte-for-byte." The actual _must_keep() logic only locates and preserves the declaration line itself — the docstring on subsequent lines is forwarded to the LLM as context, with no verbatim guarantee, and can come back paraphrased. The test suite's own docstring gets this right where the CHANGELOG doesn't: "the declaration line is ALWAYS kept verbatim," singular. If you're relying on this feature to preserve docstrings exactly, don't — that's what the separate extractive compressor genuinely does.

Security: two CRITICAL CVEs and an audit trail that undersells itself

The litellm floor moved to >=1.90.2, closing two CVEs the project's own pins cite as CVSS 9.8 — a Host-header authentication bypass and an unauthenticated SSTI leading to RCE — plus a third SSRF issue via nested user_config. I can confirm these citations are in the repo's own dependency comments; I could not independently cross-check the CVE IDs against GitHub's live advisory database from this sandbox, so I'm attributing the "two CRITICAL, CVSS 9.8" framing to trelix's own release notes rather than an outside source I verified myself. Separately, pydantic-settings CVE-2026-58203 was patched by a floor bump, and I confirmed the vulnerable feature — secrets_nested_subdir — was never actually called anywhere in the codebase, so this one was zero real exposure even before the bump; a clean case of patching a dependency that was never actually exploitable through this code.

The most substantial security addition is trelix audit prune — a real hash-chain watermarking implementation for pruning old audit-log rows without breaking chain verification, with an --dry-run mode that's confirmed not wired into any in-process scheduler. What struck me reading it is that the module's own docstring is more conservative than the CHANGELOG's framing: it states plainly that this is "tamper-evident, not tamper-proof," and it enumerates exactly which tamper shapes remain undetectable — a total wipe that also clears SQLite's own internal sequence table, for instance. A security feature whose own documentation undersells its guarantees relative to the marketing copy around it is the rare direction I'm happy to see the drift run in.

The smaller robustness releases

Two releases of CLI/API/extension hardening are worth a fast pass. v3.3.1 fixed four real VS Code extension bugs: a JSON.parse crash whenever the MCP client returned a plain-text error instead of JSON, a path escape hatch for GUI-launched editors whose PATH excludes pip/uv, inline symbol links in chat responses, and a CodeLens replacing a global QuickPick for blast-radius visualization. v3.3.7 fixed a .vscodeignore gap that had been shipping a local vector database file and gigabyte-plus of sourcemaps inside the packaged extension — vsce ignores .gitignore entirely once .vscodeignore exists, so the two files having drifted apart had real consequences. The same release moved four CLI/REST failure modes from raw tracebacks or bare 500s to labeled, structured errors: eval/eval-synthesis/migrate-vectors now print one-line labeled errors instead of stack traces; a missing golden file for synthesis eval now raises FileNotFoundError instead of silently returning zero-filled metrics; the REST API's DimensionMismatchError and ImportError now return structured JSON 500s instead of Starlette's plain-text default; and static-token auth now checks Authorization: Bearer independently of X-Trelix-Api-Key, via hmac.compare_digest, closing a path where the Bearer header was silently ignored.

The pattern behind the corrections

I want to name the pattern across this article's corrections rather than let them read as scattered nitpicks, because I think they're one thing: ReasoningBlock vs. ThinkingBlock, the half-shipped toolUse-delta logging claim, the unaudited 24.5x-52.7x figure, the docstring-verbatim overstatement, a migration guide that's stale relative to its own release date, a docs/WHY_TRELIX.md version-scope stamp that still reads v3.3.7 while the package is at v3.3.8 — none of these are code bugs. Every one of them is the layer just above the code: the sentence describing what the code does, written slightly faster or slightly more optimistically than the code itself justifies. That's a different failure mode than a MagicMock hiding a broken embedder, and it's arguably a sign of health rather than decline — it only becomes the dominant story once the release-verification discipline from earlier in this span has already closed off the cruder failure mode of a defect reaching a tag at all. It's also, not coincidentally, exactly the discipline this article itself needed applied to it: this dossier exists because the same standard — check the sentence against the source, not against the intent — got turned on the release notes themselves, and turned up a correction on nearly every theme it touched.

What v3.3.8 actually means for someone upgrading today

If you're upgrading from anywhere before v3.3.0, read the CHANGELOG for that release specifically rather than the migration guide — the guide is missing the AWS_REGION requirement and the silently-dropped temperature parameter, which are the two changes most likely to actually break your deployment. If you're grepping the codebase for the reasoning-content type, look for ThinkingBlock, not ReasoningBlock. If you're evaluating the retriever-caching fix, trust the mechanism and not the specific multiplier. If you're relying on abstractive compression to preserve a function's docstring exactly, it doesn't — only the declaration line is guaranteed verbatim. And if you're deciding whether to install the GitHub App or rely on the Actions workflow for PR review, both post the same "trelix Code Review" check, but only one of them asks you to trust a third party — and if you run the App against a codebase the size of trelix's own, know that a timeout-related self-review concern from a week ago hasn't been confirmed fixed, or confirmed to still be happening either.

None of that is a reason not to upgrade. The live PyPI check I ran directly — not read, not paraphrased, executed — confirms v3.3.8 across all four packages, published in lockstep on 2026-09-25. The release-verification infrastructure built in v3.2.3 and v3.2.4 means that check itself, along with a Docker --version smoke test and a Helm chart lint, now runs automatically against every tag before it's allowed to publish, which is a meaningfully different guarantee than the source tree alone ever gave you. What changed across these thirteen releases isn't that trelix stopped shipping defects — the parser sweep alone found 19 of them in one release. What changed is that the mechanism for catching what a green test suite misses stopped being a person doing it once, by hand, out of frustration, and started being a system that does it on every tag, whether anyone remembers to ask or not. The part that hasn't caught up yet is the sentence describing the fix, which is a smaller problem to have than the one this project started with nine weeks ago, and a more honest place to end an audit than pretending it isn't there at all.

MindForge v12.0.0: An Agentic Framework for Claude Code — What It Ships, How to Install It, and What's Actually Enforced

SAI RAM — Thu, 24 Sep 2026 05:42:32 +0000

What MindForge Is (And Isn't)

Agentic frameworks for Claude Code have exploded over the last year — swarms of subagents, skill libraries, protocol layers stacked on top of a single model. Most of them describe what they ship in one flattened register: "AI capabilities." MindForge (mindforge-cc on npm) is a governance and orchestration layer in that same category, sitting on top of Claude Code — it is not a replacement for it, and it does not run without it — but it's one of the more mature, more self-critical entries: it treats its own surface area as five mechanically distinct pieces rather than one undifferentiated pile, and it draws an explicit line between the parts that are genuinely enforced and the parts that are just well-organized advice. As of this writing, v12.0.0 is tagged and live on npm — both latest and stable point to it, with a real GitHub Release at the matching tag — and its CHANGELOG entry says, verbatim, this is "MindForge's first release deliberately cut for real external users." That framing, and how it holds up against the actual source tree, is what this piece checks.

Concretely, MindForge ships as three things: an npm package that installs slash commands, skills, personas, subagents, and hooks into a .claude/ (and mirrored .agent/) directory; a set of Claude Code plugin-marketplace entries; and a separate MCP server plus TypeScript SDK. Almost everything it ships — the commands, the skills, the personas, the workflow scripts, the governance docs — works by being loaded into the model's context. None of that is code the model is forced to obey. The one exception, and the one piece of this system that is a real technical enforcement mechanism, is a small set of Claude Code hooks. That distinction — advisory context vs. enforced blocking — is the single most important thing to understand about this project, and it's the project's own README that draws the line, not outside auditors.

This matters because MindForge's own CHANGELOG and its repo-local .claude/CLAUDE.md spend real effort disowning parts of their own marketing language. The "Unified Protocol Engine" framing that shows up in some MindForge-derived global configs — SwarmController, PersonaFactory, WaveExecutor, soul-engine.js, shard-controller.js — is explicitly called out in the repo's own .claude/CLAUDE.md as aspirational naming with no backing code: those are "role names in the specs under .mindforge/engine/, not importable code." That kind of self-correction, verified directly against the source tree rather than taken on faith, is the throughline of this whole release.

The real working loop that MindForge scaffolds is plan-phase → execute-phase → verify-phase → ship, and it is detailed rather than just a name: plan-phase spawns a research subagent and writes atomic XML plan files with explicit <verify> steps; execute-phase runs dependency-aware waves against a five-level escalating validation ladder (static, unit, build, integration, edge) and writes a Deviation Report per task; verify-phase walks the human through REQUIREMENTS.md deliverables and writes UAT.md, spawning a debug subagent on failure; ship gates on UAT.md reading "All passed," runs tsc --noEmit, eslint, npm test, and npm audit, and generates the PR description. It's real machinery, not command-name theater — though even here there's a live wrinkle worth knowing about: ship.md's Step 4 currently hardcodes a literal branch name in its git push rather than deriving the active branch, so anyone not literally on that named branch could push to the wrong place.

The Building Blocks: Five Different Mechanisms, Five Different Levels of Trust

MindForge's surface area breaks into five genuinely distinct mechanisms, and the differences between them are mechanical, not just naming conventions. Here's what it actually gives you, at a glance, verified count by verified count:

Mechanism	Count	Invoked via	What it mechanically is	Enforced or advisory
Slash commands	221	`/mindforge:<name>`	`.md` prompt specs the model reads and can choose to follow	Advisory
Skills — engine tier	232	Auto-triggered by keyword match	An LLM-followed protocol spec (`.mindforge/engine/skills/loader.md`) does the matching — non-deterministic, not a parser	Advisory
Skills — extended tier	122	Invoked explicitly by name	Same skill mechanism, lenient schema (only a name is required)	Advisory
Personas	216	`/mindforge:agent <name>`	An in-session role overlay — same context, not a new agent	Advisory
Subagents	164 (154 adapted from VoltAgent's MIT-licensed library + 10 original)	Claude Code's native subagent mechanism, via the plugin marketplace	A genuinely isolated execution context — a separate mechanism from personas	Advisory (the definition; the isolation itself is Claude Code's)
Dynamic workflows	35, across 5 tiers (Research 5 / Dev 14 / Ops 6 / Intelligence 7 / Beast 3)	Claude Code's own host-level `Workflow` tool	Curated multi-agent orchestration scripts targeting a host capability MindForge doesn't itself implement	Advisory
Hooks	8 registered (7 executed by preflight, 3 "deny-class")	Automatic, on tool-call boundaries	Real shell scripts wired into Claude Code's own hook system	Enforced — the one row on this table that actually is

"Enforced" in that last row only ever means on Claude Code, via the npx installer's --local or --global target — see the next section for exactly what that does and doesn't cover.

Commands are 221 slash-command .md files under .claude/commands/mindforge/, mirrored exactly in .agent/mindforge/ — verified by direct file count, not just README's say-so. They're prompt specs the model reads and follows; they don't execute independently of the model choosing to follow them.

Skills split into two tiers totaling 354 files: 232 "engine tier" skills under .mindforge/skills/, auto-triggered by keyword match, and 122 "extended tier" skills under .agent/skills/, explicitly invoked by name. The engine tier enforces a strict schema via tests/skills-platform.test.js — semver version, a status field, at least ten trigger terms, a "Mandatory actions" section — while the extended tier only requires a name. Here's the part worth being honest about: the file that's supposed to do the actual trigger-matching, bin/engine/skill-loader.js, is dead code — a four-line stub (loadSkill: () => null, matchTriggers: () => []) with zero real callers, and the CHANGELOG says so directly. The real trigger-matching mechanism is .mindforge/engine/skills/loader.md, an LLM-followed protocol spec, not deterministic code. That means "skill triggering" in MindForge is non-deterministic by design — the model decides what counts as a match, not a parser.

Personas are 216 named role-overlay files under .mindforge/personas/, loaded via /mindforge:agent <name> into the same session context — a role swap, not a new agent. This is a meaningfully different mechanism from subagents, and MindForge's own docs are explicit about the distinction rather than blurring it for effect.

Subagents are 164 real Claude-Code-native subagent definition files under subagents/categories/, each with proper frontmatter (name/description/tools/model) designed for Claude Code's isolated-context subagent mechanism — genuinely separate execution contexts, unlike personas. 152 of those files are attributed to VoltAgent's MIT-licensed awesome-claude-code-subagents, with the full license text reproduced; that attribution checks out against the vendored upstream repo. One correction worth flagging: the README's claimed "152 adapted / 12 original" split undercounts the adapted set by two — dotnet-framework-48-expert.md and powershell-51-expert.md are near-byte-identical VoltAgent copies with dots stripped from the filename for path safety, so the real split is closer to 154 adapted / 10 original. Small, but it's the kind of thing worth naming rather than repeating uncritically.

Dynamic workflows are 35 JavaScript scripts across five tiers (Research 5, Dev 14, Ops 6, Intelligence 7, Beast 3) under .mindforge/dynamic-workflows/scripts/, and the ones that were read in full are genuinely substantial multi-agent orchestration code — security-hardening.js, for instance, runs five parallel OWASP scout agents, then for every critical/high finding spins up a nested three-vote adversarial verification round (refute / challenge-exploitability / assess-impact) requiring a real two-of-three quorum before accepting a finding. That's not a cosmetic label. But here's the structural fact that matters most: the actual execution primitives these scripts call — agent(), parallel(), phase() — are not implemented anywhere in the MindForge repository. They're supplied entirely by Claude Code's own host-level Workflow tool at runtime. MindForge ships the script library, the JSON registry, and the CLI browser; it does not ship a self-contained execution engine. Every one of these scripts also contains a top-level return statement, so none of them are valid standalone Node.js files — they only run wrapped by that external harness. This is a real architectural fact, not a defect, but it's exactly the kind of thing a launch article shouldn't gloss over: "35 dynamic workflows" means 35 curated orchestration scripts that target a host capability, not 35 things MindForge alone can run.

Advisory Context vs. Enforced Blocking

This is the section that most agent-tooling projects don't write, and MindForge's README does, in a dedicated "What is actually enforced" section: hooks are the only real blocking mechanism in the entire framework, and they only block on Claude Code, via the npx installer's --local target. They are explicitly not enforced on Cursor, Copilot, Gemini, or Antigravity; not on --global installs; not on self-installs; and not on Windows.

The live, dogfooded numbers behind that claim hold up under direct verification: the repo's own .claude/settings.json registers exactly 8 hooks (2 PostToolUse, 4 PreToolUse, 2 SessionStart). Of those, 7 are executed by the installer's preflight check (one, mindforge-check-update, is deliberately skipped). Of the 7 executed, only 3 are "deny-class" — capable of returning a hard block: trust-gate, mindforge-block-no-verify, and mindforge-config-protection. All three were confirmed live to return exit code 2 on trigger. Everything else in this system — the 221 commands, the 354 skills, the 216 personas, the 35 workflows, the governance and audit docs — can be ignored by the model at any time. A launch article shouldn't imply that installing MindForge "activates" or "enforces" its governance or security-scan language across arbitrary tools or channels. It doesn't. It narrows enforcement to three specific hooks on one specific install channel, and says so.

The Audit Chain and Why This Release Is Different

MindForge maintains a tamper-evident audit log at .planning/AUDIT.jsonl, hash-chained with SHA-256 via a shared hasher (bin/governance/audit-hash.js) used by both the writer and the verifier. Running the verifier live against this repo's own log returned "audit chain valid: 6396 entries." SECURITY.md scopes the claim precisely — the chain detects mid-file mutation and deletion, and explicitly does not detect tail truncation or replay — and a dedicated honesty test suite (tests/audit-claims-honesty.test.js) checks that the documentation doesn't overclaim beyond that, including confirming the writer doesn't sign entries and the docs never call this a Merkle tree.

The more interesting story is what the project's own pre-ship audits actually catch, because it's a real mix of genuine bugs and genuine overclaiming, disclosed rather than buried. The v11.9.9 release describes an eight-agent audit workflow run as a release gate before shipping to real external users, and it's not a rubber stamp: that pass found 5 CRITICAL issues, including a jailbreak skill named "godmode" that had been shipping live, a security check with no real CLI entrypoint (so it could structurally never fail), and a --minimal install flag that silently shipped the full persona set anyway — plus 9 HIGH findings. Verified directly: neither the godmode skill nor any of its assets exist anywhere in the shipped tree, and a packaging-allowlist test explicitly asserts its absence. That same audit pass also fixed a set of docs (help.md, status.md, health.md, security-scan.md) that had been overclaiming a post-quantum-signature feature (PQAS) and biometric gating as active; the real code is honest about this now — bin/governance/quantum-crypto.js labels its Dilithium-5 implementation "SIMULATED... NOT real ML-DSA/FIPS-204," gated off by default in config. Earlier releases show the same pattern repeating: v11.9.8 found two real crashing bugs plus 12 pure documentation inaccuracies out of 113 claims checked; v11.9.6 found a crashing /mindforge:learn command and an auth-token-leaking browser daemon.

Two things are worth naming honestly rather than hiding. First, that PQAS-overclaim pattern had a residual instance the 11.9.9 audit missed: .mindforge/governance/policies/sovereign-default.json still marked PQAS and biometric checks as "ENABLED," even though that file is dead config the real PolicyEngine never loads. Second, the same "feature that isn't wired to what it implies" pattern shows up in a place the audits hadn't caught as of this research pass: MindForge's README describes a "two-model adversarial PR review" and a "4-voice consensus council," but as currently wired, both mechanisms route every voice and every round to the exact same single backing model — the cross-review engine's caller-supplied model names are used only as display labels, and the council's four voices all resolve through an unmapped persona to the same tier-2 default. That gap between naming and behavior was not in the CHANGELOG as of v11.9.9 — it's the kind of thing this project's own audit discipline should catch in its next pass, and naming it here is more useful to a reader than pretending it doesn't exist.

Also worth knowing: several of the CLI/doc claims that have surfaced in past audits are still true today and still disclosed rather than fixed. /mindforge:health --repair silently drops the --repair flag (byte-identical output to plain health); spawn <persona> still exits 1 with "not implemented in v1.0"; and two doc inaccuracies that survived multiple audit passes are still live — docs/commands-reference.md still lists a CLI command, quantum-verify, that doesn't exist (confirmed by running it), and docs/tutorial.md still documents an --ads flag on /mindforge:plan-phase that was never wired into the actual command file.

Cost-Aware Model Routing: The Honest Version

MindForge advertises cost-aware, difficulty-aware model routing across five real providers — Anthropic, OpenAI, Gemini, Bedrock, and Ollama — and the provider layer is genuinely live: each provider makes real HTTPS calls and computes real per-call cost through a shared pricing registry. What's worth being precise about is what actually decides which model gets used in production: it's persona plus a caller-supplied security tier (0–3), not computed task difficulty. Tier 3 forces a dedicated security model; named personas map to fixed model settings; everything else falls to sensible defaults (Haiku for quick tasks, Sonnet as the general executor).

There are, in fact, three separate difficulty-aware routing systems built into this codebase, and none of them are live. A real 1–10 difficulty scorer exists and is unit-tested, but its only caller runs it in shadow mode — it logs what model it would have picked and changes nothing. A second system, a full "multi-cloud arbitrage" broker with its own passing tests, is required nowhere except by its own test file (it even lists a fourth "azure" provider with no corresponding implementation). A third, MIR-based steering engine is explicitly gated behind a shadow_mode config flag and is imported by the one module that's supposed to call it — but never actually invoked. This is disclosed territory: the CHANGELOG documents fixing a near-identical prior bug in the same routing module (a regex mismatch that made every config value resolve to undefined), so the pattern of "wired but not connected" recurring in cost routing isn't new to this codebase — it's just not fully cleaned up yet.

Shipping Surface: MCP Server, SDK, and Three Real Channels

MindForge ships three real, independently-verifiable channels, not one. mindforge-mcp-server@12.0.0 is genuinely published on npm right now, matches the repo source exactly (including SLSA provenance), and registers exactly 8 MCP tools — 6 read-only (health, status, memory query/stats/find-related, audit log), one guarded write (memory_remember), and one guarded, open-world tool (browse, which proxies to a separate pre-existing bearer-token-authenticated Playwright daemon and never spawns it itself). Worth flagging honestly, because the project's own README already does: the official MCP registry listing for this server lags one patch version behind what's actually published on npm, and README tells readers to check what that listing actually serves, or install from npm directly, rather than trusting the registry blindly.

mindforge-sdk@12.0.0 is also live on npm and matches source. Its batchExecute spawns real child processes with SIGTERM-then-SIGKILL escalation — and there's a named regression test in the SDK's own test suite that exists specifically to guard against a previously-shipped fake stub that used to return {executed:true} without doing anything. That's a good tell for how seriously this project treats its own past mistakes: it doesn't just fix a caught bug, it writes a test whose name records what the bug was.

The underlying persistence layer for MindForge's local knowledge graph is a genuinely zero-native-dependency setup: sql.js, a WASM-compiled SQLite, backing an 11.8MB file (celestial.db) with real FTS4 full-text search, plus a separate SHA-256-checksummed, file-locked JSONL edge list for relationship data. The live dashboard at localhost:7339 is a real Express + SSE server, bearer-token-authenticated on mutating routes, rate-limited, localhost-only, with seven API-backed tabs. Worth being precise here too: the more elaborate dashboard sub-features described in some MindForge-adjacent marketing language — a Temporal Slider, Hindsight Steering Vectors, a $100/hr AgRevOps ROI hub — were not independently verified beyond confirming the dashboard's base HTTP response; treat those as unverified narrative, not confirmed capability.

Installing MindForge

MindForge ships five real, independently-checkable install channels — pick whichever matches how you actually work. As of this writing every one of them installs v12.0.0 directly.

npx (recommended) — writes .mindforge/ governance, memory, and planning into your project, and registers the 8 hooks described above:

npx mindforge-cc@latest --claude --local      # Claude Code, this project only
npx mindforge-cc@latest --antigravity --local # Antigravity, this project only
npx mindforge-cc@latest                       # interactive wizard, pre-selects a detected runtime

Run non-interactively (CI, piped, scripted) and it skips the wizard entirely and installs --claude by default, regardless of what's actually on the machine. For a system-wide install: npx mindforge-cc@latest --claude --global — note that npm install -g mindforge-cc@latest on its own only puts the mindforge/mindforge-cc binaries on your PATH, it doesn't scaffold anything; you still need to run the --global command above once. Other supported runtimes use the same flag pattern (--cursor, --copilot, --gemini). A few advanced flags worth knowing: --runtime claude,cursor (combined runtimes in one pass), --minimal (essential scaffolding only, no persona library), --force (rewrite an existing schema file with the current, stricter one).

Claude Code plugin marketplace — writes no project files at all; only the plugin's hooks fire (see the enforcement section above for exactly what that does and doesn't cover):

/plugin marketplace add sairam0424/MindForge
/plugin install mindforge@mindforge

Want just a slice instead of everything? mindforge-lang@mindforge and 9 other focused sub-packs exist for narrower installs.

Standalone MCP server — if all you want is the 8 read/write-guarded stdio tools described earlier, without any of the command/skill/persona tree:

claude mcp add mindforge -- npx -y mindforge-mcp-server

Also listed on the official MCP Registry, though that listing is manually republished and can lag behind npm — install mindforge-mcp-server directly if you want to pin an exact version.

Homebrew:

brew install sairam0424/tap/mindforge

The formula pins to a specific npm tarball with a checked sha256 — currently pointed at mindforge-cc-12.0.0.tgz, confirming this channel is already caught up for this release too, not a stale mirror.

SDK — build on MindForge programmatically, without installing any of the rest:

npm i mindforge-sdk

Once installed through any of the above, the natural entry points are /mindforge:init-project to scaffold a new project under the framework, /mindforge:next for auto-discovery of what to do given current state, and the plan-phase → execute-phase → verify-phase → ship loop described earlier for anything substantive. The subagent library is best installed through the plugin marketplace above rather than through bin/spawn-agent.js, which — despite what its own README implies about installing subagents "by name" via that script — is a dry-run/verification tool only; without --dry-run it always exits 1 with "not implemented in v1.0."

What v12.0.0 Actually Means

By the time research for this piece began, the goalpost had already moved once — from v11.9.9's own framing as a "release-readiness milestone" to a live git branch, spotted mid-research, whose commit message named v12.0.0 as the actual "first-user-release cut." It moved again shortly after: merged to main via PR #298, with a formal CHANGELOG entry titled, verbatim, "[12.0.0] — First release aimed at real external users," but not yet tagged or published. As of this update, it's moved a third and final time: v12.0.0 is tagged, published, and live — latest and stable on npm both point to it, and mindforge-mcp-server and mindforge-sdk shipped alongside it at the same version. The CHANGELOG entry says plainly what that title implies and nothing more: "MindForge's first release deliberately cut for real external users, not just internal iteration. The major-version bump is a milestone marker for that shift, not a signal of a breaking API/behavior change — there is none in this release; every item below is a fix."

That's the honest way to read a major-version bump in this project: not a rewrite, a readiness gate. It shipped because a second, independent eight-agent audit — security and STRIDE threat modeling, staff-engineer code review, dependency and license review, a live production dry-run across all six supported runtimes, docs-accuracy checking, re-verification of the prior release's deferred backlog, test-coverage gap analysis, and a full trace of the CI/CD release pipeline — was run specifically to answer one question: was v11.9.9 actually ready for real external users? The answer was no, not quite. That audit surfaced 1 CRITICAL and 4 HIGH findings, plus 8 MEDIUM/LOW findings judged worth fixing before this exact cutover, and every one of them was independently adversarially re-verified — 10 out of 10 confirmed, zero refuted — before being fixed.

The CRITICAL finding is worth naming, because it's the same species of bug this piece's own research caught mid-flight, independently, the same week: a --global install printed a fabricated "PAYLOAD MANIFEST" claiming 216 personas, 122 skills, and more were now "active," while writing none of them. The success banner had been counting the source package tree instead of what a global install actually writes by design — an entry file, commands, and subagents only. It's fixed now, replaced with an honest message describing exactly what a global install does and doesn't include. The HIGH-severity fixes closed a related gap — an install banner still claiming a confirmed no-op feature was "active," unhedged, one line below a properly-hedged claim about a different feature — plus a crash fix in one of the dynamic-workflow scripts and two new regression tests written specifically to keep the godmode-removal and --minimal fixes from 11.9.9 from silently regressing. The MEDIUM/LOW list is the unglamorous kind of hardening that doesn't make headlines but is exactly what "ready for real users" should mean in practice: a missed curl-chmod-run dropper-chain pattern in the install-time security gate, a prototype-pollution guard added to a shared config singleton, a CodeQL-flagged regex-escape bug, a non-LTS base image reverted, a persona-table doc regenerated wholesale from source instead of hand-maintained, and a release-pipeline step that had been silently failing on four consecutive prior releases for a root-caused, now-fixed reason.

So, the honest final answer to what this release means: a mature, iteratively-hardened orchestration layer, now on its 77th npm publish, just finished a second full external-readiness audit, found and fixed one more real critical bug in the process, and shipped — tagged, published, live — as exactly what its own CHANGELOG calls itself: the first release cut deliberately for real external users, not just internal iteration. That's a more credible launch story than an inflated "v1.0" narrative would ever be, precisely because the discipline of catching real bugs before shipping to real users kept recurring right up to the moment this one actually went out the door.

tracehub-mcp: Giving AI Assistants a Real Query Interface Into Your LLM Traces

SAI RAM — Sun, 13 Sep 2026 14:06:24 +0000

The Copy-Paste Problem

Here's what debugging an LLM application looks like for most people today, including me until recently: something is slow, or wrong, or expensive, and the AI assistant sitting in your editor wants to help. It can read your code. It can read your logs if you paste them in. What it cannot do, natively, is ask your trace backend a question. So you open Jaeger or Datadog in a browser tab, find the trace, copy the JSON blob, and paste it into the chat window. Then you do it again for the next trace, because "compare the last five calls to this model" isn't a query you can run — it's five manual round trips.

tracehub-mcp exists to delete that loop. It's an MCP (Model Context Protocol) server that gives Claude, Cursor, Windsurf, Gemini CLI, or any MCP client a direct query interface into your OpenTelemetry trace backend. Instead of pasting JSON, the assistant calls a tool — search_traces, get_llm_expensive_traces, get_llm_model_stats — and gets back structured trace data it can actually reason about. Find expensive calls, debug errors, compare model performance across providers, track token usage over a time window: these become questions you ask, not spreadsheets you build by hand.

The part that matters most for an LLM-observability tool specifically is that it doesn't treat a trace as an anonymous bag of span attributes. It speaks OpenTelemetry's gen_ai.* semantic conventions natively: prompts and completions pulled from actual span events, a token-usage fallback chain for providers that don't emit a clean total, finish reasons as a queryable field rather than a string buried in an attributes dictionary. That's why this is worth building over pointing an assistant at a generic OTel viewer, and it's the throughline for the rest of this piece: the five backends, the hardening bar, and the scope decision about which parts of the code get held to it.

Today is 2026-09-13. tracehub-mcp is at version 0.3.0, live on PyPI, tagged v0.3.0 — the shipping version I'm describing, not a changelog recap of getting from 0.2 to 0.3.

From Three Backends to Five, and What "Full Implementation" Means

tracehub-mcp didn't start as a five-backend project. It inherited three — Jaeger, Grafana Tempo, and Traceloop — and the growth story worth telling is how it became a five-backend, security-hardened server without the two new backends being thin wrappers bolted on for coverage.

Every backend implements the same BaseBackend interface (src/opentelemetry_mcp/backends/base.py), so every MCP tool works identically no matter which backend is configured — search_traces, search_spans, get_trace, list_services, get_service_operations, health_check, whether you're pointed at a local Jaeger container with no auth or a paid Datadog account with an API key and app key pair. Jaeger and Tempo cover local/self-hosted setups; Traceloop covers cloud LLM observability with API-key auth. Those three came from the upstream fork.

Datadog and Sentry are new, and genuinely new: full implementations of all four MCP-facing operations for each, not stubs that return "not implemented" past a basic search. src/opentelemetry_mcp/backends/datadog.py and src/opentelemetry_mcp/backends/sentry.py each implement search_traces, search_spans, get_trace, and list_services as independent code paths with their own request building, pagination, and parsing logic. That's the bar I hold "full backend" to: every tool a client can call has to actually work, not just the one that made for a good demo.

The more interesting decision was holding those two backends to a security bar that's specific rather than generic. "We hardened it" means nothing on its own. Here's what it concretely means:

HTTPS-only enforcement. Both refuse to construct at all if the configured URL isn't https:// — a ValueError raised in __init__:

if not self.url.startswith("https://"):
    raise ValueError(
        "Datadog backend requires an https:// URL - DD-API-KEY and "
        "DD-APPLICATION-KEY must not be sent over plain http"
    )

Sentry has the identical check, worded for its single bearer token instead of an API-key pair. Datadog goes further and disables automatic redirect-following on its httpx client, because its non-standard DD-API-KEY/DD-APPLICATION-KEY headers aren't stripped by httpx on a cross-origin redirect the way a standard Authorization header is — a subtlety Sentry's own module docstring calls out as the reason it doesn't need the same override. The hardening was reasoned about per-backend, not copy-pasted.

Query-injection-safe escaping. Every value spliced into a Datadog query goes through _escape_dd_query_value, which escapes embedded backslashes and quotes and wraps the result in exact quotes. Field names — the part you'd assume is safer because it isn't free text — go through a separate allowlist, _VALID_DD_FIELD_RE = re.compile(r"^@?[A-Za-z0-9_.]+$"), before they're anywhere near the query string, because a filter's field parameter is an unvalidated string reachable from any MCP tool call. Sentry has the equivalent pair, _escape_sentry_query_value and _VALID_SENTRY_FIELD_RE, adapted to Sentry's syntax.

Bounded pagination. Datadog's span search loops through cursor pages capped at _MAX_SEARCH_PAGES = 10, stopping when the requested limit is satisfied or the cursor (meta.page.after) runs out, whichever comes first, and logs a warning if it hits the cap with more data available:

for _ in range(_MAX_SEARCH_PAGES):
    remaining = limit - len(collected)
    if remaining <= 0:
        break
    ...
    cursor = page_meta.get("after")
    if not isinstance(cursor, str) or not cursor:
        break
else:
    logger.warning("Search truncated at %d pages", _MAX_SEARCH_PAGES)

Sentry does the same shape of loop through HTTP Link-header cursors instead, matching how its pagination actually works, also capped at 10 pages. A single tool call can't turn into an unbounded crawl against either backend, and the code says so out loud when it truncates rather than returning a partial result silently.

Exact-ID re-verification. This one has the sharpest story. Datadog has no "get trace by ID" endpoint, so get_trace reconstructs a trace by searching for spans matching the trace ID and grouping the results — inherently a search, which means it could match more broadly than intended. So before any span leaves the function, it's re-checked against the exact ID that was asked for:

# Belt-and-suspenders: only keep spans that exactly match the
# requested trace_id, in case the query above ever matches more
# broadly than intended.
if span and span.trace_id == trace_id:
    spans.append(span)

Sentry's get_trace does the same re-check after calling its native /trace/{id}/ endpoint, and its parser goes one step further: it rejects a trace item that's missing its own trace_id rather than falling back to the requested ID, because substituting the requested ID would make the whole re-verification check a no-op.

No fabricated data on malformed responses. Both backends reject a span outright — return None, log a warning — rather than filling in a placeholder. Datadog's parser has explicit rejection blocks for missing timestamps and for missing service/operation identity, with comments stating why: a substituted now() would silently corrupt trace ordering, a substituted "unknown" would silently merge unrelated spans. Sentry mirrors this and adds a sanity bound on duration — anything past ten years in milliseconds is rejected as almost certainly a clock-skew artifact.

None of this landed in one pass. Two rounds of adversarial CodeRabbit review hit the Datadog backend: the first cites eight actionable findings addressed (HTTPS enforcement, redirect-disabling, exact-ID re-verification, cursor pagination, trace-ID escaping, timestamp rejection, and others); the second, an explicit follow-up after CodeRabbit re-reviewed the code the first round had just introduced, addressed four more — query escaping, response-shape validation, and get_trace's pagination limit handling, the same wording in the commit subject line. A hardening pass can introduce its own gaps; it's worth re-reviewing the fix, not just the original code.

One honest caveat on Sentry: its module docstring says plainly that several response-shape assumptions — exact column names in raw event rows, the shape of Sentry's SerializedTraceItem type, the escaping convention its Discover query syntax expects — were built from published documentation and source reading, not verified against a live Sentry account. That's disclosed in the code, and I'm disclosing it here rather than implying this backend has been battle-tested in production.

The test suites reflect the same weight: tests/test_datadog.py runs 781 lines and tests/test_sentry.py runs 1117, with named tests for exactly the things above — test_datadog_backend_rejects_non_https_url, test_equals_rejects_injection_field, test_escapes_trace_id_in_query, test_requests_full_pagination_capacity, test_malformed_data_field_does_not_crash. That's the difference between a README bullet list and something a future contributor can actually run and watch fail if they break it.

The Decision Not to Touch Jaeger, Tempo, or Traceloop

The part I think is actually the most interesting design decision in the project: I did not retrofit any of that discipline onto the three inherited backends.

Open jaeger.py next to datadog.py and the gap is immediate. Four lines into Jaeger's span parser, the default service name is the literal string "unknown" — if neither the processes map nor the span's own process field resolves a real name, that placeholder ships as-is rather than the span being rejected. traceloop.py has the same shape of problem: it falls back to an empty string for service.name when the field is absent. Neither Jaeger nor Tempo has an HTTPS check anywhere — BaseBackend.__init__ just stores the URL, no scheme validation, so both will happily talk to a plain http:// endpoint. Tempo's TraceQL builder, _filter_to_traceql, splices filter values directly into f-strings with no escaping function anywhere in the file — the exact pattern _escape_dd_query_value and _escape_sentry_query_value exist to close off.

I could have quietly left that gap unaddressed and let the README imply the hardening covers "the backends." I could also have spent a week retrofitting Jaeger, Tempo, and Traceloop to match. I did neither. The README states the position directly:

"The three backends inherited from upstream — Jaeger, Tempo, Traceloop — predate this discipline and haven't been retrofitted; that's deliberate scope discipline, not an oversight, mirroring this project's own precedent of not reaching into shared/inherited code without full regression coverage for it."

There's a specific kind of dishonesty in open source that's more common than outright lying: a security-adjacent hardening pass described as covering "the project" when it actually covers the two files someone happened to be working in that week. The work that got done, got done — but the framing implies a blanket guarantee that was never earned. The honest alternative isn't silence about the gap, and it isn't burning a week retrofitting code you don't have the depth of intuition for well enough to trust the retrofit. It's saying plainly which parts got the bar and which didn't, and why. "These three predate this discipline" is a true, checkable statement — I just checked it against the actual source above. "Zero known issues" would not have been.

That "full regression coverage" framing matters because Jaeger, Tempo, and Traceloop are code I didn't write and don't have the same depth of intuition for. Reaching in to add HTTPS checks or escaping without a test suite built to catch the ways I could break it quietly is how a regression goes unnoticed until a production trace query starts silently dropping spans — which is exactly what happened in a different corner of the codebase.

The Bugs That Taught Me the Hard Way

The hardening bar on Datadog and Sentry didn't come from first principles. It came partly from bugs I hit and fixed in the inherited code, which is why I trust the no-fabrication rule enough to enforce it strictly on new code.

The sharpest one: any span carrying a gen_ai.response.finish_reasons value — any LLM span representing a completed response — was silently disappearing from every query result on both Jaeger and Tempo. Not erroring, disappearing. I found it during a live end-to-end dry run against a real Jaeger container and the actual published PyPI package, not a unit test. The root cause was a type mismatch: finish_reasons is typed list[str] in attributes.py's SpanAttributes model, but Jaeger hands the value over as a JSON-encoded string, and Tempo's OTLP parser was stringifying the raw arrayValue structure into an unparseable repr instead of extracting the actual elements. Both shapes failed Pydantic validation, and both backends wrapped span construction in a broad except that returned None — so a parse failure and "no span here" were indistinguishable. The fix added a field_validator that coerces any of those shapes into a real list without raising, and rewrote Tempo's arrayValue handling to pull each element's scalar value by key presence rather than truthiness, so a real 0 or False doesn't get dropped along with it.

That bug is why the no-fabrication rule on Datadog and Sentry isn't theoretical caution — it's a lesson learned on the backends that predate it, and it's a large part of why I trust the new backends' stricter behavior (reject the span, log why) over the older ones' looser one (silently return nothing).

The rest of this arc is smaller but real. An earlier fix replaced sequential get_trace round trips with a single asyncio.gather call across the four tools that hydrate full traces after a search — expensive_traces.py, list_models.py, model_stats.py, slow_traces.py — turning N sequential calls into one concurrent batch, with a regression test asserting the maximum in-flight call count equals the batch size. The same commit deleted a dead else 0 branch in model_stats.py's success-rate calculation, left over from before an earlier guard made it unreachable. On the CI side, v0.3.0 closed off a pattern in docker-publish.yml where ${{ github.ref }} and ${{ github.run_id }} were textually inlined into shell script text rather than passed through environment variables — not an exploited vulnerability here, since neither value is attacker-influenceable in this step, but a known-dangerous pattern worth closing regardless. The same release removed a step from that workflow that was supposed to flip the published GHCR image to public visibility but always reported success even though its gh api calls 404'd under the default GITHUB_TOKEN, chained with || into a trailing echo so the exit code was always zero. Green CI, private package, until someone checked; I removed the fake step and documented the one-time manual flip instead of fake-automating a fix. And start_locally.sh had a live Traceloop API key hardcoded as the active default — inherited from before the fork point, still live on the upstream public repo, not mine to rotate — so the fix was scoped honestly: switch the default to the credential-free Jaeger backend, without claiming to have remediated an exposure that predates this repository. The Docker image now publishes to GHCR on every version tag, for linux/amd64 and linux/arm64.

The Fork, the Pivot, and a Changelog Anomaly

tracehub-mcp began as traceloop/opentelemetry-mcp-server, Apache 2.0, with full attribution and fork history preserved in NOTICE — created via git clone plus git remote, not by copying files into a fresh repository, so the original commit history stays visible instead of collapsing into one "initial commit." The first commit landed 2025-11-02, with a first release, v0.2.0, on 2025-11-17. There was a long gap, then three commits on 2026-09-12 completed the rename and pivot to mcpsmiths/tracehub-mcp: a fork-setup commit adding NOTICE, a rename commit, and a follow-up, fix: complete the tracehub-mcp rename, remove old-org CI dependencies, closing out roughly 90 stale references to the old name across the README, CLAUDE.md, the Dockerfile, start_locally.sh, server.py, and CI, and dropping CI's dependency on the upstream org's private infrastructure — a self-hosted runner label, a private composite Trivy action, a GitHub App for version bumps — in favor of GitHub-hosted runners and the repo's own token and git identity.

That pivot also explains something you'll notice scrolling CHANGELOG.md: two separate v0.2.0/v0.2.1/v0.2.2 entries months apart, one set from November 2025 through February 2026, another from September 2026. Resetting pyproject.toml's version to 0.1.0 during the rename left the Commitizen version files still pointing at the old 0.2.2, so the automated bumper started counting up again on the new codebase — a real, mechanical drift, confirmed by dry-running the actual bump logic rather than assumed, and fixed in a commit that also caught an unrelated bug where a separate "configure git identity" CI step did nothing, because the commitizen action's own entrypoint re-runs git config with its own defaults right before committing. I'd rather the changelog look odd and be explained than look clean because I quietly rewrote history.

What This Actually Is

I ran the test suite directly against this checkout rather than trusting the README: 467 passed, 2 skipped, 87% overall coverage — the README currently says 458, which is stale, and I'm correcting it here instead of repeating it. The two skips are environment-dependent integration tests needing cassette fixture data not present in this run, not disabled tests. There are 5 open pull requests right now, every one a Dependabot bump to a GitHub Action's minor or patch version — docker/login-action, actions/cache, sticky-pull-request-comment, astral-sh/setup-uv, trivy-action — routine and low-risk. It's listed on the official MCP registry as io.github.mcpsmiths/tracehub-mcp and carries a glama.ai score badge. It has 0 GitHub stars as of today.

What I'm actually claiming is narrower than "production-hardened observability platform," and more useful for being narrow: two new backends built to a specific, checkable security bar, three inherited backends explicitly not held to that bar yet, a real data-loss bug found and fixed the hard way, and a test suite I ran myself before writing any of these numbers down. If you're running an LLM application on Datadog or Sentry and want your AI assistant to query the actual trace data instead of you pasting it in by hand, that's what this is for. If you're on Jaeger, Tempo, or Traceloop, it works today, on the discipline the fork already had — just don't assume it has the newer bar until I've said it does.

Why I Built cost-guard-mcp: Pre-Flight Cost Guardrails for AI Agents Talking to Data Warehouses

SAI RAM — Sun, 13 Sep 2026 14:02:20 +0000

The query that shouldn't have run

Give an AI agent a warehouse MCP server and, sooner or later, it will write a query that scans a full multi-terabyte table and quietly runs up a bill in the hundreds of dollars because it forgot a partition filter, or it will SELECT * from something it thought was small and get back four million rows straight into its own context window. Neither of these requires a malicious agent or a bad prompt. It just requires an agent doing what agents do: writing plausible SQL against a schema it partially understands, then running it. A human DBA would eyeball the query, guess the table size, maybe run EXPLAIN first out of habit. An agent given a generic "run this SQL" tool has no equivalent instinct, and as far as I can tell, no warehouse MCP server on the market tells the agent what a query will cost or how much data it will return before that query actually executes.

That gap is the entire reason cost-guard-mcp exists. It is a Model Context Protocol server that sits in front of BigQuery and Snowflake and gives an agent three tools: describe_engine_capabilities(engine) to learn what an engine can and can't tell you, estimate_query_cost(engine, sql, warehouse) to get a cost and byte estimate before running anything, and run_query_bounded(engine, sql, max_bytes_billed, max_rows, max_estimated_cost_usd) to actually execute a query with those caps enforced. It is a genuinely small project — 15 Python files, 715 lines under src/cost_guard_mcp/ — and it is genuinely new. The first commit landed 2026-09-12 at 22:23 IST, the feature-complete v1 landed a few hours later at 02:27, v0.1.0 shipped that same day, and v0.1.1 — the hardening pass I'll get to below — shipped today, 2026-09-13. The whole history of this project fits inside about eighteen hours. I'm not going to pretend otherwise; there's no multi-month backstory here, just a focused tool built fast and then immediately hardened.

An estimate that tells you how much to trust itself

The design decision I care about most in this codebase isn't the warehouse integration — it's that estimate_query_cost never returns a bare number. Every CostEstimate it produces carries an accuracy_tier field, and that field is not a comment or a README promise, it's a required field on the Pydantic model in src/cost_guard_mcp/types.py with no default. You cannot construct a CostEstimate without deciding what tier it belongs to. AGENTS.md in the repo states the intent behind this directly: it's "enforced structurally by the Pydantic model, not by convention." That distinction matters more than it sounds. A convention is something a future contributor forgets. A required field with no default is something the type checker refuses to let you skip.

There are three tiers: PRECISE, UPPER_BOUND, and HEURISTIC. BigQuery is where PRECISE actually happens, and it happens because BigQuery's dry-run API is not a client-side guess — it's a real submission to BigQuery's own query planner. dry_run() in src/cost_guard_mcp/engines/bigquery.py builds a bigquery.QueryJobConfig(dry_run=True, use_query_cache=False), sends it through client.query(), and reads back total_bytes_processed from the real API response. BigQuery validates and fully plans the query server-side without executing or billing it, so the byte figure that comes back is the same figure the real execution would have produced. That's what earns the PRECISE label.

Snowflake gets a structurally different treatment, and the code is explicit about why. explain_estimate() runs EXPLAIN USING JSON {sql}, parses the returned plan, and reads plan['GlobalStats']['bytesAssigned'] as its byte figure — but that number describes what the query planner expects to scan, not what actually gets scanned, and Snowflake's own documentation (quoted directly in a code comment) says runtime plan optimizations "can reduce the number of partitions and bytes scanned." Because of that, explain_estimate() doesn't have a downgrade table the way BigQuery does — it hardcodes AccuracyTier.UPPER_BOUND on line 84, unconditionally, because there is no better tier available to fall from. BigQuery's dry-run and Snowflake's EXPLAIN are answering genuinely different questions: one is "what will this actually cost," the other is "what is the most this should cost."

The place this stops being an abstract distinction and becomes something you can watch happen is inside BigQuery itself. BigQuery's dry-run response includes its own confidence signal — totalBytesProcessedAccuracy — and dry_run() reads it via query_job._properties['statistics']['query']['totalBytesProcessedAccuracy'], defaulting to 'UNKNOWN' if it's missing, then maps it through a fixed table: PRECISE stays PRECISE; LOWER_BOUND, UPPER_BOUND, UNKNOWN, and anything else all fall through to AccuracyTier.UPPER_BOUND via the dict's own default. A unit test, test_dry_run_treats_unrecognized_accuracy_value_as_upper_bound, feeds it a value BigQuery has never returned before — 'SOME_FUTURE_VALUE' — and asserts the tool still downgrades safely rather than defaulting to PRECISE. The practical consequence: run the exact same SQL text against an ordinary settled table and you get PRECISE with an empty caveats list. Run that identical SQL against a table with a pending streaming buffer, or a wildcard, or a federated source, and the tool automatically flips to UPPER_BOUND and attaches a caveat quoting BigQuery's own raw accuracy string verbatim — something like "BigQuery reported this estimate's own accuracy as 'LOWER_BOUND', not PRECISE — treating it conservatively as UPPER_BOUND." Nothing about the query changed. Only BigQuery's own confidence in its byte count changed, and the tool surfaces that shift instead of quietly reporting one undifferentiated number both times.

The third tier, HEURISTIC, is reserved for Databricks, and I want to be precise about its status: it exists in the type system — types.py defines the enum value, and the agent-facing docstring in server.py mentions it — but there is no engines/databricks.py, no pricing module, nothing. describe_engine_capabilities('databricks') raises ValueError, and a test asserts exactly that. I deferred Databricks rather than ship it, because a Databricks estimate would necessarily be a heuristic guess with no dry-run and no EXPLAIN-equivalent bound behind it, and shipping a guess dressed up as a real number is precisely the failure mode this whole project exists to prevent. Better to leave the third tier as a load-bearing placeholder in the schema than half-honest in the field.

The server itself bakes the sequencing into its own tool docstrings, not just into documentation: describe_engine_capabilities's docstring tells an agent to call it "before estimate_query_cost or run_query_bounded to understand how much to trust the accuracy_tier on their responses for this engine," and estimate_query_cost's docstring spells out what each tier means in the response itself. That's the actual contract an agent operates against — not marketing copy, code the agent reads.

Refusing is safer than guessing

The second thread running through this project is fail-closed, least-privilege design, and it shows up as a pattern rather than a single feature. The clearest instance is in run_query_bounded. If a caller passes max_estimated_cost_usd but the estimate that came back has estimated_cost_usd=None — which happens for a BigQuery Editions/capacity-billed project, since capacity billing has no fixed dollar-per-byte rate to convert from — the tool doesn't run the query with that cap silently unenforced. It refuses. The check in src/cost_guard_mcp/tools/run_query_bounded.py returns a normal BoundedQueryResult with status="refused" and reason=RefusalReason.COST_CAP_EXCEEDED, plus a plain-English hint explaining exactly why: it couldn't produce a dollar estimate, so it's refusing rather than running uncapped. This is a normal, successful MCP tool result, not an exception and not the MCP error channel — that's a deliberate choice recorded in the project's own ADR #4, so a caller can distinguish "your query is too expensive" from "your MCP client is broken" cleanly.

Two other examples of the same instinct: Snowflake connections require an explicit SNOWFLAKE_ROLE environment variable with no default, and config.py raises a ConfigError if it's unset, specifically to prevent ever falling back to ACCOUNTADMIN. And every function in both engine modules that touches snowflake.connector or the BigQuery client libraries is wrapped by a sanitize_exceptions decorator, which regex-redacts five categories of secret material — passwords, private keys and PEM blocks, tokens/API keys, and user:password@host URIs — from any exception before it can reach a caller or a log line, because snowflake-connector-python has a documented history of leaking credentials into raw exception text.

The part of this story I find most worth being honest about is that two of the security fixes weren't found by a later audit — they were found and closed the same day the vulnerable code shipped. Git history has the receipts: a commit titled fix(security): validate Snowflake warehouse name before USE WAREHOUSE, because the warehouse parameter — caller-supplied, same trust level as the SQL itself — was being interpolated straight into USE WAREHOUSE {warehouse}. That statement is executed directly against the live Snowflake session via cur.execute(), so an unvalidated warehouse name wasn't just a cosmetic risk: a caller could have appended a semicolon and a second statement after the intended identifier, turning a parameter meant to just pick a compute resource into a general SQL execution primitive scoped by whatever the connection's role could already reach. The second commit, fix(bigquery): validate project ID before interpolating into API filter, whose own message says it was "flagged by automated security review after the previous commit introduced the query filter," closes a related but structurally different gap: the GCP project ID isn't actually reachable from an MCP tool parameter today, but the code treats the identifier as if it could be — validated on the theory that anything string-interpolated into an API filter expression deserves the same allowlist treatment as untrusted input, not just the fields a caller happens to be able to reach right now. Both are closed with allowlist regex validators — _validate_warehouse() and _validate_project_id() — that fail loudly with a ValueError (itself sanitized) rather than let a malformed identifier reach a query-language context. Both incidents are written up together as ADR #6 in DECISIONS.md, which turns them into a standing rule for any engine this project adds in the future.

One more mechanism worth naming because it's small and clever: row-bounded execution on both engines wraps the caller's SQL as SELECT * FROM (<sql>) AS cost_guard_row_cap LIMIT max_rows + 1 before fetching. The +1 is the whole trick — fetch exactly max_rows and you can't tell "there were exactly that many rows" apart from "there were more, and we truncated." Fetch more than max_rows + 1 and the row cap stops doing its job of keeping the fetch itself bounded. This pattern trips Ruff's bandit-style SQL-injection ruleset (S608) in exactly two places, once per engine, and both are suppressed inline with # noqa: S608 because the string interpolation is the caller's own already-validated SQL being wrapped, not an unvalidated identifier.

Hardening the guardrail tool, immediately

I shipped v0.1.0 on 2026-09-12. One day later, before the project had a single GitHub star, I ran a security pass against my own new code and shipped v0.1.1 with PyPI Trusted Publishing (OIDC via id-token: write, scoped to the pypi environment, with zero stored API token anywhere in release.yml), CodeQL scanning on a weekly cron, OpenSSF Scorecard with SARIF results uploaded to GitHub's Security tab, secret scanning, push protection, Dependabot security updates, and branch protection requiring CI to pass on main. All of that is free for a public repository and was simply off for the first day. Concretely: secret scanning and push protection are what would catch an accidentally committed API key or private key before it ever settles into git history, not after; Dependabot security updates are what would catch a CVE landing in a transitive dependency next month, long after I've stopped actively watching this repo. OpenSSF Scorecard in particular runs a battery of named checks against the repo's own supply-chain hygiene — Dangerous-Workflow for GitHub Actions patterns that let untrusted input execute privileged code, Branch-Protection for exactly the main-branch CI gate I just turned on, and Pinned-Dependencies for whether third-party actions are pinned to a mutable tag or a fixed SHA. It felt right to treat the guardrail tool itself as something that needed guardrails, and to do it before the project attracted any real attention rather than after.

The CI structure separates required from advisory checks rather than treating everything as one gate: tests.yml runs Ruff, mypy, and the unit suite with --cov-fail-under=80, and that's what actually blocks a merge to main. audit.yml runs pip-audit but is deliberately not a required check — its own header comment explains the reasoning: a new CVE in an already-approved, unrelated dependency shouldn't block an unrelated PR from merging. integration.yml is separate again, triggered only by workflow_dispatch or a weekly schedule, and runs the five tests under tests/integration/ against real BigQuery and Snowflake credentials in a live-integration environment. Those five tests are not part of the 60 unit tests I count as the project's real coverage number — I ran pytest tests/ --ignore=tests/integration directly against this checkout and got 60 passed, 0 failed, and one asyncio deprecation warning coming from an early bootstrap test in tests/unit/test_server.py that still calls asyncio.get_event_loop() and says in its own comment that it "documents current state" pending cleanup — a small piece of unfinished housekeeping from the first day, not noise from an external dependency. The integration tests need cloud credentials I'm not going to hand to a CI runner on every PR, so they stay gated and scheduled instead.

DECISIONS.md also documents a bug that's worth naming precisely because it's the same failure class as the fail-closed cap-check I described above: the original run_query_bounded logic used a flat boolean chain that fell through to "allow" whenever a cap was requested but its matching estimate field was None. An automated review caught it. The fix — refuse rather than silently execute unenforced — is now the standing convention for the whole tool.

What this doesn't do yet, on purpose

I'd rather state the gaps plainly than let anyone find them the hard way. Databricks isn't supported — the HEURISTIC tier is a type-level placeholder with zero engine code behind it, deferred past v1 because I'm not willing to ship a guess dressed as a measurement. Snowflake's UPPER_BOUND estimate excludes Cortex AI Function ("AI Credits") cost entirely — EXPLAIN's bytesAssigned reflects warehouse compute and scan, not AI-inference calls embedded in the SQL, and both the static capability lookup and the live caveat on every Snowflake estimate say so. And BigQuery projects on Editions or capacity billing get a byte estimate with no dollar figure at all, because capacity billing charges for reserved slot-hours, not bytes scanned — there's no fixed dollar-per-byte rate to convert with, so the tool declines to fabricate one rather than print a number that would be fiction.

None of these are bugs waiting for a fix. They're the same accuracy-tier discipline applied to the project's own documentation that the project applies to its output: say what you don't know instead of rounding it up to something confident-sounding. cost-guard-mcp is two days old, has one GitHub star, zero open issues, and I built it because I got tired of agents finding out a query was expensive after it already ran. An estimate that admits what it doesn't know is more useful, and safer, than one that guesses and calls itself precise — that's the whole idea, and it's the same idea whether you're reading it off a CostEstimate object or off this changelog.

trelix v3.2.2 to v3.2.5: The Source Tree Was Fine. The Published Package Wasn't.

SAI RAM — Sun, 06 Sep 2026 12:29:30 +0000

Run this against the real, published image and watch it fail:

docker run --rm --entrypoint trelix-mcp ghcr.io/sairam0424/trelix:3.2.1 --version

Exit code 127. Not a crash inside trelix-mcp, not a stack trace, not a permissions error — 127 is the shell's own way of saying the binary you asked for does not exist. And it didn't. The console script trelix-mcp is supposed to install as part of every trelix package was simply absent from the image, on both the slim tag and the -local tag, for the entire life of the 3.2.1 release. Every unit test in the suite was green. Every line of source that builds trelix-mcp was correct. The thing a user would actually get from docker pull did not have the binary its own --version flag implies exists.

This article covers four releases — v3.2.2, v3.2.3, v3.2.4, and v3.2.5 — spanning 173 commits and 88 changed files since v3.2.1, which is where the last article in this series left off. That one was about tests that pass without exercising the code they claim to cover: a MagicMock standing in for a real embedder, an all-ones attention mask that makes masked and unmasked math identical, a unit test that asserted a bug as its own specification. This one, on the heels of the mutation-testing push that closed out that arc, is about a different and in some ways more uncomfortable failure mode: tests that pass while exercising the wrong artifact entirely. A green pytest run against src/ says nothing about whether the wheel on PyPI, the image on GHCR, or the binary on the GitHub Releases page actually does what it claims. Those are three separate build products, built by three separate pipelines, and none of trelix's 4,353 collected unit tests had ever touched any of them directly. v3.2.2 through v3.2.4 is the story of finding that gap and closing it with an actual gate, not a promise to be more careful next time. v3.2.5 is a short postscript proving the discipline stuck.

The Docker image that shipped without its own server

The 127 above wasn't hypothetical or reconstructed after the fact — it's the literal command a human ran by hand against the real published v3.2.1 image, now baked verbatim into scripts/verify_release.py's Docker check with a comment explaining why: "this exact command returned exit 127 on the published 3.2.1 image before trelix-mcp was added to the Dockerfile." Root cause was doubly blocked. The Dockerfile's builder stage never had a COPY packages/trelix-mcp/ line — it only ever copied and installed core trelix. And even if someone had added that line, the repo's .dockerignore had a single bare entry, packages/, that would have silently excluded the whole directory from the build context anyway. Two independent gaps, each individually sufficient to explain the missing binary, both present at once — the kind of thing an ordinary Dockerfile-diff review would not catch, since the second gap lives in a different file entirely. The fix, across commits aabaa14 and 08233f8, rewrites .dockerignore to packages/* followed by !packages/trelix-mcp, and adds the missing COPY packages/trelix-mcp/ packages/trelix-mcp/ plus its install into the same pip install invocation as core. Bundling it costs roughly 84MB on the slim tag by the release's own measurement — trelix-mcp's dependencies are mcp, fastmcp, and trelix itself, nothing that pulls in torch — but that 84MB figure is reported prose in the changelog, not a number any script in this repo computes or asserts. The mechanism that makes it plausible (no ML dependency in trelix-mcp's own pyproject.toml) is verifiable; the exact delta isn't.

The second 3.2.2 bug lived one layer up, in the console script itself. Before the fix, trelix-mcp's main() never inspected sys.argv at all — not incorrectly, not partially, not at all. There was no import argparse, no reference to sys.argv, nothing. Running trelix-mcp --help from a real shell didn't print usage; it silently launched the actual MCP stdio server and sat waiting for JSON-RPC input on stdin, the exact opposite of every CLI convention a --help flag implies. The fix is small enough to quote in full:

def main() -> None:
    """Entry point for the trelix-mcp server (stdio transport).

    Parses argv only for --help/--version/unknown-flag rejection — the normal path (no
    args, launched by an MCP client's server config) falls straight through to running
    the server, unchanged from before this parser existed.
    """
    parser = argparse.ArgumentParser(
        prog="trelix-mcp",
        description="MCP server for trelix — semantic code search over stdio.",
    )
    parser.add_argument("--version", action="version", version=f"trelix-mcp {__version__}")
    parser.parse_args()

Both 3.2.2 bugs share the same shape: something was correct in principle and broken in the specific packaging or invocation path a real user takes. Neither one could have failed a unit test, because no unit test in the suite ever installed a wheel, pulled an image, or ran a console script as a subprocess against sys.argv. They were found by a manual production-verification pass — installing and running the actual shipped PyPI packages and Docker images rather than the source tree — and that pass had, by this point, been run by hand twice: once for 3.2.1, once for 3.2.2. Running it a third time by hand was the thing the rest of this arc exists to stop doing.

The wildcard leak and the response that mattered more than the fix

v3.2.3's actual bug is narrow and specific: Database.bm25_search's path_filter branch, plus three path_filter-scoped queries in src/trelix/retrieval/grep_search.py, built a SQL LIKE pattern directly from a caller-supplied path without escaping it. LIKE treats % and _ as wildcards inside the pattern value itself, which has nothing to do with SQL injection — every one of these queries was already parameterized with ? — and everything to do with LIKE's own semantics. A path_filter of src_auth matches not just the directory src_auth but also, say, srcXauth, because the underscore in the filter is read as "any single character," not a literal underscore. Given how common underscores are in real directory and file names (test_utils.py, my_function), this wasn't an edge case; it was a routine collision waiting for the right two sibling paths to exist in the same repository. The fix lives in escape_like_pattern() at src/trelix/store/db.py:291, whose docstring is worth quoting because it states the distinction precisely:

def escape_like_pattern(value: str) -> str:
    """Escape `%`, `_`, and `\\` in `value` so it is safe as a SQL LIKE pattern segment.

    LIKE treats `%` and `_` as wildcards in the PATTERN VALUE itself — this has nothing
    to do with SQL injection (the callers here already use `?` parameterization) and
    everything to do with LIKE's own semantics.
    """

The sibling method get_index_metadata_with_prefix had already been doing this correctly for a while, pairing escape_like_pattern() with an explicit ESCAPE '\\' clause — so the fix wasn't a novel idiom, it was applying an already-proven pattern to the four call sites that had been missed. It's also not a total fix: grep_search.py's own docstring flags a second, separate, still-open instance of the same defect class on the symbol-name prefix match in _name_search, explicitly out of scope for this release. I'd rather state that than let the fix read as more complete than it is.

What makes v3.2.3 matter for this article isn't the bug — it's what came after it. The team's own reasoning, written into the new test suite's module docstring, is that no existing test, including Click's CliRunner-style in-process mocking and the MCP SDK's own official in-memory Client, is structurally capable of seeing a defect at the real process boundary — the exact class that let trelix-mcp's --help bug ship in 3.2.1. In-memory transports never cross an actual OS process boundary; they can't see a console script that isn't installed, or a subprocess that hangs instead of exiting. So tests/e2e/test_mcp_stdio_e2e.py spawns trelix-mcp as a genuine child process, located via shutil.which("trelix-mcp") — the test fails loudly if the binary isn't on PATH, not just importable from src/ — and talks to it over real stdin/stdout JSON-RPC via the MCP SDK's stdio_client. A companion test, test_pypi_dist_install_e2e.py, builds real wheels for all four published packages, installs each into a fresh venv, and asserts the installed __version__ matches the working tree — never editable, never a live PyPI pull, by explicit design.

That test suite only matters if something forces it to run before a release goes out. In .github/workflows/release.yml, the publish job's dependencies now include a new smoke-test-built-artifacts job that downloads the just-built wheel artifacts, installs them with pip install — never editable — and runs the e2e suite against that install before publish is permitted to execute. It is a real edge in the workflow's dependency graph, not a comment or a convention. Two smaller CI extensions round it out: ci.yml's Docker job now runs docker run --rm --entrypoint trelix-mcp trelix:ci-test --version against every built image, and helm-lint.yml asserts the rendered Helm chart's image tag matches Chart.yaml's appVersion. Both are the kind of cheap check that would have caught a 3.2.1-shaped regression on the next PR instead of the next production audit.

Automating the verification instead of re-typing it

v3.2.4's capstone answers a question the last two releases raised without answering: once you've written the checks, who runs them, and when? Before this release, the answer was a person, watching the GitHub Actions tab for both the Release and Docker Publish workflows to finish, then running scripts/verify_release.py by hand and reading its output. .github/workflows/verify-release.yml replaces that with a workflow that listens for workflow_run completion events from both Release and Docker Publish. Because that event fires twice — once per upstream workflow finishing — the job does a single, non-blocking check via the GitHub CLI (gh run list --workflow "$wf" --branch "$tag") to see whether the other workflow has also gone green for the same tag; if not, it exits cleanly rather than polling, and whichever of the two triggering runs happens to fire second is the one that proceeds. Once both are confirmed green, it runs the same scripts/verify_release.py script a human used to run by hand — PyPI installs into fresh venvs for all four packages, Docker images smoke-tested with the exact --entrypoint trelix-mcp command that returned 127 on 3.2.1, a Helm chart checked out into an isolated worktree and rendered across all three backends, a GitHub Release binary actually executed, and a pip-audit plus wheel-content secret scan — and posts its own pass/fail summary as a workflow run. CONTRIBUTING.md still documents the manual command for ad-hoc re-verification, which matters, because automating the check doesn't mean giving up the ability to run it by hand when you're debugging the check itself.

The other audit: when a symbol's nickname collides with someone else's

Running in parallel with the artifact-verification work, v3.2.4 also shipped a distinct methodology: an audit for silent symbol-collision, not artifact drift. The pattern, once named, shows up in four unrelated corners of the codebase: something that should carry a full, unique address instead answers to a bare nickname, and when two different things share that nickname, one of them silently disappears or gets misattributed.

The clearest instance is in Java. Before the fix, every nested class got a bare qualified_name — a class named Config nested inside ServerConfig and a different class also named Config nested inside ClientConfig both indexed under the identical name Config, with parent_id set to None regardless of which outer class actually enclosed them. Re-indexing one silently left the other's stale row permanently in the database. The fix threads an outer_qualified_name recursively through the class walk in java.py, so a nested class now gets f"{outer_qualified_name}.{name}" and a real parent_id. The Java record-component bug is a different kind of mistake in the same file: the extractor checked for tree-sitter node types record_parameters (the container) and record_component (the item) that the installed grammar simply doesn't emit — it emits formal_parameters and formal_parameter — so every record in the corpus indexed zero fields, silently, because the code was written against a node vocabulary that never existed in the grammar it ran against.

Rust has the identical collision shape one file over. A function defined inside mod inner { } previously got the same bare qualified_name as any other function of the same name elsewhere in the file — the fix in rust.py threads a module_path through the module walk and rewrites the qualified name to inner::nested for anything defined inside it. A separate Rust fix corrected the brace-import flattener, which checked for node types use_tree_list and use_tree that don't exist in the installed grammar either — the real node types are scoped_use_list and use_list — so the dominant Rust import form, use foo::{Alpha, Beta}, fell through to a generic fallback that stored the literal text "{Alpha, Beta}" as a single bogus import name, making every brace-imported symbol invisible to import-graph queries.

The same audit surfaced the collision shape outside the parsers entirely. AgentLoop._do_get_symbol, when asked for an exact qualified name that didn't resolve, used to fall back to an arbitrary bare-name match — silently handing the agent a different symbol's body than the one it asked for, with no error signal. The fix requires exactly one exact match or reports "not found." Federation's make_scip_symbol_id() used to hash a cross-repo symbol identity from just (package, version, qualified_name), omitting file_path — meaning two different files in the same package defining def main() hashed to the identical 16-character id, and INSERT OR IGNORE against that id as a primary key silently dropped the second file's row rather than erroring. It's the most explicit statement of the whole pattern in the release: an identity hash that omitted one required scoping field, and a database that dropped the collision without complaint.

Not every bug the same audit pass caught fits that collision shape, and I'd rather say so than force the frame. The chunker's token_count bug was a stale cached value — computed against the pre-truncation text instead of the actual, truncated chunk that got stored. bm25's stop-word handling had a related but distinct problem: a query made entirely of stop words was supposed to hit the empty-query fallback and return the FTS5 sentinel path, but the stop-word filter ran after that check instead of before it, so a stop-words-only query was treated as a normal, non-empty query instead. The walker's fix normalizes rel_path to NFC via unicodedata.normalize, because a single real file could take two different Unicode-normalization forms across walks (NFD from an HFS+ filesystem versus NFC elsewhere), fracturing one file's identity into what change detection read as a phantom delete-and-add. The graph community-detection fix removed a force-cast to int on node ids that crashed with ValueError on any ticket-linked repo, because ticket node ids are strings by design. And the CLI's update_index command now actually checks result["status"] and exits 1 on failure, instead of printing the JSON error payload and returning exit 0 regardless. Different defect shapes, same audit pass, same release — worth naming honestly as a second act rather than folding into a pattern it doesn't share.

A short coda: the binary that gave advice it couldn't act on

v3.2.5 is one fix, and it's a fitting close because it was caught by the same discipline the verify-release capstone exists to encode: someone read the error message the real, shipped v3.2.4 binary actually prints, in production, rather than the source that built it. The standalone GitHub Release binary's local embedder — used when sentence-transformers isn't available — told users to run pip install 'trelix[local]'. That advice is correct for the pip-installed package. It is actively useless for the frozen PyInstaller binary, which is built with sentence-transformers, torch, and the rest of the ML stack deliberately excluded to keep it small, and which never consults the host's Python or pip environment at all — installing anything on the host has zero effect on a binary that doesn't read it. The fix, in src/trelix/embedder/base.py, checks getattr(sys, "frozen", False) — the standard attribute PyInstaller sets — and gives the frozen binary a message pointing at an API-backed provider or the Python package instead, leaving the pip-installed package's message byte-for-byte unchanged. It's covered by a test that asserts the frozen message never contains the string "pip install." One line of misdirection, found the same way the Docker gap and the console-script gap were found ten weeks and four releases earlier: by running the actual thing that ships, not the code that built it.

trelix v3.1.2 to v3.2.1: The Tests That Passed Without Testing Anything

SAI RAM — Wed, 26 Aug 2026 13:53:41 +0000

In FlagEmbedding 1.4.0, from FlagEmbedding import FlagModel is an alias for from .base import BaseEmbedder as FlagModel — the encoder-only base class, whose default pooling_method is "cls" and whose pooling() for that method is one line: return last_hidden_state[:, 0]. BAAI/bge-code-v1 is not an encoder. It is a causal Qwen2 decoder, published with 1_Pooling/config.json declaring pooling_mode_lasttoken: true and pooling_mode_cls_token: false. Position 0 of a causal decoder cannot attend forward. So the CLS vector bge-code was computing depended on exactly one token and nothing else that followed it.

I measured what that means with a randomly initialized Qwen2Model pooled by FlagEmbedding's real pooling(): two sequences identical at token 0 and different in every token after it came back with cosine similarity 1.0 and max absolute difference 0.0 — bitwise identical. The same hidden states through the real last_token_pool gave cosine 0.10; through mean, 0.65. Across 200 resamples of every token after position 0, position 0's output never moved once. And because encode_queries prefixes every query with the same instruction, token 0 is that shared prefix for every query trelix ever sends — meaning every query embedding bge-code produced was identical to every other one, full stop. Five distinct code chunks sharing a first token collapsed into a single vector, against five distinct vectors from last_token_pool on the same input.

This shipped in 28 tagged releases — every tag from v2.0.0 through v3.1.6 contains the commit that introduced the embedder — and stayed green through every one of them because tests/unit/test_embedder_bge.py replaces FlagModel with a MagicMock. A MagicMock answers to any attribute you ask of it. It never once called the real pooling(), so a defect that destroys every query embedding a provider produces never had a chance to fail a test.

v3.1.6 fixed a real, adjacent bug: BGECodeEmbedder.dimension called a method FlagEmbedding has never had, raising AttributeError before an index could even start, and the dimension it fell back to (768) was wrong anyway — the model emits 1536. That fix was correct and it shipped correctly. But removing the raise made a path reachable that had never been reachable before: the pooling defect. The v3.1.6 changelog entry read as though bge-code now worked. It didn't, and one release later, v3.1.7 said so in public, in the same file: "Retracted: bge-code does not 'work now'." I don't read that as an embarrassment. A team that ships a wrong claim and then corrects it in the next release, with the receipts, is doing exactly what a correctness process is for. The alternative — quietly letting the framing stand — is the actual failure mode.

Eight releases, one arc

This is the story of eight tagged releases — v3.1.2, v3.1.3, v3.1.4, v3.1.5, v3.1.6, v3.1.7, v3.2.0, and v3.2.1 — spanning 190 commits and 324 changed files since v3.1.1, where the last article in this series left off. v3.2.1 is the current shipping release: tagged, dated 2026-08-26, live on PyPI, installable today with pip install trelix==3.2.1. The unit suite sits at 4,376 tests collected on this exact checkout. trelix itself is about nine weeks old — the first commit landed 2026-06-25 — which matters for how to read what follows: this isn't a decade of debt being paid down, it's a project auditing itself hard and often, early, while the cost of finding a defect is still one release instead of ten.

Unlike the v3.0.0 span this series already covered — audit trail, OIDC SSO, the VS Code extension, context compression, model-aware budgeting, extended thinking — this span has no comparable feature drop. It is, almost entirely, a correctness and audit arc. I'm not going to apologize for that framing or bury it under a feature list that doesn't fit the material. The interesting thing about these eight releases isn't what got built; it's what got caught, and specifically the shape of what got caught, which repeats often enough across independent parts of the codebase that it deserves to be named as a pattern rather than four unrelated bug reports.

The pattern: a green suite that never touched the defect

Here is the pattern, stated once so the four instances below don't have to keep re-deriving it: a test can pass without ever exercising the behavior it claims to cover. That's not the same claim as "the tests were bad" or "coverage was low." These tests ran. They asserted things. They turned green in CI, release after release. And in every one of the four cases below, the reason they turned green is diagnosable and specific — not vague test debt, but one of four concrete mechanisms.

bge-code's pooling defect survived because a MagicMock answers any attribute asked of it, so no test ever called the real pooling() method that carried the bug. The sparse-vector padding defect, below, survived because every existing fake for the tokenizer returned an all-ones attention mask by construction, which makes masked and unmasked aggregation produce identical numbers regardless of whether the masking code is even present. The intent_hint dispatch bug survived because a unit test asserted the observed, buggy output as the correct value — it had to be deleted, not fixed, once the real behavior was understood. And the federated search-all deduplication bug survived because the existing test happened to use the one input distribution — globally unique row identifiers — that cannot trigger a collision, out of all the distributions that could.

Four independent parts of the codebase — an embedder, a different embedder's aggregation math, a retrieval planner, and a federation layer — produced the same failure mode by four different routes. That repetition is why v3.2.1 exists, and why I'm leading with it rather than a version-by-version changelog walk.

The sparse leg: clean queries scored against contaminated documents

SparseEmbedder.embed tokenizes a batch with padding=True, which pads every sequence in that batch out to the length of its longest member, and then aggregates with torch.log(1 + torch.relu(logits)).max(dim=1)) — a max taken over every sequence position, padding included. A masked-language-model head predicts a real, nonzero logit distribution at pad positions too; it doesn't know they're padding. Nothing in the aggregation multiplied by the attention mask to zero those positions back out. grep -c attention_mask src/trelix/embedder/sparse.py returned 0. A tree-wide grep -rln attention_mask src/trelix/ matched no file in the codebase at all.

I measured it against the real naver/splade-v3-distilbert at top_k=128. A 28-token chunk embedded alone, compared against the identical chunk embedded in a batch alongside a 185-token chunk, gained 28 phantom terms and lost 28 real ones at the cutoff — 22% of the stored vector — with 0.478 max weight drift on the terms that survived both runs. The phantom terms are ordinary English words the chunk never contained: where, store, numbers, text, gene, sequence, phrase, messages. The control that pins the cause is the one that matters most here: the same chunk batched with an equal-length chunk, which needs no padding at all, came back bit-identical to the chunk embedded alone. Batching was never the problem. Padding was.

Two things made this worth a minor release rather than a footnote. sparse_embeddings is a persisted table in store/db.py, so the corruption wasn't transient — reindexing the same repository in a different file order, or with a different TRELIX_SPARSE_BATCH_SIZE, rewrote every row with a different, equally wrong answer, and no index was reproducible against itself. And embed_query routes through embed([text]) — a batch of exactly one, which needs no padding — so every query trelix ever issued against the sparse leg was clean, scored against documents that weren't. No amount of query-side tuning could have surfaced or corrected that asymmetry, because the query side was never where the bug lived.

The existing test_sparse_embedder.py had ten tests covering this embedder, and every one of its fakes returned attention_mask=torch.ones(...). An all-ones mask makes the masked and the unmasked aggregation mathematically identical, so all ten tests passed whether or not the masking code existed at all — which it didn't. The replacement, test_sparse_padding_contamination.py, is nine tests, every one mutation-verified: deliberately dropping the mask multiplication from the aggregation fails five of them while leaving the equal-length control passing, which is exactly the signature that localizes the cause to padding rather than to batching in general.

intent_hint: a test that asserted the bug as the spec

intent_hint is an optional parameter on both the REST /search endpoint and the MCP search_code tool, meant to let a caller skip trelix's own LLM-based intent classification and specify a retrieval strategy directly. The hint builder had a routing bug: it stamped the "direct answer" tier onto all eight recognized intent values while separately setting the correct leg strategy for each — but the executor checks the tier first, so the strategy it computed was never read. Every intent_hint value, regardless of which of the eight intents you actually asked for, took the same direct-answer shortcut.

I measured this against trelix's own codebase: all eight intent values returned byte-identical output — 40 README sections, zero code files, and no overlap at all with the actual correct result set for any of them. intent_hint had been silently disabling retrieval since v2.10.0, over both protocols, for anyone who used it.

The reason it took this long to catch is the plainest of the four: a unit test existed that asserted the broken output was the expected one. That's not a gap in coverage — it's a test that actively encodes the bug as the specification. It had to be deleted and replaced with assertions on the actual retrieval outcome, not patched.

search-all: a test that used the one distribution that can't fail

Federated search-all fans a query out across every registered repository and merges the results. The merge deduplicated on each result's per-database autoincrement row id — and every repository's ids start at 1. Two repositories' top hits collide on id 1 unavoidably; first-seen wins the merge, and "first" was whichever repository's thread finished first in the pool, which is nondeterministic. The consequence: an entire repository's results could vanish from a federated search, silently, while the response reported repos_skipped: 0 — nothing told you a repository had been dropped, because nothing had been skipped; its results had simply lost a dedup race.

The existing test for this path passed because it constructed its fixture repositories with globally distinct identifiers — the one input distribution, out of every distribution the real world produces, under which two repositories' row ids cannot collide. The fix keys deduplication on a globally unique identity instead of the raw row id, and the new test deliberately reuses row ids across repositories to force the collision the old test structurally avoided.

How do you know your check actually checks: the audit trail and the self-audit

Two pieces of this arc aren't part of the four-instance spine above, but they belong in the same conversation because they ask the identical question from a different angle: not "does this test exercise the code," but "does this check actually check the thing it claims to."

v3.1.4 found that trelix audit's list, verify, and export commands — the read path over the hash-chained audit trail introduced in v3.0.0 — used the same store constructor as the writer, which runs CREATE TABLE IF NOT EXISTS on open. Point any of the three read commands at a file that wasn't an audit log, and they added the audit schema to it on the spot, then reported "Audit chain intact" at exit 0, because the chain they had just silently created was empty, and an empty chain is, trivially, consistent. One measured case took an 8 KB, one-table SQLite file to 32 KB and five tables just by being read. A CI integrity gate pointed at the wrong path would pass green while mutating the exact file it was supposed to verify. All three commands now open the file read-only (file:<path>?mode=ro), run no DDL, and exit 2 unless the audit schema is already present. It is the same question as the mutation-testing thesis, asked of an integrity check instead of a unit test: a verification step that can succeed against input it was never supposed to accept isn't verifying anything.

v3.1.2 is the plainest statement of the whole arc's thesis, predating the four-instance spine above by weeks. It's a self-audit: the team indexed trelix's own repository with trelix and checked whether every feature the configuration turned on was actually doing anything. The finding, in the release's own words, was "features that were on and doing nothing." File summaries were enabled and the index had zero of them. PageRank boosting had never once fired. taint_flows was empty on a repository semgrep does find real flows in. The query planner had classified all 219 recorded queries as the same one of its eight intents. None of these failures were visible from outside, because each failure path either logged at DEBUG while the CLI runs at WARNING, or was swallowed by a bare except, or reported a number nobody had reason to doubt. That release's own retrospective on its test suite reads almost like an early draft of this article's thesis: the taint parser was verified against a fixture invented to match its own misreading; every planner test supplied no credentials, so none could notice credentials being silently dropped; the eval metric tests used unique IDs only, so none could notice a repeated ID scoring twice.

Briefly, because it doesn't compete for space with any of this: v3.1.3 also shipped real security fixes — a REST API containment check that validated a caller-supplied path against a root the same caller supplied, and a stored XSS in generated graph HTML from an unescaped symbol name or filename. Both are fixed and both matter, but they're a different kind of bug from everything above, and this article already has its through-line without them.

The fix: mutation testing that measures whether a test can fail

v3.2.1 is a test-infrastructure release. It changes no stored vectors, no retrieval behavior, and no CLI surface — its one production fix is that retry.py's status-code extractor imported every supported LLM provider SDK (anthropic, openai, google.genai, boto3, azure) on every retry decision regardless of which backend actually raised, pulling torch transitively into processes that use no LLM feature at all. Everything else in the release is instrumentation, and it exists because coverage percentage — the metric the previous eight releases had been implicitly relying on — cannot answer the question every instance above turned on: can this test actually fail?

The core of it is a new scoped mutation-testing driver, scripts/mutation.py, wrapping mutmut. It runs against a throwaway git worktree rather than the live tree — unscoped mutant generation writes roughly 143 MB of Python that has no business near a commit — and it ratchets a per-module survivor count in scripts/mutation_baseline.json, deliberately never a survivor ratio: a repo-wide ratio moves whenever an unrelated segfault appears or disappears in a part of the codebase mutation testing can't safely reach, and parts of trelix — anywhere a real torch model gets constructed — are exactly that. First real measurements landed for the parser's per-language extractors (split from one coarse scope key into 23 granular ones), store.db, store.vector, indexing.chunker, compression.extractive, and graph. Closing the gaps mutation testing surfaced meant real new coverage: walker extension-map handling, ContextualChunker boundary conditions, graph community-detection survivor patterns, and operator-environment-variable leak/scrub coverage.

Branch coverage is now on (--cov-branch), with per-package floors enforced separately from the unit run itself — and here the changelog for this release actually gets its own mechanism wrong, so it's worth stating plainly: the floors are not in a file called tests/coverage-floors.json, which doesn't exist anywhere in this repository. They're a FLOORS dict hardcoded directly inside scripts/check_coverage_floors.py, checked in CI against a coverage report the unit job writes to coverage-unit.json (pytest tests/unit/ --cov-report=json:coverage-unit.json, followed by python scripts/check_coverage_floors.py coverage-unit.json). The script's own docstring explains why it's a script and not a test: pytest-cov only writes its report at session finish, after every test has already resolved, so a test can't read its own run's coverage — and a check that can only skip when the report file is missing is a green identical to a check that never ran, which is precisely the defect class this whole release exists to close.

The rest of the hardening follows the same logic. The suite is now hermetic: outbound sockets are banned via pytest-socket, every job is timeout-bounded, and the live-LLM integration tests in tests/integration/test_llm_e2e.py now require an explicit TRELIX_LIVE_LLM_TESTS=1 instead of running whenever a .env file happened to be present — cutting roughly half the suite's wall clock along with the credential exposure that came bundled with it. Sixteen known Java and Rust extractor defects are pinned as xfail(strict=True) rather than left silently uncovered, so each one XPASSes loudly, failing the build, if it's ever fixed without an accompanying changelog entry. A new marker taxonomy means a typo in a -m selector can no longer silently collect and pass the entire suite. None of this catches every possible defect, and it doesn't claim to; the mutation baseline is a ratchet, not a finish line. What it does is make "the test passed but never touched the bug" structurally harder to repeat than it was for 28 releases of bge-code, one release of the sparse leg, and however many releases intent_hint and search-all shipped broken before anyone measured directly instead of trusting the green check.

trelix v2.11.0 to v3.1.1: Six Feature Areas, Every One of Them Off By Default

SAI RAM — Sat, 15 Aug 2026 15:33:10 +0000

Seed three events into an audit database, then reach past the application and change one row by hand:

$ sqlite3 audit.db "UPDATE audit_log SET principal='attacker' WHERE id=2"
$ trelix audit verify --db audit.db
Audit chain TAMPERED — first divergent entry id: 2
$ echo $?
1

Delete the newest row instead and it still catches it, naming id 3, even though the surviving rows form a perfectly valid chain. Point it at something SQLite cannot open and it exits 2 rather than 0, because "I could not check" and "I checked and it is clean" must never collapse into the same green build.

None of that existed six releases ago. trelix audit verify is one command out of six feature areas that landed in trelix v3.0.0, and it is the one that most changes what the project is for.

What the major bump actually is

The span from v2.11.0 to v3.1.1 is six releases — v2.11.1, v2.12.0, v3.0.0, v3.0.1, v3.1.0 and v3.1.1, the last of them dated 2026-08-15 — 68 commits, 137 files changed, +19,829/-1,211 lines. v2.11.0 closed out the Jira and Linear connector work, which has its own story. Everything after it is a different kind of release.

v3.0.0 carries six new feature areas: Anthropic extended thinking, a model-aware context budget, a VS Code extension that acts instead of merely displaying, a hash-chained append-only audit trail, OIDC SSO, and query-conditioned context compression. Alongside them, an opt-in FTS5 declaration boost for keyword ranking.

It is a major bump because of scope, not breakage. Every one of those six is additive and off by default: TRELIX_AUDIT_ENABLED=false, TRELIX_OIDC_ENABLED=false, TRELIX_LLM_THINKING_ENABLED=false, TRELIX_RETRIEVAL_COMPRESSION=false, declaration_boost_enabled False, and context_token_budget still the exact 12_000 integer it was in v2.12.0. A default v3.0.0 install assembles context byte-identically to a default v2.12.0 install, and there is a test that proves it rather than a release note that asserts it.

An audit trail you can hand to somebody else

The trail is a separate SQLite file. AuditConfig.resolved_db_path defaults to <cwd>/.trelix/audit.db, never the index database, for one blunt reason: the index is disposable, and an audit trail that disappears on reindex is not an audit trail.

Each row's entry_hash is sha256(prev_hash || canonical_json(content)), canonicalised in ten lines of src/trelix/audit/store.py with sort_keys=True and separators=(",", ":"). Exactly eleven columns are hashed, in a fixed _CONTENT_COLUMNS tuple, and the DB-assigned id is deliberately not among them: a writer cannot hash a value it does not have until after the INSERT.

That exclusion is why row ordering needs a second defence, and it explains the deleted-tail catch from the opening: a hash chain is structurally blind to truncation. An audit_meta table carries a running count and head_hash, upserted inside the same with self._conn: transaction as the insert, and verify_chain checks both against the chain it just walked. audit_log.id is INTEGER PRIMARY KEY AUTOINCREMENT rather than a rowid alias, so a delete-then-refill cannot close the gap it made.

export TRELIX_AUDIT_ENABLED=true turns the trail on and trelix serve ./my-repo starts recording (TRELIX_AUDIT_DB_PATH moves the file if you want it off <cwd>/.trelix/audit.db), trelix audit list -n 50 reads it, and trelix audit export --format ndjson writes one JSON object per line so a SIEM can be fed by Filebeat or Vector. All of it is stdlib sqlite3, 54 tests cover the surface, and the store is fail-open by default (AuditConfig.fail_closed: bool = False) so a full disk is not an outage.

SSO is the identity half, and it is deliberately narrow: trelix is a resource server, not an OIDC client. No /login, no /callback, no code exchange, no refresh. It verifies tokens callers already hold. Three variables stand it up — TRELIX_OIDC_ENABLED=true, TRELIX_OIDC_ISSUER and TRELIX_OIDC_AUDIENCE — on top of pip install 'trelix[sso]'; TRELIX_OIDC_ALGORITHMS defaults to ["RS256", "ES256"] and is asymmetric-only, enforced in OidcVerifier.__init__, again in authenticate against the unverified JWS header before any key resolution, and a third time in jwt.decode. The test that matters skips trelix's own header gate entirely: it signs a token with HS256 using a real 2048-bit public key's PEM as the HMAC secret — the canonical algorithm-confusion forgery — and asserts jwt.decode alone still refuses it. Principal.principal_id is f"{self.subject}@{self.issuer}", never derived from email, because email-keyed identity is an account-takeover primitive, and the JWKS fetch is HTTPS-only, host-pinned and capped at 1 MiB — not hypothetical, since the pre-fix response.read() let a hostile issuer drive RSS from 59 MB to about 662 MB in 0.2 seconds.

The limits are in the docs rather than the marketing. The trail is tamper-evident, not tamper-proof: the chain and the anchor that checks it live in the same file, so anyone with write access can rewrite a row, recompute every subsequent hash and update the anchor in one transaction. There is sha256 and no key. Only the HTTP surface is audited, so an MCP-only deployment produces an empty trail and the agent loop's per-turn calls are invisible. TRELIX_AUDIT_RETENTION_DAYS is declarative only — nothing prunes — and /health is audited, so liveness probes will dominate the row count. This is authentication, not authorization — OIDC groups are captured and stored but enforced nowhere — and there is no SAML, with no plan for it; put a broker in front.

Thinking, and a budget that knows which model you are talking to

Extended thinking is a request parameter, not a model. AnthropicBackend._thinking_kwargs() returns one dict, merged into both messages.create and messages.stream:

return {
    "thinking": {
        "type": "enabled",
        "budget_tokens": self._config.thinking_budget_tokens,
    }
}

No model string in trelix changes. TRELIX_LLM_THINKING_ENABLED defaults False and TRELIX_LLM_THINKING_BUDGET_TOKENS to 4096.

Only one component opts in — RetrievalSynthesizer, at two call sites. The index-time callers, ContextualChunker and FileSummarizer, do not pass the keyword at all; a single global flag would have turned an indexing run into roughly five to ten times the LLM cost to produce reasoning nobody reads.

One consequence is worth stating plainly: enabling thinking forces temperature to 1.0 in both complete() and stream(), overriding whatever the caller passed — and the synthesizer does pass 0.0 and 0.2, so a flag named thinking_enabled silently ends near-deterministic synthesis. Thinking also bills as output tokens with no separate counter, so ChatResponse.output_tokens mixes answer and reasoning.

The model-aware budget is opt-in via a value rather than a flag. context_token_budget: int | None still defaults to 12_000; set it to null, none, auto or empty and Retriever._resolve_effective_budget() returns int(window * context_window_fraction), fraction defaulting to 0.5. resolve_window() is a first-match scan over a hand-ordered 37-entry table in src/trelix/llm/context_windows.py, so gpt-4o-2024-11-20 resolves to 128000. Provider-prefixed Bedrock ids do not resolve: us.anthropic.claude-sonnet-4-20250514-v1:0 returns None and falls back to 12,000 with a WARNING — exactly the default a Bedrock user setting auto was trying to escape.

Raising the budget alone changes very little, and the config docstring says so: rerank_top_n defaults to 15 and top_k_vector/top_k_bm25 to 20/20, which cap the candidate pool before the packer ever sees a budget. TRELIX_RETRIEVAL_SCALE_TOP_K_TO_BUDGET is the explicit opt-in that multiplies top_k_vector and rerank_top_n — but not top_k_bm25 — by effective_budget / 12_000, taking top_k_vector from 20 to 106 on gpt-4o's 64,000. It stays off because it raises per-query cost on three axes at once.

Compression that cannot cost you a result

pack_compressed runs two waves. Wave 1 is the exact pre-existing uncompressed pack, with the compressor never called. Wave 2 walks the same candidate pool and skips anything already selected, so it only ever gets a second look at candidates wave 1 could not fit — the ones the old packer silently dropped. Accepted entries are appended, never substituted, then re-sorted into ranking order. The compressed selection is a strict superset of the uncompressed one, so Recall, MRR and nDCG cannot regress by construction. That is a property you read; no ranking experiment required. It cannot blow the ceiling either: remaining is recomputed per candidate, and anything that will not fit even at _FLOOR_RATIO = 0.01 lands in stats["skipped"].

Citation fidelity is structural. format_compressed_blocks() emits one [Lines a-b] <qualified_name> header per kept span with an explicit # ... N lines elided ... marker in every gap, and each header's text is sliced from the body by the same arithmetic that produced the header. Spans are clamped where they are created and re-filtered again at render time — two independent gates on one invariant, which matters because roughly 35 extractor sites store a truncated body while keeping the full AST span. A header claiming lines its own text does not contain manufactures a confident, wrong citation in a prompt that will quote it verbatim.

ExtractiveCompressor, the default provider, does zero query-time inference — it reads sub-chunk vectors that already exist from index time. The tripwire that lets me claim the off path is unchanged is tests/unit/test_assembler_backcompat_golden.py, which loads the pre-compression assembler out of git show v2.12.0:src/trelix/retrieval/assembler.py and diffs the new one across 8 intents — 177 assertions, including [id(r) for r in new_ctx.results] == [id(r) for r in old_ctx.results], so no reordering hides behind equal text.

The headline "query-conditioned" path is by default lexically conditioned: sub-chunk rows require TRELIX_CHUNKER_MULTI_GRANULARITY=true (default false), are Python-only even then, and need the query embedding already resident in the embedder's LRU. The changelog's performance claim keeps its hedge: roughly 30-60% fewer synthesis input tokens and 15-35% lower latency on a network-API synthesis path, on the compressible intents. That is an expectation, not a benchmark.

The editor surface, and one ranking flag

The VS Code extension is the most visible feature of the release and it needed no server work, because the MCP surface already was the API: it spawns trelix-mcp over stdio and builds all four features from search_code, get_symbol, ask_agent and blast_radius.

Code lenses re-fire on every keystroke, so the performance contract is enforced in the provider rather than documented. provideCodeLenses makes zero MCP calls — its only external call is vscode.executeDocumentSymbolProvider — and only the count-bearing lens reaches MCP, through resolveCodeLens, for lenses VS Code actually paints, keyed on ${docUri}@${docVersion}::${symbolName}. The version component is the difference between a cache and a correctness bug: any edit must miss rather than show a stale dependent count.

Chat became possible because of a bug fix that is the same change as the feature. TrelixMcpClient.ask() used to call getPrompt({ name: "trelix-search" }) and render the joined result as the answer — but getPrompt returns a filled-in prompt template, so what users saw was scaffolding telling them to use the search_code tool. No type checker could have caught it, because both calls return well-formed data and only one of them is an answer. The fix calls ask_agent, which runs the multi-turn ReAct loop and returns {answer, session_id, turn_count}; that session_id is what makes follow-up questions work, so multi-turn chat was structurally impossible until ask() moved. The regression test asserts getPromptCalls === 0.

The limitations sit next to the constants that cause them. const THREAD_KEY = "default" exists because the 1.95 chat API gives the handler no stable thread id, so two chat threads open at once share one agent session. The /explain, /search and /impact slash commands are stateless one-shot calls carrying no session at all. The extension is not on the Marketplace: cd workspace-vscode && npm install && npm run build && npm run package gets you a .vsix. Its manifest is still at 0.3.0, so the file name does not tell you which feature set it contains.

The declaration boost is the smallest feature in the release and the easiest to demonstrate, because trelix's own search engine could not find its own search engine. Query the live self-index for bm25_search and the method that implements it — Database.bm25_search in src/trelix/store/db.py — comes back at rank 34 of 90 matches under default unweighted FTS5, beaten by sixteen test functions and four documentation headings that merely mention the term. With top_k_bm25 defaulting to 20 that is a recall failure, not a ranking nuisance: it is out of the candidate pool before fusion or reranking runs. declaration_boost_weight=5.0 moves it to rank 9. The mechanism is a reweighted bm25(symbols_fts, ?, ?, 1.0, 1.0, 1.0) call applying the weight to name and qualified_name only — no schema change, no reindex — and it is one-directional: rows with no name match score byte-identically at both weights. It is gated twice, declaration_boost_enabled False and declaration_boost_weight 1.0, with the default no-op pinned by an exact-equality test over the returned score pairs. To reproduce the rank move: export TRELIX_RETRIEVAL_DECLARATION_BOOST=true and export TRELIX_RETRIEVAL_DECLARATION_BOOST_WEIGHT=5.0.

Then three releases proving the surface was true

A big feature drop earns a specific obligation, and v3.0.1, v3.1.0 and v3.1.1 are what paying it looks like.

v3.0.1 is the consequential one. PythonParser.parse() recorded local indices into its symbols list during the walk, then ran symbols.insert(0, <module>) afterwards whenever the module had a docstring — shifting every recorded index by one. Symbol.parent_id, CallEdge.caller_id and TypeEdge.from_symbol_id all pointed into that list, so one off-by-one corrupted the call graph, the symbol hierarchy and the type graph at once. It never raised and never produced a NULL, because Indexer builds local_to_db from the final symbols list, so a pre-insert index still resolved to a valid row. Just the wrong one. A validity check passes; only a correctness check catches this. The fix reserves symbols[0] before the walk. The release-time measurement was 8,815 of 8,815 index references wrong across 139 source files, which is arithmetic rather than sampling: all 2,179 symbols lived in the 131 docstring-bearing files.

v3.0.1 requires a reindex. v3.0.0's release note said upgrading from v2.12.0 needed no reindex and no migration — true of the schema, false of the graph. The graph the old parser wrote is wrong on disk and no code path corrects it in place. If you upgraded from v2.12.0 or v3.0.0 and did not reindex, your call graph is still wrong: run trelix index ./my-repo again. v3.1.0 and v3.1.1 need no reindex and no schema change.

v3.1.0 is a rendering-correctness release, and its best proof needs no attacker at all: src/trelix/indexing/parser/extractors/rust.py:1035 contains re.sub(r"^//[/!]?\s?", ...), which Rich reads as an unmatched closing tag, so trelix ask against trelix's own repository died with MarkupError instead of showing results. The silent mode is worse: a balanced-looking pair is swallowed and the command exits 0 having dropped characters. A directory named deep[ then raised the same MarkupError from inside the except block meant to skip one file, discarding the whole index run and hiding its own cause.

The machine-readable side needed the opposite fix: _print_json() leaves the payload byte-identical and disables the renderer instead. soft_wrap=True is the part that mattered, because Rich hard-wraps at console width and a wrap landing inside a JSON string injects a newline that json.loads rejects — which the existing short-payload contract tests were too short to catch. Under FORCE_COLOR=1, routinely set in CI, trelix graph --json | jq produced garbage at exit 0 because the spinner wrote to stdout. 51 new regression tests, all but five demonstrated failing against v3.0.1 — the exceptions are negative and byte-identity controls that must pass on both sides — and the markup tests each assert the payload's literal characters appear in the output rather than merely that nothing raised.

v3.1.1's headline is a safety guarantee that was never true. From v2.x through v3.1.0, SECURITY.md said trelix "does not follow symlinks outside the repo boundary." FileWalker had no symlink handling at all: _iter_files used entry.is_dir()/entry.is_file(), both of which follow, and rel_path is computed on the unresolved path, so an out-of-tree file was indexed and reported as sitting inside the repo. TRELIX_WALKER_FOLLOW_SYMLINKS=false now makes the boundary real by comparing resolved paths on both sides, because Path.is_relative_to is lexical and an unresolved comparison would let repo/link -> /etc straight through. It is opt-in for a non-security reason: confining by default would silently drop files from any repository that symlinks to vendored directories.

The unit suite sits at 2,457 collected on this branch, from 2,341 at v3.0.0 and 2,445 at v3.1.0. One fact about that span is more telling than the count: src/trelix/audit/ and src/trelix/auth/ have not been touched since the commit that introduced them. The integrity core and the token verifier needed zero source changes across three releases; the code that renders them needed three. And the audit verify guard that started this article shipped with a test that only ever passed a directory — a path SQLite genuinely cannot open — so it exercised the branch that already worked. Covering the working half of a two-branch guard buys confidence rather than earning it, which is exactly what three releases of looking for the other halves were for.

What the 3.x line is for

If you are deciding whether to build on trelix, the useful summary is not the feature list. It is that the 3.x line has a verifiable trail with a CI-gateable exit code, a real OIDC resource-server gate with a forged-token test rather than a config assertion, a context packer whose quality floor is provable by reading it, and an editor integration built entirely out of the MCP surface. Each is off until you turn it on, and each has its boundaries written next to the code that draws them: tamper-evident and HTTP-only, authentication without authorization, lexical compression by default, one agent session across two chat threads.

pip install 'trelix[serve,sso]' at 3.0.0 or later gets you the audit and SSO surface. v3.1.1 is cut and dated but not yet tagged or published; when it lands it will change nothing at default settings. The one instruction that carries real consequences is the oldest in the span: if you came from v2.12.0 or v3.0.0, reindex.

Trelix v2.11.0: Jira and Linear Now Live Inside the Code Graph

SAI RAM — Sun, 02 Aug 2026 10:13:29 +0000

I've had the same conversation with trelix a hundred times: "what does _verify_auth do." It answers instantly, correctly, and completely uselessly for the question I actually had, which was why does this function exist. Git blame gets you partway there — a commit message, maybe a PR description if you're lucky. But the real answer, the actual requirement, almost always lives somewhere else entirely: a Jira ticket, a Linear issue, a bug report. Code and "the work" have always lived in two systems that never talk to each other. trelix v2.11.0, shipped 2026-08-02, is the release where that stops being true.

The headline feature is two new connectors — Jira and Linear — joining the Xray and TestRail connectors trelix already had for test-management platforms, with all four now sharing one interface. Sync a project's tickets and by the time the command returns, those tickets are already wired into the code graph and already influencing search ranking. Not a CSV dump, not a webhook you have to build your own consumer for. Real typed integrations, with each platform's actual authentication quirks handled correctly, sitting behind one shared contract.

One interface, four connectors

Every connector — JiraConnector, LinearConnector, XrayConnector, TestRailConnector — implements the same ArtifactSource abstract base class (src/trelix/indexing/connectors/base.py). Two methods: validate_config() and fetch(). That's it.

validate_config() exists specifically so a missing credential fails loud and fast. Look at what JiraConnector.validate_config() actually checks:

missing = [
    name
    for name, val in (
        ("TRELIX_JIRA_BASE_URL", self._config.base_url),
        ("TRELIX_JIRA_EMAIL", self._config.email),
        ("TRELIX_JIRA_API_TOKEN", self._config.api_token),
        ("TRELIX_JIRA_PROJECT_KEY", self._config.project_key),
    )
    if not val
]
if missing:
    raise ValueError(f"JiraConnector is missing required config: {', '.join(missing)}")

If you forgot TRELIX_JIRA_API_TOKEN, you get told that directly, before a single HTTP request goes out — not a 401 three requests deep into a paginated search, not a silent empty result you have to debug later. fetch() is the other half of the contract: paginate internally, however the target platform does pagination, and hand back a fully materialized list[Artifact]. Callers never see a page token or a cursor.

That uniform shape is the whole reason adding Linear as the fourth ArtifactSource implementation was tractable inside this release cycle instead of a future one. It didn't have to invent its own calling convention — it just had to satisfy the same two methods over a completely different platform reality underneath.

The part that actually matters for what you get out of this, though, is sync(), which lives once on the base class and every connector inherits unchanged:

def sync(
    self,
    db_writer: ArtifactWriter,
    linker: ArtifactLinker | None = None,
) -> ConnectorSyncResult:
    self.validate_config()
    fetched = self.fetch()
    written = 0
    errors = 0
    edges_linked = 0
    for artifact in fetched:
        try:
            db_writer.upsert_artifact(artifact)
            written += 1
        except Exception:
            errors += 1
            continue
        if linker is not None:
            try:
                edges_linked += linker.link_one(artifact.source_ref)
            except Exception:
                pass
    return ConnectorSyncResult(
        artifacts_fetched=len(fetched),
        artifacts_written=written,
        errors=errors,
        edges_linked=edges_linked,
    )

Read that loop carefully: when a linker is supplied, every successfully-written artifact gets passed through linker.link_one() in the same call. Not queued for a later batch job. Immediately, synchronously, before sync() returns. Run trelix connector sync ./repo jira and the output isn't "downloaded 84 tickets into a table" — it's Synced jira: fetched 84, wrote 84, errors 0, linked 79 edge(s). Those 79 edges already exist in generic_edges and are already visible to PageRank the moment the process exits. There's a standalone trelix link-artifacts command if you want a manual full re-link pass — useful after a --no-link bulk sync, or after a schema change surfaces new matches against artifacts you already synced — but it's an option, not a requirement. The default path gives you a fully linked graph in one command.

Jira and Linear: same contract, opposite platforms

This is the part I find genuinely interesting to build, because Jira and Linear satisfy the identical ArtifactSource contract while being about as different as two ticket-tracker APIs can be.

Jira Cloud is REST, authenticated with HTTP Basic — email plus API token, Jira's own recommended scheme for this kind of integration, no OAuth dance required. Pagination is a cursor token (nextPageToken), not offset/limit. The interesting wrinkle is the description field: Jira Cloud's v3 API returns description as Atlassian Document Format, a nested JSON tree, on every single ticket, unconditionally — never a plain string. That's not a rare shape you might hit on a rich-text ticket; it's the only shape. So the connector ships a real renderer, _adf_to_text(), that walks the tree — paragraphs, headings, bullet and ordered lists, code blocks, inline code, blockquotes, panels, expand sections, rules, links, embedded cards — and produces plain text good enough for ArtifactLinker's regex matching downstream. It's not a full ADF renderer (no tables, no user-mention name resolution), but it covers everything actually observed on a live Jira Cloud site.

Linear is a different animal entirely. There's no REST surface at all — it's GraphQL-only, a single POST endpoint (https://api.linear.app/graphql), and there's no official Python SDK, only a TypeScript one. Pagination is Relay-style: first/after arguments, pageInfo { hasNextPage, endCursor }. The loop has to terminate on hasNextPage, not on a short page — Relay pagination can legitimately hand back fewer results than requested while hasNextPage is still true, so checking len(results) < page_size the way TestRail and Xray do would end the sync early and silently drop issues.

The auth detail is worth calling out by name, because it's the exact kind of thing that breaks silently if you're not paying attention: Linear's personal API key goes into the Authorization header with no Bearer prefix. Just the raw key. Every other connector in this codebase uses Bearer <token> — Xray's connector literally does f"Bearer {self._token}" — so copy-pasting that pattern for Linear produces a request that authenticates against nothing and fails with a 401 whose message doesn't obviously point at the missing prefix. The module docstring in linear.py calls this out explicitly as "the one detail most likely to get corrected back to Bearer by someone skimming Xray's connector — don't."

Linear also runs a real query-complexity budget: 10,000 points per request, where a connection multiplies its children's cost by its own first argument. This connector's field selection costs roughly 7.7 points per issue, so at page_size=100 that's about 1 + 770 ≈ 771 points — comfortably under the cap, with real headroom for schema drift. And Linear signals rate-limiting differently from everything else in this codebase: HTTP 400, not 429, carrying a GraphQL body error with extensions.code == "RATELIMITED", and no Retry-After header at all. Reset timing has to come from reading X-RateLimit-Requests-Reset/X-RateLimit-Complexity-Reset response headers directly, so _fetch_issues_page() runs its own small, connector-local retry loop for exactly this case rather than teaching the shared retry contract a Linear-specific response shape.

What actually makes a synced ticket useful

Downloading a ticket into a table doesn't do anything by itself. The piece that turns a synced artifact into something that changes search results is ArtifactLinker (src/trelix/indexing/artifact_linker.py), and it's worth walking through because the design has a real precision problem baked into it that's easy to get wrong.

Stage one is regex extraction: _IDENTIFIER_RE finds identifier-shaped tokens in an artifact's title and body — function/class/variable-looking strings — and checks each one, casefolded, against an index of every symbol name and qualified name in the codebase. A hit becomes a GenericEdge with edge_kind="references_artifact" and weight=1.0 — a real graph edge connecting that ticket directly to that code symbol.

The obvious failure mode here is that plenty of function names are also ordinary English words. A ticket titled "the test suite failed to run and update the process" would, on pure token overlap, spuriously link to functions literally named run, test, update, and process — none of which have anything to do with that ticket. ArtifactLinker handles this with _COMMON_WORD_STOPLIST, roughly 40 common English/programming words (get, set, run, test, process, update, data, file, main, and so on) excluded specifically from the bare-name side of the index. It's a real precision guard, not an afterthought, and it's scoped carefully: it only gates the bare name key, not qualified_name, so a genuinely qualified match like auth.run still links correctly. Only the literal stoplisted string on its own gets excluded.

Stage two is an opt-in embedding-similarity fallback, gated behind ArtifactLinkerConfig.embedding_fallback_enabled, and it only runs for artifacts where stage one found zero matches. Those fallback edges get weight=0.5 — deliberately half the weight of a regex hit — so a fuzzy semantic match never outranks an exact identifier match for PageRank purposes. Exact reference wins; embedding similarity is a second-chance net, not a peer signal.

Xray and TestRail: proving the abstraction generalizes

If the ArtifactSource contract only worked for issue trackers, I'd be less confident in it. Xray and TestRail are both test-management platforms, and they slot into the exact same shape.

TestRail is the simpler of the two: HTTP Basic auth (username plus API key), offset/limit pagination capped at TestRail's own 250-per-page ceiling. Xray Cloud is odder — its tests are Jira issues under the hood, but test-specific content (steps, expected results) has no Jira-native equivalent, so a single GraphQL getTests query pulls both the Jira fields and the Xray-specific step data in one call, with no separate REST round-trip needed. Auth is its own thing too: a client_id/client_secret pair, distinct from a personal Jira API token, exchanged via POST /api/v2/authenticate for a short-lived bearer JWT attached to every subsequent request. Both connectors write artifact_kind="test_case" records, and both flow through the exact same sync() → ArtifactLinker path as Jira and Linear's tickets. Same registry lookup, same CLI surface, same auto-link behavior. The abstraction holds.

The CLI is identical across all four: trelix connector sync ./repo <jira|testrail|xray|linear>, with a --link/--no-link flag defaulting to on, resolved through one factory — get_artifact_source("jira"|"testrail"|"xray"|"linear") in registry.py — that mirrors the same match-statement pattern trelix already uses for embedder and vector-store selection.

The plumbing underneath

None of the above would mean much if a sync silently swallowed failures or nobody could tell why a request hung. This release also landed a unified retry/backoff contract (core/retry.py, built on tenacity) shared across every LLM backend, embedder, and connector — full-jitter exponential backoff on 429s and 5xx responses, honoring a server's Retry-After header when present. Structured JSON logging shipped alongside it, with trace-ID correlation when OpenTelemetry is enabled. And Personalized PageRank landed as an opt-in flag (TRELIX_RETRIEVAL_PAGERANK_PERSONALIZATION) that concentrates teleport mass on ticket-linked symbols instead of spreading it uniformly — its own docs flag a real interaction risk: combined with pagerank_boost_enabled on a repo with few ticket-linked symbols, it can invert the ranking get_top_central_symbols() produces. Opt-in for a reason.

None of this was validated against mocks alone, either. Live-testing the Jira connector against a real production Jira Cloud site turned up two real bugs — ADF descriptions silently rendering as empty bodies on every ticket, and bad credentials silently reporting success because /rest/api/3/search/jql returns an identical empty result set for a bad token, no token, and a genuinely empty project. Both got fixed this cycle: a real _adf_to_text() renderer for the first, and a pre-flight auth check against /rest/api/3/myself for the second, since that endpoint correctly returns a real 401 where the search endpoint doesn't.

What this actually unlocks

The full unit suite sits at 1,909 passing tests as of this release, up from 1,643 at v2.9.0 — not because coverage was thin before, but because four connectors, a linker, a retry contract, and Personalized PageRank all needed their own test surface.

But the number that matters isn't the test count. It's that trelix can now sit in front of a question like "why does this function exist" and actually answer it, because the ticket that requested the change, or the bug report that prompted the fix, is sitting in the graph one edge away from the symbol itself. Ask "what's supposed to happen when I touch this code" and the answer isn't just call-graph traversal anymore — it's the test case that verifies it, linked in through the same mechanism, the same weight-1.0 edge, the same ArtifactLinker pass. Before this release, trelix understood code. Now it understands why the code is there.

trelix v2.7 to v2.9: The Release Where the Pipeline Itself Became the Product

SAI RAM — Fri, 24 Jul 2026 16:44:32 +0000

On 2026-07-09 I shipped trelix v2.7.0. The architecture felt done — seven retrieval legs, a knowledge graph, an agentic loop. Then I opened the GitHub Release page and counted two binary assets where there should have been three. release.yml built the macOS and Linux PyInstaller binaries both as dist/trelix, and softprops/action-gh-release uploads assets by basename — two files with the same name collide into one asset, with no metadata telling you which OS survived. I genuinely could not tell, from the published release, whether the surviving binary was macOS or Linux. That's not a bug in trelix's retrieval logic; it's a bug in the thing that puts trelix in front of users, and it shipped because nobody checked the page after the workflow went green.

That bug is why this article exists. v2.7.1 through v2.9.0 — six releases over two weeks — spend a surprising amount of energy on parts of the project that aren't retrieval quality at all: the release pipeline, the concurrency model, a language-parser migration, the deployment story, and a VS Code extension and GitHub App with real, embarrassing problems. I'm grouping by theme instead of walking the changelog top to bottom.

Shipping infra is its own product surface, and it broke

The binary collision got fixed in v2.7.1 (2026-07-10) by renaming each binary uniquely before upload. Fresh eyes on the pipeline turned up company: the PR-time CI workflow had never built a Linux binary, even though the release workflow built one at tag time. I added the missing matrix entry and a verify step.

The most embarrassing find: trelix-mcp's own test suite had never run in CI. It let a real regression sit undetected — a test asserted the MCP server had "exactly 6 tools" when it had actually registered 8 since two subscription tools shipped in v2.5.0. A wrong-but-passing test is worse than no test. I wired the tests into CI and fixed the assertion.

Smaller mistakes, same release: three companion packages' dependency floors on trelix core had been bumped in v2.7.0 on the assumption they used new APIs. None did, so I reverted them. The changelog had also rotted: an entry was defined twice with conflicting URLs, silently resolving to the last definition, so the first link was dead. Rebuilt the footer from the actual git tags.

Five concurrency bugs, found only once I wrote real stress tests

v2.7.2 (2026-07-12) reads like the payoff of finally taking concurrency seriously instead of assuming check_same_thread=False meant "safe." It shipped real scale features — Qdrant Cloud readiness, incremental per-symbol embedding on partial re-index, an opt-in parallel BM25 read pool — but the five bugs stress-testing surfaced matter more.

A TOCTOU race in the sparse embedder's lazy-load checked whether the model was loaded before acquiring its lock, so two racing threads could both see "not loaded" and both start loading; fixed with double-checked locking. An MCP stdout write race let concurrent notification writes interleave partial JSON-RPC lines, corrupting a client's output; fixed with a lock around the write-and-flush pair. And unbounded subscription-registry growth let a misbehaving client grow the registry forever, fixed with a max-subscriber cap and a TTL sweep.

The one I find genuinely unsettling in hindsight: silent foreign-key corruption on partial re-index. The parent-symbol, call-callee, and type-edge columns are all set to null on delete, which sounds safe until deleting a changed symbol's old row silently nulls those links on every row that referenced it — including unchanged rows with nothing to do with the edit. No error, just quietly severed graph edges accumulating as files change. I added snapshot-and-repoint helpers so the indexer captures stale links before the delete and repoints them.

And the BM25 lock was incomplete even after I thought I'd handled it. The shared database connection is opened with check_same_thread=False, which I'd treated as a green light for concurrent use — it isn't; that flag disables SQLite's thread-affinity check, not concurrent-statement safety. Grep, sparse, and vector legs all hydrate through that connection from sibling worker threads. I added a real lock everywhere it's touched from a worker thread, verified with a 60-thread by 10-iteration by 3-leg stress test and zero errors — the only way I'd trust this, given the previous fix had looked equally trustworthy.

Same release: a Qdrant client API migration to keep pace with an upstream deprecation, pinned so a future major bump can't break it again. One honest non-win: Windows ARM64 binaries were briefly added to the build matrices, then reverted — two core dependencies publish no wheel for that target. Linux ARM64 shipped; Windows ARM64 didn't.

v2.7.3 was a pure documentation release. A full README audit fixed 15+ factual bugs: wrong env var names, fabricated pip extras, a broken Homebrew tap, a config value that crashes on use, wrong REST method and table names. The architecture diagram got redrawn to show all 7 retrieval legs instead of 3. I backfilled the changelog's empty [2.2.0] entry, which had shipped 5 real features — agentic ReAct loop, data-flow analysis, taint analysis, sparse+dense hybrid retrieval, multi-granularity indexing — never documented anywhere. The README shrank by roughly a third by consolidating duplicated API sections into pointers. None of it changed behavior; it changed whether docs matched reality.

Becoming genuinely deployable, not just runnable

v2.9.0 is where trelix stopped being "clone it, pip install, run the CLI" and became something you could put in front of other services. Four pieces went in, and the order was the point.

Typed REST API response models came first — every route now declares a real Pydantic model instead of an untyped object in the OpenAPI schema, groundwork for the next item. Cursor pagination on the search endpoint came second, the one deliberately narrow breaking change in an otherwise additive release: the endpoint now returns a results-plus-cursor envelope instead of a bare list, matching the MCP tool's existing pagination contract, done so the new SDK wouldn't lock in against the old shape.

The TypeScript SDK came third — a hand-written HTTP client with generated types covering every route, with /ask's streaming response getting its own async generator because token-plus-terminator-plus-error-frame semantics don't fit the same shape as everything else. Fourth, OpenTelemetry tracing — opt-in, off by default, emitting one span per retrieval leg plus spans for each pipeline stage. Worth explaining: OTel's context propagation doesn't cross a thread-pool boundary automatically, and trelix's parallel sub-query execution runs inside exactly that kind of pool, so naive instrumentation produces disconnected, orphaned spans. The fix wraps the traced function to carry the current context across the pool boundary — the docs note the underlying span-naming conventions are still "Development," not "Stable."

The deployment story rounds out with an official multi-arch Docker image in two variants — a slim, API-embedder-only build and a -local build bundling the offline embedding stack — running as a non-root user, with the entrypoint overriding the CLI's loopback-only default to listen on all interfaces (a loopback bind is a silent dead-end in a container unless overridden). Plus a Helm chart modeling the server's real behavior: every route re-derives its config from the request's own repo parameter, so one Deployment is already multi-repo-capable — but the chart's persistent volume is a shared data directory across every repo you serve through it, called out loudly in the docs. Ingress defaults to disabled: the server ships with zero auth middleware, and I'd rather state that directly than bury it.

Building this surfaced the same documentation-rot pattern again: docs referenced an embedder env var that was actually a silent no-op, plus a nonexistent CLI flag and Docker examples with the wrong port — docs drift silently unless something forces someone to run the commands in them.

Python 3.13, and how one dependency swap turned into six bugs

The Python 3.13 item sounds like the least interesting line in the whole changelog: bump the minimum version, done. The actual blocker was tree-sitter-languages, abandoned upstream with no wheels for the new interpreter. The fix was a single swap to the actively-maintained tree-sitter-language-pack, behind trelix's one chokepoint for grammar loading — that's the whole diff at the chokepoint. It is not the whole blast radius: the new library exposes different AST node names and shapes, which propagated silently into nearly every per-language extractor, because each had been written against the old grammar's node vocabulary without anyone realizing how much was implicit knowledge rather than a documented contract.

Six separate bugs came out of this: C#'s grammar name changed, silently tolerated before this release only because the old library accepted both names; Kotlin's extractor had to be rewritten entirely after its field-lookup API stopped returning anything, breaking class, interface, enum, function, and property extraction at once; Python's docstring extraction broke because the new grammar drops a wrapper node; Go's interface-method node was renamed; TypeScript's interface-body node got a new name; and C#'s using alias imports lost their wrapper node.

Every one of these was catchable only because per-language extractor tests already existed before the migration — without them, most would have shipped as silent failures. A seventh casualty I almost missed: the PyInstaller build spec still imported the retired package, breaking every binary build with a flat import error, because that build runs in its own workflow, outside the test suite and linters. Dropped the stale entries, added the new package as a hidden import, verified with a local build plus a smoke test.

One behavior change worth knowing: grammar loading is now network-on-first-use with local caching, not bundled in the wheel — fine on a laptop, mildly alarming for an air-gapped job. A prefetch function warms the cache during image builds; CI runs it automatically now.

The VS Code extension, the GitHub App, and an admission I'm not going to soften

The search command became a debounced (250ms) search-as-you-type picker with live snippet preview, instead of the old one-shot input-then-static-list flow. The preview runs through a virtual document provider that gets real syntax highlighting for free by keeping the real file's extension on its virtual URI. The state machine lives in its own testable class, verified with a fake-timer harness proving the debounce actually debounces.

The security fix here is direct: the ask-panel's Webview interpolated the raw, unescaped LLM answer string straight into HTML, with no content-security policy and no script restrictions. A crafted or adversarial answer — not a hypothetical when you're piping retrieval results through a model — could execute arbitrary script inside the Webview's context. Fixed with HTML-escaping plus disabled scripts and an explicit deny-all CSP; it's an XSS vulnerability, and it's fixed now. Separately: search results had been silently mis-parsed the entire time the extension existed — it read the wrong field names off each result, confirmed against the actual server source, so those fields were always empty strings and clicking a result opened a broken, empty file URI.

And then the line I'm quoting close to verbatim because paraphrasing softens it: the trelix Code Review Check has never posted a single real annotation since the workflow shipped. Status messages ran unconditionally to stdout even in JSON output mode, and combined with the workflow's output redirect, the annotation-posting step's JSON parse had been throwing on every run, silently swallowed since the day this workflow shipped. Every PR ever reviewed by this Check got nothing. Fixed by routing status output to stderr and narrowing the redirect to stdout only. Even after that fix, the mapping logic still wouldn't have worked — wrong response keys, lowercase severity strings compared against real uppercase values. New regression tests were verified against the pre-fix code first — most failed with the exact parse error this bug produces — before trusting they passed for the right reason. The GitHub App also reached GA-readiness: real installation-token minting behind an expiry-aware cache, webhook signature verification via constant-time comparison, a request-body size cap matching GitHub's own limit, and a subprocess timeout on the review shell-out.

Multi-repo federation, and the security fix found before it hit anyone

v2.8.0 and v2.8.1 shipped the same day, 2026-07-20, and the second was a direct response to auditing the first before letting it sit. v2.8.0 exposed the existing federation infrastructure to MCP clients through four new tools, and added a CLI command apparently missing despite the underlying method already existing. Persistent agent memory landed for the agentic loop too — a follow-up call can now resume a prior conversation with full context, sessions auto-evicting after a week of inactivity.

Building this surfaced two real, previously invisible bugs: federated search had silently lost repo provenance in an earlier refactor, so the search command's repo column had been blank with no test catching it; and a per-repo weighting setting — settable, stored, documented — had never actually been forwarded into the fusion math, silently doing nothing since it was added.

v2.8.1 is where the real security finding lives. All four federation MCP tools passed a caller-supplied config path straight into the registry's load/save calls with zero validation, meaning an MCP client — including a prompt-injected agent, exactly the threat model MCP has to take seriously — could point registry I/O at an arbitrary filesystem path. I found this in a pre-push audit of v2.8.0, before it reached anyone running the released version. The fix confines the path to one of two known-safe directories via a proper containment check, not a naive string-prefix check, which would also incorrectly match a similarly-named sibling directory. Same release: repo-count and fan-out caps so a runaway add-repo loop can't scale every search linearly against an unbounded repo count, plus a pagination fix for a per-repo candidate pool that had been widening as the cursor grew, letting later pages get fused from a differently-shaped pool than earlier ones.

Tombstone v1.3-v1.4: Resilience Was the Easy Layer

SAI RAM — Thu, 09 Jul 2026 19:57:31 +0000

I removed four || true statements from a GitHub Actions workflow on July 9th and watched CI go red in four different ways within the same run. Not one failure. Four — a Python pytest install that had never actually finished, a ruff lint violation nobody had looked at, a Ruby require path that resolved to nothing, and a Java Gradle wrapper that didn't exist in the repo. All four had been "passing" for who knows how long, because the test steps were configured to succeed no matter what came back.

That's the theme of this release window. v1.2 was about making the running system survive failure — retries, circuit breakers, idempotency keys, DLQs. This one is about making the layers above the running system — the Helm chart, the SDKs, the GitOps pipeline, the CI config — tell the truth about their own state. Resilience isn't a feature you ship, it's a property you discover you're missing.

The Helm chart was only deploying two of five services

Tombstone has five application services — flag-api, gateway, evaluator, intelligence, marketplace — plus the operator. Until v1.3.0, the Helm chart (infra/helm/flagmind) only had Deployment templates for two of them. This wasn't a secret; it was written down in COMPATIBILITY.md under a section literally titled "Known Gap." Run helm install in a fresh cluster and you got flag-api and gateway, nothing else. Anyone deploying evaluator, intelligence, or marketplace was hand-rolling manifests or copy-pasting the two existing templates and hoping the env vars lined up.

v1.3.0 closes that gap: deployment-evaluator.yaml, deployment-intelligence.yaml, and deployment-marketplace.yaml now ship in the chart. evaluator gets an optional HPA behind evaluator.autoscaling.*. intelligence exposes IS_PRIMARY_REGION straight from values.yaml, which matters for the multi-region setup where secondary regions run in read-only relay mode.

Here's the footgun worth flagging, because it isn't obvious until it bites you: every one of those templates uses tombstone.selectorLabels in spec.selector.matchLabels, not the separate tombstone.labels helper, which includes a version label. It's tempting to use the one "labels" helper everywhere for consistency. Don't. Kubernetes Deployment selectors are immutable once the object exists. If your selector helper includes a label that changes on every release — like a chart version — the second helm upgrade you ever run will fail outright, because the new selector no longer matches the old one. tombstone.selectorLabels is a narrower, stable subset — name and component, nothing that changes across releases — specifically so matchLabels never drifts. One line in a template, and it's the difference between a chart that upgrades cleanly forever and one that works exactly once.

SDK parity is a correctness bug, not a feature request

The TypeScript SDK (@tombstone/core) has had the full 5-step evaluation pipeline since v2.0 — preliminary checks, prerequisites, individual targeting, rule matching, fallthrough — with the complete operator set and semver comparisons. The Python SDK didn't. It had basic targeting but was missing large chunks of the operator surface and all of the prerequisite evaluation logic. That's not a nice-to-have gap: a flag with a semver_gte rule or a prerequisite on another flag could evaluate to true in a Node service and silently fall through to the default in a Python service, evaluating the same flag against the same user. Two SDKs, two answers, same input — the kind of bug that looks like a targeting mistake in the dashboard when it's actually a parity gap in the client.

v1.3.0 closes it. packages/sdks/flagmind-python/tombstone/matching.py now implements the full operator set — eq/neq/in/nin/contains/startsWith/endsWith, the four numeric comparisons, and all five semver operators (semver_gt/gte/lt/lte/eq) — plus date_before/date_after. The semver comparison is hand-rolled: a _padded_version() helper left-pads each numeric segment to 5 characters and appends a ~ sentinel for 3-part releases, so 1.0.0-beta sorts below 1.0.0 using pure string comparison. It's the same GrowthBook paddedVersionString() pattern the TypeScript SDK already used, with zero new runtime dependencies — no semver package, no extra install footprint.

Prerequisite evaluation was the other half. evaluation.py now threads an evaluation_cache: dict[str, bool] through the recursive prerequisite check, so a flag with three prerequisites sharing a common ancestor doesn't re-evaluate that ancestor three times. Circular chains are rejected via a _seen_keys tracking set rather than recursing forever. The SDK also distinguishes two failure modes with dedicated exception types: InconclusiveMatchError means a targeting condition couldn't be evaluated locally — missing attribute, type mismatch — and the caller should move on to the next rule. RequiresServerEvaluation means the evaluation genuinely needs data the local cache doesn't have, and the client falls back to a REST round-trip instead of silently returning a wrong default. Conflating the two used to mean callers couldn't tell "skip this rule" apart from "call the server."

Making the API impossible to not find

The previous article's two worst bugs — the Slack kill switch sending environment as a query param when the handler read it from the JSON body, and the four-eyes approval routes that were built, reviewed, and merged but never registered in flag-api/cmd/main.go — share a root cause that has nothing to do with the code itself. Nobody had an easy way to look at what flag-api actually exposed versus what the proto files said it should expose. The OpenAPI spec existed; nobody was looking at it, because there was nowhere convenient to look.

v1.3.0 adds a Redoc explorer at GET /api/v1/docs, embedded via go-redoc rather than pulled from a CDN — it reads the existing grpc-gateway OpenAPI spec at /api/v1/openapi.json, so there's no second source of truth to keep in sync. The implementation detail that made this take longer than expected: the plan referenced a chi adapter for go-redoc that doesn't exist in any published version — the library only ships gin/fiber/echo adapters. The fix was to call goredoc.Redoc{SpecPath: specURL}.Body() directly for pre-rendered HTML and wrap it in a plain http.HandlerFunc. Small feature, but it's the direct answer to how those two bugs happened in the first place: the API surface wasn't something anyone could casually glance at.

GitOps: ordering discipline, then a second controller on purpose

v1.4.1 brought Flux CD v2.3+ into gitops/, structured as three layered Kustomizations — infrastructure, apps, flags — each declaring dependsOn against the one before it. The infrastructure layer also sets healthChecks against the tombstone-operator HelmRelease, which matters more than it sounds: dependsOn alone only blocks applying a Kustomization until the upstream has been applied — it does not wait for the upstream to actually be healthy. Without healthChecks, Flux considers the operator's CRDs "done" the moment the HelmRelease manifest hits the API server, before the CRDs are actually established, and the apps layer — which deploys FeatureFlag CRs via the flagmind chart — could start reconciling against CRDs that technically exist but aren't ready yet. Pairing dependsOn with healthChecks on the upstream is what actually guarantees ordering — this is exactly the kind of GitOps bug that works fine in every test until the day the operator pod is slow to come up.

The more interesting decision is v1.4.2's addition of Argo CD v2.11 — not as a Flux replacement, as a second controller with a deliberately split job. Flux keeps infrastructure: the operator's CRDs, image update automation across the tombstone container images. Argo CD takes over the flagmind chart plus the FeatureFlag and FlagPolicy custom resources themselves. The reason to split: FeatureFlag CRs have a rolloutPct field that the intelligence service's ML rollout recommendations mutate live in the cluster, independent of Git. A naive GitOps setup would see that mutation as drift and revert it on the next reconcile — the platform's own ML-driven rollout logic fought and undone by its own deployment tooling every sync interval.

The fix lives in gitops/clusters/production/argocd/apps.yaml: an ignoreDifferences block excluding /spec/environments/production/rolloutPct and /spec/environments/staging/rolloutPct from diff detection, paired with RespectIgnoreDifferences=true in syncOptions. Both are required — ignoreDifferences alone only suppresses the OutOfSync display; without the sync-options flag, Argo CD still overwrites the field back to the Git value on every sync. Miss either half and the ML rollout percentage gets silently stomped on a fixed interval, a nasty class of bug to chase down because nothing in the intelligence service's logs would look wrong.

Argo CD also needed Lua health checks for Tombstone's own CRDs — FeatureFlag (Pending to Progressing, Synced to Healthy, Error to Degraded) and FlagPolicy (Compliant to Healthy, Violation to Degraded) — because its generic health rollup doesn't understand custom CRD status fields out of the box. Without them, the root Application would just show every FeatureFlag as permanently "Unknown" rather than reflecting real reconciliation state.

Blast radius, but as a deployment gate now

This is the part I like best, because it closes a loop back to the v1.0 launch: blast-radius scoring — the LOW/MEDIUM/HIGH/BLOCKED classification the evaluator computes for flag changes — now gates Kubernetes deployments, not just flag rollouts. v1.4.1 adds an Argo Rollouts AnalysisTemplate named tombstone-blast-radius that polls GET /api/v1/blast-radius?flag_key=<key> on the evaluator service during a canary step. gitops/apps/production/tombstone/rollout-analysis.yaml wires this into flag-api's own Rollout: step to 20% traffic, run the analysis template, promote to 100% only if the result is LOW or MEDIUM, abort immediately on HIGH or BLOCKED (failureLimit: 1, so the first bad reading stops it, no averaging across a window). The same scoring engine that decides whether a flag change is safe to ship now also decides whether a code deployment is safe to ship — the same signal doing double duty at two layers of the stack.

Argo CD Notifications routes sync-failed events into the existing marketplace Slack endpoint (marketplace.tombstone.svc:8086/api/v1/marketplace/slack/actions) rather than standing up a second webhook — one less integration surface to maintain.

Removing the safety net

Back to where I started. The || true removal in .github/workflows/ci.yml (commit 9c9bff9) touched test steps across the Python intelligence service, the TypeScript SDK build, the Python SDK, the Ruby SDK, and the Java SDK — every one configured to report green regardless of outcome. The very next commit (031d041) is titled, accurately, "fix pre-existing test failures surfaced by removing || true," and it fixes four of them:

The Python SDK's pytest step installed mmh3 and the package itself but never installed pytest — the runner didn't exist in the environment, so python -m pytest had presumably been failing at the shell level the whole time.
ruff check . --ignore F401 in the intelligence service flagged unused local variables (now_ts, model) a real lint gate should have caught the moment they were written.
packages/sdks/flagmind-ruby/lib/tombstone.rb didn't exist at all — spec files require "tombstone" but the gem's actual entry point is flagmind.rb, a leftover from the gem's rename. I added a one-line alias file.
The Java SDK's step ran ./gradlew test, but no gradlew wrapper is committed to the repo — the step was failing to even start.

Fixing the Java one turned into its own small saga across four more commits, because each fix uncovered the next problem: swapping to gradle/actions/setup-gradle failed because the pinned SHA wasn't resolvable; falling back to apt-get install gradle got Gradle installed, but the Ubuntu-runner package is old enough that it chokes on useJUnitPlatform(); installing Gradle 8.7 directly from services.gradle.org fixed that but exposed that build.gradle declared sourceCompatibility = JavaVersion.VERSION_21, an enum literal the older Gradle parser doesn't accept, which needed to become the plain string '21'. Even after all that, the Java tests still fail, but for a real reason: the source files declare package io.tombstone.* while living under io/flagmind/ directories, the same rename leftover that broke the Ruby require. That's now continue-on-error: true with a comment pointing at v1.5.0, not a silent || true — the failure is visible in the CI UI and tracked, instead of invisible and untracked.

Four bugs weren't introduced in this release. They'd been there for a while, hiding under a shell operator that made "it ran" indistinguishable from "it passed." The same commit mirrors a fix v1.2.1 made earlier, adding pytest-asyncio back for a similar reason — a test suite that can fail silently isn't testing the thing it claims to test, it's testing that the CI runner can execute a command.

Supply chain and the honest caveat

Alongside the CI honesty pass, all remaining GitHub Actions workflow files got pinned to immutable commit SHAs instead of mutable tags like @v4 — nine files in one commit (bfb167f), closing out the supply-chain hardening that earlier workflows (flux-bootstrap.yml) had already established as the convention. Two other release blockers landed in the same commit: an ImageUpdateAutomation resource still on the wrong beta API version, and a structural fix to the production rollouts kustomization that was silently missing a resource reference.

Everything above — the layered Flux Kustomizations, the Argo CD split, the blast-radius AnalysisTemplate — is validated against a local k3d test cluster: every kustomize build passes, Flux bootstraps cleanly, Argo CD installs and reconciles. What it is not yet validated against is the actual production target, Oracle Cloud Kubernetes, blocked on an Oracle Cloud account signup that hasn't happened yet. The operator Helm chart is already published to ghcr.io/sairam0424/charts/tombstone-operator at v0.1.0, ready for the day the cluster exists. Better to say that plainly than let "GitOps shipped" imply more than k3d has actually proven.

The throughline across v1.2 through v1.4 is the same lesson at three different altitudes. v1.2 was runtime resilience — the system surviving its own dependencies failing. v1.3 was correctness at the API and SDK boundary — two clients agreeing on what a flag evaluates to. v1.4 is deployment and CI honesty — the pipeline that ships the system telling the truth about its own state, in the right order, without silently eating failures. None of these layers were broken in an obvious way. They were all quietly returning something other than the truth, and the only way to find that out was to stop letting them.

trelix v1.0 to v2.7: When "It Works" Meets "It Scales"

SAI RAM — Thu, 09 Jul 2026 19:39:19 +0000

Twelve days after I shipped trelix v1.0.0, I was staring at a RetrievalConfig object with two conflicting sets of values and no idea which one was actually running. I'd built AdaptiveRouter to accept a retrieval_config parameter so callers could override the environment-variable defaults programmatically. Except it didn't. The constructor took the parameter, and then quietly ignored it and built its own instance from env vars anyway. Nobody had wired the plumbing from Retriever through QueryPlanner down to AdaptiveRouter.__init__. It's the kind of bug that doesn't throw — it just makes your carefully-set config a decoy.

That fix landed in v2.7.0, PR #55, thirteen days and seven minor releases after launch. In between, trelix went from "search my repo well" to something closer to a platform: a knowledge graph, seven fused retrieval legs, an agentic loop, federated multi-repo search, and a GitHub Actions bot that reviews your PRs. Here's what actually shipped, grouped by what it was trying to solve rather than by version number.

From flat search to a knowledge graph

v1.0 already had hybrid BM25 + vector + call-graph search. What it didn't have was any notion of the codebase as a system — which files cluster into modules, which symbols sit at the center of the import graph, which concepts a human would use to describe an architecture. v2.0.0 (2026-06-28) and v2.1.0 (2026-06-30) fixed that.

The new trelix/graph/ module builds a CodeGraph as a NetworkX MultiDiGraph, unifying call, import, and type edges into one traversable structure. On top of that, Louvain community detection clusters the graph into architectural modules — run trelix graph ./repo and you get the top communities, not just a flat symbol list. ConceptExtractor layers an LLM on top of symbol batches to name those communities in plain English, and it's built to fail quietly: any extraction error returns [] rather than crashing the pipeline. GraphVisualizer.export_html() renders the whole thing as an interactive Pyvis HTML page with community coloring, gated behind pip install trelix[knowledge-graph] so the base install doesn't inherit the dependency weight.

Graph search became a first-class retrieval leg — graph_search_enabled=True runs a CodeGraph BFS as a fourth leg after RRF fusion — and pagerank_boost_enabled uses import-graph centrality to boost symbols that sit at architectural chokepoints. None of this is static: GraphUpdater.update_file() is wired into trelix watch, so the graph and its communities update incrementally as files change, instead of requiring a full rebuild.

This came with the release's one deliberate breaking change: trelix graph — which used to mean "show me callers and callees of this symbol" — got renamed to trelix call-graph. The name trelix graph now means "build the knowledge graph." I made the call that a growing surface area needed the more intuitive name reserved for the bigger feature, and documented the rename explicitly in the changelog rather than let people discover it by trial and error.

Seven legs, one fusion function

v1.0 had three retrieval legs. By v2.2.0 it had seven, and every one of them is grounded in a specific paper rather than a hunch.

The fourth leg was the graph BFS above. The fifth is RAPTOR-style (arXiv:2401.18059) file-level summarization — file_summary_leg_enabled, gated behind TRELIX_FILE_SUMMARIES_ENABLED=true at index time — which lets trelix answer "explain this codebase" questions that no symbol-level chunk could answer alone. The sixth is HyDE (arXiv:2212.10496): instead of embedding your raw natural-language query, hyde_fallback_enabled generates a synthetic code snippet and embeds that, closing the semantic gap between "how do I validate a JWT" and the actual token-validation code. The seventh is multi-query expansion, which decomposes one query into N variants and RRF-fuses the independent retrievals for broader recall.

Layered over all seven is FLARE (arXiv:2305.06983) — a confidence-gated re-retrieval loop that watches synthesis output for uncertainty phrases and triggers another retrieval pass when it finds them, rather than committing to a possibly-wrong first answer.

None of this is worth shipping without a way to measure whether it's actually better, so v2.1.0 added a CoIR-format eval harness (ACL 2025, arXiv:2407.02883) — trelix eval --golden <file> reports nDCG@10, Recall@10, and MRR, implemented as pure-Python trelix.eval.ndcg with zero pandas dependency. Every query now also writes a row to a query_telemetry SQLite table — latency, intent classification, result count — surfaced through trelix telemetry.

Here's the detail that actually matters: in v2.1.0, MultiQueryExpander existed as a class but nothing called it. It took until v2.3.0 (2026-07-02) for it to get wired into _retrieve_standard, and even then I had to be careful about one specific line — variants[1:] is used, not variants[:], so the original query never runs twice through the fusion. It's a one-character difference between "seven legs" and "seven legs, one of them redundant."

Teaching it to act, not just retrieve

v2.2.0 (2026-07-01) shipped across four parallel feature branches — PRs #29 through #32, merged via release PR #33 — and it's the release where trelix stopped being purely a retrieval system.

The agentic loop (trelix/agent/) is CodeAct-style ReAct: instead of one retrieval-then-synthesize pass, the agent can decide it needs another lookup, run it, and fold the result back into its reasoning before answering. Alongside it, trelix/analysis/taint.py and defuse.py added real data-flow and taint analysis — tracing how a value flows from a source to a sink across function boundaries, which is a different kind of question than "what code is semantically similar to this query."

The other two branches were retrieval-quality work: SPLADE-Code sparse retrieval (trelix/embedder/sparse.py, trelix/store/sparse_store.py) gives trelix a learned sparse representation to sit alongside BM25 and dense vectors, and multi-granularity indexing (trelix/indexing/multi_granularity.py, MGS3-style) means the index isn't forced to choose one chunk size — function-level, class-level, and file-level granularities can all be retrieved against.

Production hardening: guards built before the bugs happened

The most interesting engineering in this window isn't a feature — it's a class of failure I preempted instead of debugging in production. DimensionGuard, added at Retriever.__init__ in v2.3.0, checks embedding provider and dimension at startup and raises DimensionMismatchError with the exact recovery command (trelix migrate-vectors --reset) if they don't match what's on disk. Without it, switching from an Azure embedder (3072-dim) to a local model (384-dim) doesn't error — it silently returns wrong results, because cosine similarity between mismatched-dimension vectors still computes a number, just not a meaningful one. That's the worst kind of bug: no stack trace, no crash, just quietly bad answers. v2.5.0 (2026-07-06) extended the same guard to FileWatcher.__init__, so a provider mismatch fails fast at watch startup instead of at query time, days later, when nobody remembers which embedder was configured when.

The MCP surface grew up in the same window. v2.3.0 added MCP Resources (trelix://index/stats, trelix://repo/{path}/manifest, trelix://repo/{path}/symbols/{name}) and MCP Prompts (trelix-search, trelix-explain, trelix-blast-radius) — reusable, application-addressable primitives instead of one-off tool calls. v2.5.0 went further: trelix-mcp now advertises resources.subscribe=True, and a thread-safe SubscriptionRegistry tracks who's watching which URI, so notify_file_changed() can fire notifications/resources/updated the moment watchfiles detects a change. That notification path had a gap of its own until v2.7.0 Phase 1 (PR #55): FileWatcher._do_reindex only fired the notification on hash-identical skips, never on an actual successful re-index — the one case where a subscriber genuinely needed to know. The same release added idx_files_rel_path as an index on files.rel_path, eliminating a full table scan that GraphUpdater.update_file() was silently paying on every single file-change event.

Federation, and finding your code's twin

DiffReviewer and trelix review <repo> [--diff] [--base] [--head] (v2.3.0) turned trelix into something you point at a diff, not just a repo — it parses git diffs via DiffParser.from_git() (subprocess with shell=False, no injection surface), turns each hunk into a retrieval query, and generates review comments that are crash-safe by construction: DiffReviewer.review() never raises. v2.4.0 (2026-07-04) connected that to GitHub directly — GitHubPRClient plus trelix review --pr owner/repo#N --post-comments, authenticating only via GITHUB_TOKEN, handling all seven GitHub file-status values, and warning past a 3,000-file truncation limit. v2.7.0 Phase 3 (PR #57) closed the loop with .github/workflows/trelix-review.yml, which runs that same review command on every PR and posts findings as GitHub Check annotations with file and line references — continue-on-error: true on the indexing step, because CI runners without local embedding models shouldn't fail the whole workflow. The same phase shipped workspace-vscode/, a VS Code extension scaffold with trelix.search and trelix.ask commands, talking to the existing trelix-mcp package over stdio — no new backend, just a new front door.

The other axis of growth was going multi-repo. RepoRegistry (v2.3.0) manages ~/.config/trelix/repos.json, and FederatedRetriever fans a query out across every registered repo in parallel, RRF-merges the results, and dedupes by (file_path, symbol_id) — crash-safe, returning [] if every repo fails rather than propagating one bad repo's exception. v2.4.0 added a SHA-256-keyed TTL cache (cache_ttl=120.0) tuned for the query patterns of an actual debugging session, where you ask variations of the same question five times in ten minutes. v2.7.0 Phase 2 (PR #56) pushed federation further with make_scip_symbol_id() — stable, SCIP-style cross-repo symbol IDs, sha256-truncated and pipe-separated so scoped npm packages like @scope/pkg resolve unambiguously — and DiffEmbedder, a CCRep-style (arXiv:2302.03924) before/after body-pair encoder for PR diff hunks. search_similar_diffs() finds historically similar changes via cosine similarity, with a NaN guard and dimension-mismatch protection baked in from day one, because I'd already been burned once by silent dimension corruption.

Scale work, and holding the frontier to a real bar

v2.6.0 (2026-07-08) tackled the two things that get expensive as a codebase grows: recomputing the whole community graph on every file save, and blocking on a full index pass before you can search anything. The DF Louvain frontier heuristic (compute_affected_frontier(), detect_communities_incremental(), arXiv:2404.19634) reprocesses only the seed nodes, their neighbors, and their existing community members — falling back to a full recompute only when the affected frontier exceeds 50% of the graph. TRELIX_INDEXER_STREAMING=true (v2.7.0 Phase 2) makes indexing itself lazy — _iter_files() yields files one at a time into a bounded Queue(maxsize=64), with a try/finally guarantee that the producer sentinel always gets sent even on an exception. It's off by default, and I mean that literally: zero behavior change on the path everyone is actually running.

I also shipped two things I'm not willing to oversell. The XTR late-interaction reranker (NeurIPS 2023, arXiv:2304.01982) is cheaper than ColBERT/PLAID by reusing tokens you already retrieved instead of reloading every document's full token set — that's a genuinely good idea. But it's explicitly marked EXPERIMENTAL in the changelog, it emits a UserWarning on first use, and it has not been benchmarked against CoIR or CoREB on code-specific retrieval. PLAID stays the production-validated default. Same discipline applies to the GroUSE-inspired synthesis harness (arXiv:2409.06595, COLING 2025) — SynthesisEvalHarness scores hallucination, completeness, and faithfulness across seven failure modes, because I'd been leaning on "does GPT-4 think this answer sounds right" as an implicit quality bar, and that correlation is not a substitute for actually checking whether the citations are real.

Test count tells the same story as the changelog: 929 unit tests at the v1.0.0 baseline, 1,467 unit plus 41 MCP tests — 1,508 total — by v2.7.0. pip install "trelix[local]" still gets you a fully offline setup, and the index is still one SQLite file. Growing the surface area from three retrieval legs to seven, plus a knowledge graph, an agentic loop, and federation, didn't require growing the infrastructure footprint at all — every new leg, every graph feature, every federation layer is opt-in behind a config flag that defaults to off. That was a deliberate constraint, not an accident, and it's the one I'm least willing to relax as this keeps growing.

I Built trelix Because I Was Tired of Grepping My Way Through Codebases

SAI RAM — Sun, 05 Jul 2026 11:38:03 +0000

I spent my most of day's on a new team grepping through 80,000 lines of code trying to find where authentication worked.

Four hours. Three teammates interrupted. Twelve dead ends across files I didn't understand. The code was fine — it was well-written, well-organized, reasonably documented. The tooling was the problem. I was using grep to understand something that wasn't a text search problem. Code has structure: call edges, import chains, type hierarchies, AST relationships. Grep ignores all of it.

That day stuck with me. I kept running into the same pattern on different teams, different codebases, different languages. Every time I joined something new or came back to a project after six months away, the first few days were archaeology. Tracing calls manually. Reconstructing context that should have been queryable.

I built trelix to fix this. It's an open-source code intelligence engine that indexes any repository with Tree-sitter, embeds every symbol, and answers natural-language questions using hybrid BM25 + vector + call-graph search. It works offline. No API key needed. Zero infrastructure.

pip install "trelix[local]"
trelix index ./my-repo
trelix ask ./my-repo "how does the authentication middleware work?"

The Problem With Code Search

The tools we have for understanding code — editors, grep, ctags, language servers — were designed for writing code, not for understanding it at scale. They're excellent at navigating to a known destination. They're poor at answering questions like "how does the request lifecycle work end-to-end?" or "what calls this function, and what does that caller depend on?" when you don't already know the answer.

The fundamental limitation of grep is that it treats your codebase as a document corpus. It finds strings. Code isn't a document corpus — it's a graph. Functions call other functions. Modules import other modules. Classes extend other classes. When you ask "how does authentication work?", the answer isn't a file or even a few files. It's a traversal of that graph, starting from a semantic entry point and following edges to collect the relevant context.

Vector search solves part of this — semantic similarity gets you closer to the right files without knowing the exact tokens. But pure vector search misses structural relationships. It doesn't know that UserRepository.get_by_token() is always called by AuthMiddleware.verify() which is called by every protected route handler. That's call-graph knowledge, not embedding knowledge.

trelix uses both.

What trelix Does

trelix indexes any repository into a single SQLite file (.trelix/index.db) and then answers questions about it.

The index contains: every symbol extracted via Tree-sitter (functions, classes, methods, their bodies and line spans); call edges and import edges between symbols and files; a hybrid search index combining sqlite-vec HNSW vectors with FTS5 BM25; and since v2.1.0, a Code Property Graph that unifies all of the above into a traversable NetworkX graph.

A query like trelix ask ./repo "explain how authentication works" goes through a 3-tier adaptive router:

Tier 1 (Direct) — for simple factual patterns like "what is X" or "define X", trelix skips retrieval entirely and answers from the LLM directly. No unnecessary round-trips.

Tier 2 (8-intent) — for most code queries, it classifies the intent into one of eight categories (symbol_lookup, feature_flow, dependency_map, blast_radius, etc.) and runs the appropriate retrieval strategy.

Tier 3 (Multi-step) — for complex queries like "walk me through the request lifecycle end-to-end", it decomposes the question into 2-3 sub-queries, runs each independently, and merges the results.

Results from all active retrieval legs are fused via Reciprocal Rank Fusion (k=60) before being assembled into the context window for LLM synthesis.

How It Actually Works

The indexing pipeline runs in four phases:

Phase 1 (Parse) — Tree-sitter walks every file and extracts symbols with their source, line spans, and AST structure. Runs in parallel via ThreadPoolExecutor.

Phase 2 (Write) — Symbols and chunks are written to SQLite. Cross-file parent_id relationships are resolved.

Phase 3 (Embed) — Every chunk is embedded asynchronously in batches of 4 concurrent API calls. With the local provider (sentence-transformers, no API key), this runs entirely offline.

Phase 4 (Resolve) — Cross-file call edges are resolved with a 3-priority strategy: qualified name first, then type_hint+name, then name-only fallback. This gives about 40% fewer false-positive cross-file edges compared to name-only matching.

The result is a single .trelix/index.db file that contains everything: vectors, BM25, call graph, import graph, symbols, file hashes for incremental updates.

Zero Infrastructure, Full Power

This was a deliberate design decision and one I keep coming back to.

Most code intelligence tools require running a vector database, a relational database, and often a separate API server. That's a lot of infrastructure to maintain for what is fundamentally a local developer tool. trelix's default is a single SQLite file using sqlite-vec for HNSW vector search and FTS5 for BM25. Zero external infrastructure. Works on a laptop with no internet connection.

When you need to scale: LanceDB backend for 100k+ chunks (3-5× faster vector insert on ARM/Apple Silicon), Qdrant for 500k+ chunk deployments with multi-repo shared collections. But the default handles most codebases and most developers will never need to switch.

# Default (sqlite) — up to ~100k chunks
trelix index ./my-repo

# LanceDB — 100k+ chunks
TRELIX_STORE_BACKEND=lance trelix index ./my-repo

# Qdrant — 500k+ chunks
TRELIX_STORE_BACKEND=qdrant trelix index ./my-repo

Beast Mode: Seven Retrieval Legs

The default setup (BM25 + vector + grep + call graph) handles most questions well. But trelix has five additional retrieval legs that you can enable when you need higher recall or more sophisticated query handling:

Leg 5: File-summary semantic search — RAPTOR-style (arXiv:2401.18059). At index time, trelix generates LLM summaries of every file and embeds those summaries separately. This surface is especially good for "explain this codebase" or "what files deal with payment processing?" queries — questions where the answer is at the file level, not the symbol level.

Leg 6: SPLADE-Code — sparse+dense hybrid via learned sparse retrieval. SPLADE encodes queries into sparse high-dimensional token vectors, expanding vocabulary beyond exact matches in a way that complements both BM25 and dense vector search.

Leg 7: Multi-granularity — indexes code at block AND statement level simultaneously. Some queries are better answered by a full function body; others are better answered by a single statement. Having both granularities in the index improves recall on precise questions.

Plus query-side enhancements: HyDE (generates a hypothetical code answer as the ANN query vector, improving recall on abstract questions), FLARE (confidence-gated re-retrieval — when synthesis spans show uncertainty, trelix re-queries before finalizing the answer), and since v2.2.0, an agentic ReAct loop that does multi-turn retrieve→observe→re-retrieve with self-correction.

# Enable everything
TRELIX_RETRIEVAL_AGENTIC=true \
TRELIX_GRAPH_SEARCH_ENABLED=true \
TRELIX_RETRIEVAL_FILE_SUMMARY_LEG=true \
TRELIX_RETRIEVAL_HYDE_FALLBACK=true \
TRELIX_RETRIEVAL_FLARE=true \
TRELIX_RETRIEVAL_SPARSE=true \
TRELIX_CHUNKER_MULTI_GRANULARITY=true \
trelix ask ./my-repo "explain the full request lifecycle"

The Features I'm Most Proud Of

GitHub PR review. This is v2.4.0 and it's become one of the most-used features. trelix review --pr owner/repo#42 fetches the PR diff from GitHub, retrieves codebase context for each changed hunk, runs an LLM review, and can post findings back as a single batched review comment with --post-comments. The key insight is that reviewing a diff without understanding the surrounding codebase is like proofreading a sentence you've never read before.

trelix review --pr sairam0424/trelix#42
trelix review --pr sairam0424/trelix#42 --post-comments

Federated search. trelix search-all "query" fans out across all registered repos in parallel via ThreadPoolExecutor and RRF-merges the results. With trelix watch-all, a single watchfiles.awatch() call watches all registered repos simultaneously. The TTL cache on FederatedRetriever gives about 90% hit rate for typical debugging-session query patterns.

trelix federation add api ./services/api
trelix federation add web ./services/web
trelix search-all "JWT validation"
trelix watch-all

MCP integration. One command and trelix is available inside Claude Code, Cursor, Windsurf, and Continue.dev:

pip install trelix-mcp
claude mcp add trelix -- trelix-mcp

Then inside Claude Code: "index my repo at /path/to/repo, then find how authentication works".

What Surprised Me Building This

I expected the hardest part to be the embedding and retrieval architecture. It wasn't. The hardest part was making the system opinionated enough to be useful without being so opinionated that it broke on unusual codebases.

The call-graph resolver was the most representative example. My first version used name-only matching for cross-file call edges — login() in file A calls login() in file B. This produced a dense, noisy graph with maybe 40% false-positive edges. The fix was a 3-priority resolution strategy: try qualified name first (most precise, lowest recall), then type hint + name (moderate precision), then name-only as fallback. That reduced false positives significantly while maintaining recall on codebases that don't have full type annotations.

The other thing that surprised me was how much value came from the structural metadata rather than the semantic embeddings. The call graph, import graph, and type hierarchy are what make trelix's answers qualitatively different from a vector search over code files. Semantic similarity gets you to the right neighborhood. Graph traversal gets you to the right answer.

What I'm Still Uncertain About

The 3-tier query router works well for the queries I've tested it on. I'm less confident about it on very large codebases (millions of lines) where the graph becomes expensive to traverse. The current implementation caps BFS depth at 2, which is usually right but occasionally misses important connections. I'm still figuring out the right heuristics for adaptive depth.

I'm also still calibrating the GraphRAG map-reduce threshold. The current default (activate at >20 results or >8k tokens) is conservative. For some query types it activates too eagerly; for others, not eagerly enough. This is the main retrieval parameter I'm watching in practice.

Try It

# Offline — no API key
pip install "trelix[local]"
trelix index ./your-repo
trelix ask ./your-repo "how does your main feature work?"

# With LLM synthesis
pip install trelix
export OPENAI_API_KEY=sk-...
trelix ask ./your-repo "explain the request lifecycle end-to-end"

# MCP in Claude Code
pip install trelix-mcp
claude mcp add trelix -- trelix-mcp

# Review a PR
trelix review --pr owner/repo#42 --post-comments

Everything is MIT licensed, on PyPI, and at github.com/sairam0424/trelix. The full documentation is in the repo README including the beast-mode activation block if you want all seven retrieval legs at once.

What's the longest you've spent trying to understand a piece of code you didn't write? I've had 4-hour archaeology sessions on codebases with good documentation. I'd like to know how much of that time you think was the code being genuinely complex versus the tooling failing you.