DEV Community: constantine makoid

The documentation is a person, and they are on holiday

constantine makoid — Fri, 04 Sep 2026 08:39:00 +0000

Before publishing: set published: true, and check canonical_url — dev.to needs the
article to already exist at that URL, so it goes up on the site first. cover_image points
at the existing product cover; swap it for a documentation-specific one if that gets made.

There is a diagram your organisation has never drawn. On it, one box holds most of what is
known about how the system works: which service owns the settlement flow, why the retry lives
on the consumer and not the producer. The box is a person. They are competent, generous with
their time, and currently in Croatia.

Everyone knows this. What gets missed is that the wiki does not fix it. The wiki was written
during a documentation push eighteen months ago and has been quietly wrong since the week
after. The README describes a deployment that no longer exists. The architecture page still
shows the monolith that was split in March.

An absent document and a false one are not the same failure. An absent one sends you to the
code, which is annoying and correct. A false one sends you somewhere confidently, and you find
out forty minutes later, having built a mental model on top of it. The wiki nobody trusts is not
neutral. It is a trap already sprung on several new joiners, and the response is always the
same: ask the person.

So the person is the interface, and the wiki is a decoy pointing at them.

Two ways to go stale

Take any statement about a codebase — "requests are authenticated in the gateway". At some
moment it stops being true. What happens then is the entire distinction.

Hand-written documentation goes stale silently. There is no event. Somebody merges a pull
request, and a page in Confluence becomes false. Nothing fires, no build turns red, the
timestamp does not move. Its confidence is unchanged, which is the problem: the page is now
more sure of itself than the code justifies. A stranger discovers it months later, at the worst
possible time, and between falsity and discovery the document costs more than not having it.

Documentation derived from the code goes stale loudly, because the generator runs again.
The commit that made a statement false is the input to the next run. Staleness stops being an
unobserved property of a page and becomes an event with a timestamp: a job, a diff, a
regenerated file, a failure. You can put that on a dashboard. You cannot page on Confluence
being wrong, because nothing knows.

That is the whole argument, and it is smaller than it sounds. Not that machines write better
prose — they very often do not. It is a claim about observability of decay, and decay is what
kills documentation, not prose quality.

Where the loud kind fails, before I sell you on it

Four honest limits, because the sales version of this argument skips all of them.

A generator recovers what, never why. It can tell you this module publishes to that
topic. It cannot tell you the retry sits on the consumer because of an incident in 2023 and a
vendor who would not fix their idempotency. That was never in the code and no parsing recovers
it; it lives in the person in Croatia. Generated documentation shrinks the set of questions that
need a person; it does not empty it.

Regeneration is not free. Every regenerated page is model spend. A system that rewrites
everything on every commit gets switched off, and a generator that is switched off is a
hand-written wiki with extra steps.

Silent success is the real bug. A run that produces nothing looks identical from outside to
a run that worked: job completed, no errors, folder empty. If a pipeline fails quietly, the
loudness you paid for is gone.

Derived documentation must name what it does not cover. A generated set that silently skips
six of nine services is the same trap as the stale wiki, in a fresher timestamp.

Build this yourself and those four are the specification. Below is one worked answer.

One worked answer

This part is about Celmis, which is mine — a self-hosted
platform for the middle of a development cycle, AGPL-3.0-or-later, under docker compose. It
keeps an index of the code current, and several things come out of that index: alerts you can
act on, a dependency and vulnerability picture, questions answered across repository
boundaries. Documentation is the one this piece is about. Every claim below is in the tree,
including the two places where it does less than you would assume.

Noticing. A scheduled sweep, once a day by default, does one git ls-remote per registered
repository: one round trip, no clone, no fetch. It records four outcomes, not two — up to date,
behind, never indexed, and could not tell. The fourth matters most: a check that cannot reach
the remote and renders as "no new changes" answers wrongly while wearing a fresh timestamp. The
sweep is the floor under the push webhook: a webhook only fires where somebody registered one.

Narrowing. When the branch has moved, the re-index is incremental: it walks
git diff last_sha..HEAD and touches only what changed. A reverse index — built by reading
which source files each generated note names in its frontmatter — resolves those changed files
back to the documents that name them. Their vectors are dropped from the search index at once,
so search stops returning removed content before any rewrite starts. Module notes are then
rewritten against the current code; everything else is re-embedded from disk, which restores
search without paying a model for prose that may not have moved.

And the limit I would rather write than have you find. Feature and integration notes record
their symbols as file::name graph ids, so a changed file resolves to them. A module note
records a directory plus bare symbol names, and the lookup is an exact match on the changed
file's full path — so editing src/auth/login.py does not currently select modules/auth.md.
The note type that gets the real rewrite is the one the index is worst at selecting. Thirty
lines in src/vault/reverse_index.py, and exactly the class of bug a hand-written wiki has no
way to even have.

Not paying twice. Each note carries the commit it was written from, and for module notes —
the bulk of a vault — a matching commit is skipped outright next run. Better: when the commit
differs but none of that module's files changed between the two, the stamp is advanced and no
model is called. If git cannot answer — a shallow clone, a missing revision — the code
regenerates rather than skips. Feature and integration notes carry the same stamp but are not
yet gated on it: generate twice at one commit and you pay for those twice.

Refusing. Ask it to document a repository that was never indexed and it refuses that
repository by name and says why, rather than writing pages from filenames. Ask for everything,
and a missing_only condition means what people actually want — the services that have none —
not a rewrite of all of them.

Failing loudly. A run in which every document failed used to report success, because the
markdown folder existed. It now raises. So does a subtler one: three production jobs reported
completed, attempts: 1, last_error: null and wrote zero vectors — the documents existed,
only the embedding half had failed. The job is not done unless both halves land.

Saying what produced it. Every note a model wrote carries a provenance block: generator and
version, commit, engine, model, timestamp, and — the useful one — the number of index lookups
the writer made. There are two engines. One packs code and metadata into a single prompt.
The other is an agent holding the platform's own MCP tools and nothing else — no shell, no file
reads, no grep — so it cannot open a file and can only ask the index: what calls this, what is
the public surface, what is deprecated. A document written after twelve lookups deserves
different weight from one written from a single prompt: on an agent-written document that count
is in the frontmatter, not a log, and a single-prompt document names its engine and carries no
count. The three assembled pages — index, architecture overview, security findings — come from
the graph and the scanner rather than a model, and carry no block yet. An awkward gap: the
architecture page is the one most likely to be exported and forwarded.

The block survives export, and the model name is rewritten on the way out to drop this
installation's own workspace identifier, which would otherwise ride into an emailed Word file
and tell a stranger nothing but an internal id. That redaction is on the export path, not the
stored note: open the frontmatter on disk and the full deployment name is still there.

Naming the gaps. Exporting every repository's documentation as one archive writes a
MISSING.txt listing the repositories that have none. A download that silently covers six of
nine services is the failure this is meant to avoid, and counting folders is not how anyone
finds out.

Underneath is a tree-sitter symbol graph — hand-written extractors for the major languages,
generic tag-query extractors for a long tail, and Dockerfiles, compose files, Kubernetes
manifests, Helm charts, Terraform and CI workflows in the same graph, not a separate one.
Repositories can be grouped, and deployment references that cross a repository boundary — a
compose or Kubernetes image, a build context — are resolved to the repository that provides them
and stored as edges in the group's graph; cross-repository calls and imports are not
materialised that way yet. When the platform answers in prose, the citations it renders as links
are re-checked against the files on disk: the file must exist, the cited line must be inside it,
and any quoted code must actually appear there — that last check exists because a model once
invented a plausible function and hung it under a real path. Failures are reported beside the
answer, not dropped.

What it still does not do

It does not know why. It does not replace the person in Croatia; it shortens the queue outside
their inbox and moves the questions in it from "how does this work" towards "why did we decide
this". It cannot document a repository it has not indexed, and says so rather than guessing. It
costs model spend, bounded but not zero. And two of its own mechanisms are narrower than you
would assume — which is why they are named above rather than left for you to trip over.

None of that is fixed by better prose. It is fixed by documentation being an artefact with a
build, a provenance record, and a failure mode that shows up on a dashboard instead of in the
face of a new joiner nine months from now.

The wiki cannot fail. That is what is wrong with it.

A memory server remembers your conversation. That is not the same as knowing your code.

constantine makoid — Tue, 01 Sep 2026 12:43:48 +0000

Before publishing: set published: true, and check canonical_url — the article must
exist at that URL on the site first. Without it this competes with the original in search
instead of pointing at it. Tags are from the verified top-1,000 list; mcp was not in that
cache and is not used here.

A session ends. Your agent had worked out, over forty minutes, that the retry logic lives in
one service and the thing that gives up on it lives in another, that the queue name is spelled
two different ways, and that the person to ask about any of it left last year. Tomorrow you
open a new session and it knows none of that. Neither does your colleague's session. Neither
does the agent reviewing the pull request that comes out of it.

It is the same forty minutes a new engineer spends in week one, and the same forty minutes the
README would have saved if it were still true. It is why a manager asking "where is this up to"
has to interrupt someone who knows. The knowledge exists; it has nowhere to live but in people
and chat logs.

The reflex is to reach for memory. That reflex is worth interrogating, because there are two
different problems hiding under one word, and only one of them is what memory servers are for.

What MCP actually specifies

It helps to be exact, because "MCP memory" gets said as though it were a feature of the
protocol. It is not — and the current revision makes that harder to miss rather than easier.

Read the base protocol's own three-line summary in revision 2026-07-28: JSON-RPC message
format, stateless, self-contained requests, per-request capability negotiation. Servers
offer three features — Resources, Prompts and Tools. Clients offer one: Elicitation. Sampling
and Roots, which used to make that three, were deprecated in this same revision under SEP-2577,
along with Logging and Dynamic Client Registration; the migration note against Sampling reads
"integrate directly with LLM provider APIs".

There is no memory primitive and no persistence primitive. There is no memory extension
either — the official list is the two authorization extensions, MCP Apps, Skills over MCP, and
Tasks. Tasks is the one worth pre-empting, because it advertises "durable handles" and that
sounds adjacent: a task ID survives a disconnect so a client can resume polling a long-running
call, it carries a TTL, and what it holds is the status and eventual result of that one call.
It is durable in the sense a job ID is durable. It is not somewhere knowledge goes to live.

So every memory you have seen over MCP is a server implementing memory with ordinary tools.
Anthropic's own Memory MCP server keeps a knowledge graph in a JSONL file; other community
servers do the same job over different storage. Claude Code, separately from MCP altogether,
ships Auto Memory — a MEMORY.md per project, on by default since 2.1.59.

This is a compliment to those projects, not a criticism. They took a gap the protocol
deliberately left open and filled it with plain tools — exactly what the tool primitive is for.

What they remember, and why that is the right thing for them to remember

A memory server stores what was said. Your preferences. The decision you made on Tuesday and
the reason you gave. The fact that you want British spelling and no bullet lists. Entities,
relations and observations, accumulated from conversation.

Two properties of that content matter here.

It is personal. My conversational memory is a bad thing to hand to you. It contains my
half-formed conclusions, my shortcuts, and things I said and then revised. Sharing it is not
an unimplemented feature; it is a category error. The value is that it is mine.

It is stated. It is true because someone asserted it. Nothing in the repository can
contradict it, because it was never derived from the repository in the first place.

For remembering a conversation, both properties are correct. Now hold them against the other
problem.

Knowing the code has the opposite shape

"Where is this symbol used, in every repository, with file and line" is not personal. There is
exactly one right answer and everybody who has read access to those repositories is entitled
to the same one. Storing it per person means storing the same fact many times and being wrong
in a different way in each copy.

It is not stated, either. It is derived — a function of the current commit, and therefore
perishable in a way conversational memory is not. If I told you on Tuesday that I prefer
tabs, that is still true on Friday. If an agent noted on Tuesday that apply_refund had three
callers, that is a claim about a commit, and a merge on Wednesday can make it false without
anyone touching the note. A memory store cannot know that happened, because nothing said it.

That is the failure mode worth naming, and it is the same one that makes stale documentation
worse than none. A remembered fact about code does not decay into silence. It decays into a
confident wrong answer, indistinguishable in tone from a right one.

So: conversation is personal, stated and durable. Code knowledge is shared, derived and
perishable. Same word, opposite requirements. The fix for the second is not a better memory —
it is not memory at all. It is an index that is rebuilt from the code, that many callers read,
and that has a stated relationship to a commit.

One implementation, which you can read

I build Celmis and it is mine, so treat the description as a
worked example rather than a recommendation. It is a self-hosted platform for most of a
development cycle — the alert that arrives, the fix that goes out, the dependency and SBOM
evidence underneath — and what this article is about is one layer of it: the index. It is
AGPL-3.0, with a carve-out for ee/ that today holds no product code, and every claim below
names the file, so you can disagree with me by reading it.

Here is the wider loop, so the layer this article is about has something to be a layer
of. Thirty-four seconds, one real incident, every frame a screenshot of it:

It builds a tree-sitter symbol graph per repository and serves that same graph over MCP. Eight
languages have hand-written extractors — TypeScript, Vue, Python, Go, PHP, Java, C#, C++ — and
sixteen more (Ruby, Rust, Kotlin, Swift, Scala, Elixir, Dart, Lua, R, Solidity, OCaml, F#, Elm,
Gleam, Racket, Fortran) come from the grammar authors' own tags queries, registered at the
lowest priority so a real extractor always wins. Dockerfiles, Compose files, Helm charts and CI
workflows are matched by filename or path rather than suffix, and Kubernetes manifests by
sniffing the first few kilobytes for apiVersion and kind. Terraform is the ordinary case —
.tf.

The HTTP mount registers 23 tools, eighteen of which read and five of which write. A sample of
what the read half returns:

search_symbols — name, kind, file, line, signature and repo slug, for definitions matching a name across every repository in a project.
find_consumers — who calls a symbol, with repo, file and line. The list of what breaks.
get_api_surface — functions whose names look like HTTP handlers, with a route path guessed from the name by turning underscores into slashes, plus method, file and line. A name-convention heuristic over the symbol index and nothing more; it does not read route decorators, and the source says so.
get_architecture — the cached orientation summary for a repository, with the model that produced it and the timestamp, so you can see how old it is.
get_owner — top git-blame authors plus matched CODEOWNERS entries for a path. This is the one that answers "who do I ask" without asking.
list_accessible_repos and get_my_access — the agent asking what it is allowed to look at, and being told which path globs are denied.
start_integration_walk — an ordered checklist of the other tool calls, returned as data, for a client that would otherwise fire ten guesses.

Three things about that list are load-bearing.

It is the same index, not a copy. The MCP tools open settings.repo_graph_path(slug). So
does the retrieval layer answering a human's question in the web UI. So does the pull-request
reviewer. One graph file per repository, three readers. Celmis does hold its own clone and index
it — one copy, on your infrastructure, not one per agent or per session.

It is the same access rules, with one exception I will name. Every tool that returns code,
ownership or review content calls caller_access, which calls resolve_access from
src/access — the identical function behind the human REST endpoint and behind multi-repository
question answering. A repository you may not research is omitted and named in blocked_repos;
a denied path is filtered out of the matches, and search_symbols also reports how many it hid,
in hidden_symbol_count, though not every tool yet returns that count. The exception:
list_deprecations reads its table today without a workspace or access predicate. That is a
bug on my side, not a design, and it is being fixed. While I am being exact: the scope filter on
tools/list is listing hygiene, not authorisation — a scoped read client is not shown the
write tools, but the HTTP mount carries no per-call scope check, and a token with no scopes sees
everything. Writes are gated on the token resolving to a workspace, not on its scopes. Do not
treat a read scope as a boundary.

It has a stated relationship to a commit. A daily sweep runs git ls-remote — one network
round trip, no clone, no fetch — and compares the branch head with last_indexed_sha. It
reports three outcomes, not two: up to date, behind, and could not tell, because a check that
cannot reach the remote and renders as "no new changes" is worse than no check. Behind enqueues
an incremental pass that diffs last_sha..HEAD, drops the symbols for every touched file and
re-extracts them, rather than re-parsing the whole repository because one file moved.

The stdio server carries a raw-Cypher escape hatch too: it tokenises the query and rejects any
of eleven write keywords before running it — a denylist, not a parser, as its own docstring says.
It is not one of the HTTP mount's 23.

What this does not do

Celmis's MCP server has no memory. Zero hits for memory, persist, recall or remember across
src/mcp_server/. It will not remember that you decided to deprecate the old endpoint, or why.
That is conversational, it is yours, and a memory server is the right tool for it — run one
alongside, and let it keep the decisions while the index keeps the code.

The honest claim is narrower than "your agent finally has long-term memory", and better. The
next engineer's agent starts out knowing what yours knew about the code, because that knowledge
was never in your chat log to begin with. It is in an index, derived from the commit, readable
by every session under the same rules, and rebuilt when the commit moves.

I build Celmis — self-hosted, AGPL-3.0 (ee/ carve-out, empty
today), docker compose. Its pull-request reviewer placed 17th of 50 on the Martian Code Review
Bench offline set — seventeenth under all three judges, F1 between 42.7% and 47.5% depending on
who is judging. That is a deliberately unflattering number about one surface of the product and
it stays. The cross-repository capability described above contributed nothing to it: the
benchmark set is isolated single-repository pull requests and the graph came back empty on all
50, so nothing in that table is evidence for or against this path. Check it the way I would want
it checked — register two services, put them in one project, call find_consumers with a symbol
you know one calls in the other, and see whether the repo, file and line are right.

Introducing Celmis: self-hosted code intelligence over a symbol graph

constantine makoid — Thu, 27 Aug 2026 16:37:48 +0000

Celmis is self-hosted code intelligence. It reads a set of repositories once and
keeps a symbol graph of them; asking questions, reviewing pull requests, auditing
dependencies, routing alerts and serving an MCP endpoint are then all different ways of
reading that one index, rather than five products each holding their own copy of your
code. It runs on one machine under docker compose, with whichever model provider you
already pay for behind it.

This is the introduction — what it does, what it refuses to do, and how to run it. The
quickest way to say why it exists is to show the thing a diff-only tool structurally
cannot do.

One question, two repositories

I asked a question that spanned two repositories, and the answer quoted both.

The question was ordinary: "How does the gateway talk to the payments service? Name the
function on each side."

The answer found a Python publisher:

def publish(self, batch_id: str, entries: dict[str, int]) -> None:
    """Emit one settlement event on the published topic."""
    self.producer.send(
        SETTLEMENT_TOPIC,
        json.dumps({"batch_id": batch_id, "entries": entries}).encode(),
    )

and a TypeScript listener in a different repository:

start(): void {
  this.bus.on(SETTLEMENT_TOPIC, (event) => {
    const payload = JSON.stringify({ type: "settlement", ...event });
    for (const s of this.sockets) s.send(payload);
  });
}

They never call each other. They meet on a Kafka topic. Then the answer added something
nobody asked for:

Duplicated contract. The topic name (payments.settlement.v2) and the event payload
structure are hardcoded in two separate repositories — src/config.py in payments and
src/contract.ts in the gateway. Changing the topic name or the payload schema in one
repository without updating the other will silently break the integration.

A reviewer that reads only the diff cannot say that. It never had the other repository
open. That is not a model-quality problem — no amount of reasoning recovers a file that
was never in the context.

The shape of the idea

Read the repositories once. Build a symbol graph — deterministically, with tree-sitter,
no model involved. Then everything else is a different way of reading that one index
rather than a separate product with its own copy of your code:

Ask questions, answered with file:line citations, across repository and language boundaries.
Review pull requests, with knowledge of who else calls the function in the diff — including from a repository not in the pull request.
Audit dependencies, and produce the artefacts a buyer or an auditor asks for.
Route what your running services are shouting about — alerts arrive, and the index already knows which repository the failing service is, and who owns it.
Serve it over MCP, so your own editor or agent reads the same index under the same access rules.

The index is the product. The rest are surfaces.

What the review actually catches

On a real pull request in a demo repository, three findings, all on real lines:

6:  const first = vals[0];              // vals is {} — undefined, not a TypeError
7:  for (let i = 0; i <= vals.length; i++)   // off by one
16: const n = parseInt(raw);            // never throws, so the catch below is dead

Three inline comments with suggestion blocks, one summary comment, zero false
positives on that run. I verified all three against the file in the PR branch by hand,
because a review tool that is right four times out of five is a tool you stop reading.

The MCP server

Eighteen tools over the same index. Here is a real session, trimmed:

--> initialize
<-- 200   serverInfo: { "name": "celmis", "version": "1.29.1" }

--> tools/list
<-- 200   18 tools
          list_projects        get_api_surface       bootstrap_client
          search_symbols       list_accessible_repos start_integration_walk
          find_consumers       get_review            route_incident
          get_owner            get_review_policy     get_dep_audit
          get_architecture     list_deprecations     list_dep_findings
          …

--> tools/call  search_symbols
          { "project_id": "083bd97a-…", "query": "SETTLEMENT_TOPIC" }
<-- 200
          {
            "matches": [
              { "repo_slug": "…celmis-demo-gateway",  "kind": "variable",
                "file": "src/contract.ts", "line": 2 },
              { "repo_slug": "…celmis-demo-payments", "kind": "constant",
                "file": "src/config.py",   "line": 9 }
            ],
            "count": 2
          }

One query. Two repositories, two languages, the same contract symbol — from a client that
has never checked either of them out. find_consumers is the one I use most: it answers
"what breaks if I change this" across the whole set.

Alerts, and why they belong on the same index

This is the surface I nearly left out of this article, which was a mistake, because it is
the one that closes the loop.

Your services are already producing alerts. They land in a channel where somebody has to
work out which repository the failing service actually is, who owns it, and whether the
thing that broke was touched recently. That lookup is the expensive part — not the alert.

So the same index answers it. An ingest endpoint takes the alert, a binding routes it, and
the card arrives in chat:

POST /webhook/alerts/{token}
  { "severity": "critical",
    "repo_hint": "celmis-codereviewer/celmis-demo-gateway",
    "title": "checkout: unhandled exception in settle()" }

→ notif_delivered event=alert_received
  repo=celmis-codereviewer/celmis-demo-gateway severity=critical

Review results ride the same rails — a finished pull-request review posts its own card:
Review CHANGES · PR #4 — 0 critical · 3 error · 0 warn · 0 info.

Two things worth stealing from how this is wired, both of which I got wrong first:

The webhook signature is checked before anything else. Wrong signature → 401.
Correct → 202. Replay the exact same delivery → {"status": "duplicate"} rather than a
second review and a second bill. Delivery IDs are cheap; duplicated model calls are not.

A failed channel test must not echo the URL it tested. Google Chat webhook URLs carry
key and token in the query string — the URL is a credential. httpx puts the
request URL in the exception text, and an early version of the endpoint returned
str(exc) verbatim, which meant a failed test handed the caller back the secret it was
testing. If you are building anything that tests a user-supplied webhook, go and check
that path in your own code right now.

The surface that quietly became urgent

The dependency audit is the one part that is deterministic end to end — native
auditors and OSV.dev, no LLM in the loop. It produces two things.

A CycloneDX SBOM, and an evidence pack:

sbom/<repo>.cdx.json   CycloneDX, one per repository
findings.json          what was found against those components
timeline.jsonl         when each fact entered the record
MANIFEST.json          sha256 of every file above

That last file is the point. An archive of files is not evidence — nothing in it stops
the contents from having been edited afterwards. A manifest of digests means a third
party can verify the pack without trusting the machine that produced it:

{
  "algorithm": "sha256",
  "files": {
    "findings.json": "921412d4bf97eb32fa4b3e8ad09447dab762f19d31ffec45bddb6d7962bf08e5",
    "sbom/gateway.cdx.json": "6eadeafad0043b85a51b65ddddba84ef7b43081064ed9e1b320c62838ddc3d8e",
    "timeline.jsonl": "b1815a8e3607712aa9332cd1f8d2d0ec6f93cd8dce53dd92844b8edcde790b3d"
  },
  "generated_at": "2026-08-26T19:09:41Z"
}

Why now: from 11 September 2026, under the EU Cyber Resilience Act, manufacturers
must report an actively exploited vulnerability to ENISA and their national CSIRT within
24 hours. The SBOM itself is not mandated until December 2027 — which is the trap,
because on a 24-hour clock the first question is not how to word the notification. It is
whether you ship the component at all, in which service, at which version.

The document is due in 2027. The visibility it describes is needed fifteen months earlier.

The part I am most attached to

The audit reports what it could not check, as prominently as what it found:

Not fully checked (4). Treat a zero here as unknown, not as safe.

- …demo-gateway — npm via npm-audit: no lock file
  (package-lock.json / pnpm-lock.yaml / yarn.lock) — cannot resolve the tree
- …demo-gateway — all via osv-scanner: recognised no manifest or lock file here
- …e2e-probe/requirements.txt — PyPI via pip-audit: dependency resolution failed
  — audited 4 pinned requirements directly, without the transitive tree

An unchecked ecosystem reports zero vulnerabilities exactly like a clean one. If your
tooling cannot tell you which of the two you are looking at, that gap is the finding.
Steal this behaviour regardless of what you use.

Running it

One machine. Postgres and Qdrant bundled — no external cluster to provision.

git clone https://github.com/Celmis-labs/Celmis.git celmis
cd celmis
./scripts/init-env.sh          # generates .env, every secret in the format it needs
docker compose --env-file .env up -d
docker compose ps

On a clean server that took 197 seconds from git clone to six healthy services —
measured, not estimated. About 1.1 GB of RAM at peak during indexing, 565 MB at rest.

Bring your own model key: Gemini, Anthropic, OpenAI, OpenRouter, Groq or Mistral. A free
Gemini key is enough to evaluate it. No telemetry, no licence check — the only
outbound calls are the ones you configure. AGPLv3, the whole thing, not open core.

The number I am not hiding

17th of 50 on the Martian Code Review Bench offline set, stable under all three
judges.

It measures one of the surfaces above — pull-request review on isolated
single-repository PRs. That set has no sibling service for a symbol to have consumers in,
so the cross-repository work this is built around contributes nothing to the score. It is
on the front page of the site with that explanation next to it rather than instead of it.

I also audited every one of the 79 findings the benchmark counted against us. 33 were
real defects the reference set was silent about. That audit is published in full, with
the code for each one — partly because it is the honest thing to do, and partly because
"our precision is actually higher" is a claim nobody should accept without the evidence
attached.

Known rough edges

Semantic search needs a generated vault. Until you make one, the UI says so in an orange banner rather than quietly returning worse answers.
The execution sandbox has deliberate internet egress, because pip install and npm ci need it.
No long-term support branch. Fixes land on the latest release.
It does not make anyone compliant with anything, and it says so in its own output.

Source, docs and the benchmark audit: https://celmis-labs.github.io

Happy to answer anything — especially about the benchmark methodology, which is the part
that deserves an argument.