ectocheck: hunting the ghosts in an AI's answer, with a local judge that gets audited

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine

What I Built

Every AI answer is a little séance. The model channels something that sounds exactly like your data, in a calm and confident voice, and some of what comes through was never said by anyone. Those are the ghosts: claims that read as fact, cite nothing, and cost you trust the moment you repeat them.

ectocheck is a Go CLI that hunts them. You hand it an AI-generated answer and a folder of local source files (notes, logs, docs). It splits the answer into claims, retrieves candidate evidence from your files, asks a local open-weight model to judge each claim, and then - the part I care about most - it audits the judge. Every citation the judge produces must point at a real excerpt it was actually shown, or the verdict is thrown out. The exit code is 1 unless every claim checks out, so it drops straight into a script or a pipeline.

The theme pushed me here: the scariest thing about LLMs isn't that they're wrong, it's that they're wrong in the same tone of voice as when they're right. I wanted a skeptic at the table.

Demo

The repo ships a small haunted house for testing: examples/homeserver/ holds notes and a backup log for a fictional home server, and examples/answer.md is an "AI answer" about it with two invented claims mixed in - a Pi-hole that never existed and a disk failure that never happened. This is the real output of ectocheck check examples/answer.md --src examples/homeserver --model qwen2.5:1.5b, captured from my terminal, not curated:

✓ SUPPORTED  The home server is a Beelink Mini S12 with an N100 CPU and 16 GB of RAM.
              The evidence [1] directly states that the home server is a Beelink Mini S12 with an N100 CPU and 16 GB of RAM, which matches the claim.
              [E2] examples/homeserver/README.md:3 — The box is a Beelink Mini S12 with an N100 CPU and 16 GB of RAM. It runs Debian 12 and liv…

? UNVERIFIED Nightly backups copy /opt/stacks to /mnt/usb/backups and verify the result.
              model claimed support but cited no valid evidence
              (note: the judge cited evidence it was never shown; verdict not trusted)

✓ SUPPORTED  Jellyfin serves media on port 8096.
              The evidence [1] confirms that Jellyfin services are run as docker compose stacks, which includes serving media on port 8096.
              [E3] examples/homeserver/README.md:6 — Services run as docker compose stacks in /opt/stacks. Jellyfin serves media on port 8096. …

? UNVERIFIED The server also runs Pi-hole for network-wide ad blocking on port 53.
              model claimed support but cited no valid evidence
              (note: the judge cited evidence it was never shown; verdict not trusted)

? UNVERIFIED Backups failed twice last week because the USB disk filled up.
              model claimed support but cited no valid evidence
              (note: the judge cited evidence it was never shown; verdict not trusted)

5 claims: 2 supported, 0 partial, 0 ghosts, 3 unverified
No ghosts, but not every claim could be trusted. Treat unverified claims as unchecked.

Both planted ghosts come back flagged and untrusted, and the exit code is 1. The case I like most is the middle one: a genuinely true claim (the backup schedule) also comes back UNVERIFIED, because the judge asserted support while citing evidence it was never shown. The tool refused to rubber-stamp a right answer for the wrong reason.

To be precise about what happened: the planted claims were caught by citation validation - the judge claimed support but its citations didn't survive checking - not by the judge cleanly identifying them as fabrications. That's the guard doing its job, and it's also the tool's biggest limitation: citation checking stops invented references, not a judge that misreads real evidence. The README's "Honest limitations" section keeps that distinction front and center, including the time the weaker qwen2.5:0.5b model approved an invented claim outright.

Code

DhanushNehru / ectocheck

Hunt the ghosts in an AI's answer: verify every claim against your local source files, with a local open-weight judge whose citations are themselves checked. Go CLI, runs fully offline.

ectocheck

Hunt the ghosts in an AI's answer before you trust it. Work in progress - see the open pull request for the initial implementation.

View on GitHub

The seed commit is on main; the full implementation (CLI, judge, citation validation, tests, examples) sits on feat/initial-implementation in PR #1, which I still have to merge. To run the demo today, check out that branch:

git clone https://github.com/DhanushNehru/ectocheck.git
cd ectocheck && git checkout feat/initial-implementation
go build -o ectocheck .
go test ./...   # 22 tests
ollama serve    # if it isn't running
./ectocheck check examples/answer.md --src examples/homeserver

How I Built It

Go, one binary, standard library only. The judge is qwen2.5:1.5b served by Ollama on the same machine; there are no API keys and no network calls beyond localhost. Claim splitting is deterministic, retrieval is keyword overlap over small file chunks, and the judge must answer in a strict JSON schema with positional [n] citations into the evidence it was shown, with one format retry when it mangles the schema. After the verdicts come back, a separate validator replays every citation against the actual evidence list: a citation that points at nothing, or at an excerpt the judge never saw, downgrades the claim to UNVERIFIED with a "verdict not trusted" note. 22 unit tests cover the splitter, the citation validator, the retry path, and the exit-code semantics.

Two findings shaped the design. First, format discipline from a small model is fragile enough that the retry and the validator earn their keep immediately. Second, model size matters more than I wanted it to: the 0.5b judge once approved an invented claim, which is why 1.5b is the documented default and why SUPPORTED is only ever "worth trusting more", never "verified fact".

Why Does Open Innovation Matter?

The answers most worth checking quote private things: home-server logs, customer notes, internal runbooks. Shipping them to a hosted API just to verify them defeats the point. Because the judge is an open-weight model running on the same machine, the answer and the evidence never leave the laptop. The whole pipeline works with the internet off, costs nothing per run, and the judge is one flag away from being any model Ollama can run - so when a better small open model lands, ectocheck gets better with a --model flag, not a rewrite. A closed API could do the judging, but it couldn't give me any of that, and it wouldn't let me read every moving part.

The Spirit of It

Día de los Muertos is about keeping the line to the past honest: real memories, told carefully, checked against the people who were there. ectocheck is that same instinct pointed at a machine that talks like it remembers everything. When the séance ends, the only question that matters is the one the tool asks of every single claim: show me where you saw that. If it can't point at the page, it doesn't get believed.