The fence
G.K. Chesterton’s rule: if you find a fence across a road and can’t see why it’s there, don’t tear it down until you know. Code is full of fences, like a redundant-looking if, an odd retry, or a guard nobody remembers. A pull request deletes one, the tests pass, and the bug it was built for comes back.
Here is a PR that “simplifies” a charge function by deleting five lines:
Nothing on that page says those lines exist because customers were charged twice. So git-seance says it:
It traced the removed lines to the commit that added them, followed fixes #1 to the issue, and posted one comment with links and a verified quote. The next PR, which deletes an unused variable, got no comment at all:
What it is
A CLI (with an example GitHub Action) that runs on your own machine with a local open-weight model, Gemma through Ollama. No paid API, and your code never leaves the machine.
Git finds, the model writes
My first rule: the model never decides what the origin commit is. Git does.
Parse the diff and keep removed or rewritten hunks.
Trace them with git blame -w on the merge-base, falling back to git log -S and -G.
Collect the origin commit message, the merged PR and linked issues.
Score the evidence with plain rules: fix, revert, hotfix, workaround, linked bug. No signal means no comment.
Only then does Gemma, at temperature 0 with a JSON schema, write up the evidence.
A verifier drops every claim whose quote isn’t found verbatim in the cited evidence. It doesn’t reword; it drops.
One comment is created, or updated in place, on the PR.
The trust design
A bot that cries wolf gets muted, so every warning has to point at something a human can open.
Commit, PR and issue text is untrusted. The model has no tools, and the bot’s only action is one PR comment.
Cite or drop. Claims must cite evidence ids and quote text that appears verbatim in them.
Model text is stripped of links, @mentions, HTML and #123 references, and wrapped in fixed, hedged wording. The #123 rule came from my own demo, where GitHub auto-linked a #1 inside a quote.
Advisory only. It never blocks a merge.
What broke
My GPU crashed. On a 6 GB RTX 4050, gemma4:e4b died with CUDA error: shared object initialization failed. I first blamed my curl quoting, which was wrong. The likely cause is VRAM, though I didn’t prove it. One setting, SEANCE_NUM_GPU=0, forces CPU and works at about 20 tokens a second.
git log -L lies. It walks back to the file’s first commit, not the commit that added the lines. A confidently wrong origin is worse than silence, so I dropped it from the chain.
A one-indent bug hid the issue link. In my first live run the linked issue was missing. I blamed the token, the regex and a stale cache before finding a loop sitting inside if ctx.pr:, so issues only appeared when a PR existed. Every test supplied a PR. The new test has an issue and no PR.
The numbers
I replayed 10 scripted cases in temporary repos: a plain fix, a fix hidden behind a reformat commit, a renamed file, a revert, a workaround, a reason that appears only in the commit body, three harmless deletions, and a commit message carrying a planted prompt injection. Gemma 4 e4b, CPU only.
Metric Result
Right origin commit cited 7/7
Silent on harmless deletions 3/3
Planted injection leaked into the comment 0/1
Unsupported claims posted 0
Claims proposed / dropped by the verifier 7 / 0
Slowest case 30 s
Read these honestly:
The cases are my own, so 7/7 shows the hazards I thought of are handled, not that real-world history is.
The verifier dropped nothing in this run, so the live model never exercised it. Its unit tests do.
“Zero unsupported” is true by design, because the verifier enforces it.
The injection result is a single case.
Limits
Same-repo PRs only. Fork PRs aren’t supported.
The GitHub Action in the README is untested. I demoed the CLI posting from my laptop.
A non-whitespace reformat commit that isn’t in .git-blame-ignore-revs can still win blame.
The verifier checks grounding, not intent. A faithfully quoted injected sentence would pass it, and the real protection is no tools, one action, sanitised output and fixed wording.
Squash merges collapse history, so the PR text becomes the main source.
GitHub responses are cached on disk with no expiry.
Why open mattered
Private code stays on your machine.
There is no per-review fee.
The model is one setting (SEANCE_MODEL). When the GPU path broke, changing a setting fixed it. A closed API would have given me a 500 and no recourse.
Links
Code: https://github.com/kashyapanand21/git-seance
Project page: https://kashyapanand21.github.io/git-seance/
Demo repo (open PRs): https://github.com/kashyapanand21/seance-demo
Replay report: https://github.com/kashyapanand21/git-seance/blob/main/eval/report.md
Top comments (0)