Quick update on Synod, the multi-agent code reviewer I've been building for
the Qwen Cloud hackathon.
What changed
I added a Semgrep pre-filter in front of the security agent. Before, every
finding came purely from the LLM reading the code and reasoning about it —
which works, but LLMs are stochastic. Same file, different run, sometimes a
different result.
Now Semgrep scans first with deterministic rules, and the security agent
validates and enriches those candidates instead of starting from zero every
time.
Did it actually help?
I was skeptical of my own change, so I benchmarked it properly instead of
assuming. Ran a single-agent baseline against the full council, with and
without the pre-filter, same vulnerable file, checked against known ground
truth:
MethodPrecisionRecallF1Single agent75%75%, but ranged 0–75% across runs75%Council, LLM-only75%same variance issue75%Council + Semgrep100%100%, every run100%
The interesting part wasn't the top-line numbers — it was that the
single-agent and LLM-only council both had real run-to-run variance.
Sometimes it caught everything, sometimes it missed half. That's not a
reviewer you can trust in CI.
Adding the deterministic scanner as a floor fixed that. It's not smarter,
it's just consistent — and consistency turned out to matter more than I
expected.
Also shipped
A GitHub webhook — open a PR, Synod reviews the diff and comments
directly, findings grouped by severity.
A small CLI, closer to how tools like Claude Code feel in the terminal,
for reviewing files or whole directories without touching the API
directly.
Repo's still open source: github.com/02NIN20/Synod
Built for the Global AI Hackathon Series with Qwen Cloud — Track 3: Agent
Society.
Top comments (3)
Ahh jetzt....sorry, wir passen perfekt zusammen!!!! ich habe einen security sys-prompt nach BSI vorgaben gehärtet mit 94% schild bei 32 attacken und 0,2-0,01% hallozinationswahrscheinlichkeit. Das sollten wir zusammen packen. Grüße
Good question — only the pre-filter (Semgrep) is deterministic, it's one static-rule scanner feeding candidates into Sentinel. The LLM still reasons over those candidates, decides what's real, explains impact, and proposes fixes. It's rules for pattern-matching, AI for judgment — not one replacing the other.
Hello friend
I am very interested