Synod Update: Adding a Deterministic Safety Net (and Proving It Helps)

#alibabachallenge #ai #hackathon #github

Quick update on Synod, the multi-agent code reviewer I've been building for
the Qwen Cloud hackathon.

What changed

I added a Semgrep pre-filter in front of the security agent. Before, every
finding came purely from the LLM reading the code and reasoning about it —
which works, but LLMs are stochastic. Same file, different run, sometimes a
different result.

Now Semgrep scans first with deterministic rules, and the security agent
validates and enriches those candidates instead of starting from zero every
time.

Did it actually help?

I was skeptical of my own change, so I benchmarked it properly instead of
assuming. Ran a single-agent baseline against the full council, with and
without the pre-filter, same vulnerable file, checked against known ground
truth:

MethodPrecisionRecallF1Single agent75%75%, but ranged 0–75% across runs75%Council, LLM-only75%same variance issue75%Council + Semgrep100%100%, every run100%

The interesting part wasn't the top-line numbers — it was that the
single-agent and LLM-only council both had real run-to-run variance.
Sometimes it caught everything, sometimes it missed half. That's not a
reviewer you can trust in CI.

Adding the deterministic scanner as a floor fixed that. It's not smarter,
it's just consistent — and consistency turned out to matter more than I
expected.

Also shipped

A GitHub webhook — open a PR, Synod reviews the diff and comments
directly, findings grouped by severity.
A small CLI, closer to how tools like Claude Code feel in the terminal,
for reviewing files or whole directories without touching the API
directly.

Repo's still open source: github.com/02NIN20/Synod

Built for the Global AI Hackathon Series with Qwen Cloud — Track 3: Agent
Society.