We didn't build a code-review agent. Here's why.

Mehmet Adem Şengül — Thu, 27 Aug 2026 09:13:35 +0000

We didn't build a code-review agent. Here's why.

We just shipped a Code Review API for Consensus Room, and the most interesting part of building it wasn't the AI — it was deciding what not to build.

The tempting wrong answer

The obvious pitch is: "point our agent at your repo, it reviews everything, done." That's what a lot of AI code review tools do. We almost built it too.

Then we looked at what it would actually require:

Tool-calling/function-calling wired up per LLM provider (we had zero of this — our stack is pure text completion via Claude, GPT, Gemini, and a dozen OpenRouter models)
A sandboxed execution environment to let models poke around an arbitrary user-submitted repo — a real security surface on a multi-tenant SaaS
Open-ended cost. If an agent decides how many files to read and how many tool calls to make, you can't tell a user "this will cost $0.03" before you start. Our whole product's trust model is a pre-flight cost estimate before anything runs.
Much higher latency — a multi-step tool loop per review vs. a couple of LLM calls.

None of that is impossible. It's just a different, much bigger project than "let multiple models review this code and merge their findings."

What we built instead

A single, boring, fast endpoint: POST /api/v1/code-review. You send one code unit — a file, a function, a diff — and get back a structured report from multiple models plus a merged verdict. That's it. No repo access, no tool calls, no crawling.

The trick is who calls it. If your own coding agent (Claude Code, Cursor, a CI script, whatever) already knows how to walk a repo and decide what's worth reviewing, it can just call our endpoint once per file as part of its own loop. We don't need to reinvent repo traversal — every coding agent already does that. We just need to be a really good "get a second, third, and fourth opinion on this specific piece of code" primitive that's cheap and fast to call repeatedly.

This reframing cut the actual engineering scope by an order of magnitude, and it's honestly a better fit for how people already work with coding agents.

The part that was still a real problem: structured output across providers

We wanted findings back as structured data — severity, category, line number — not just a paragraph of prose. Obvious answer: JSON mode.

Except we support Claude, GPT, Gemini (via an OpenAI-compatible endpoint), and whatever OpenRouter has that week. Checking the actual Spring AI option classes: OpenAiChatOptions has a responseFormat field. AnthropicChatOptions does not. There's no single JSON-mode contract that works uniformly across all of them.

So we didn't fight it. Each reviewer is instructed to emit one strict line format instead:

- [SEVERITY:HIGH] [CATEGORY:security] [LINE:42] SQL injection — user input is concatenated directly into the query.

The server regex-parses that into structured JSON for the API response. If a model ever drifts from the format, the raw text is still returned in full alongside the (possibly incomplete) parsed findings — a parsing miss never silently drops information, it just means one field is less structured than usual.

The moderator step takes every reviewer's raw findings, deduplicates the ones multiple models independently flagged ([CONFIRMED BY:2/3]), and closes with a single VERDICT: APPROVE|COMMENT|REQUEST_CHANGES line.

Making it resumable, because money is involved

Every LLM call costs real money the moment it completes, and it's charged immediately — not batched at the end. That means a review that fails halfway through (a provider hiccup, balance running out mid-flight) must not throw away work you already paid for.

So the run loop is checkpointed: every reviewer's finding is persisted the instant it comes back. If a review fails or pauses, calling it again reloads what's already stored and only retries what's missing — nothing gets recharged, nothing gets lost. This wasn't a nice-to-have we added later; it came directly from asking "what happens when this fails halfway, with a customer's money on the line."

Trying it

curl -s "https://consensusroom.com/api/v1/code-review?wait=90" \
  -H "Authorization: Bearer cr_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "code": "def get_user(username):\n    query = \"SELECT * FROM users WHERE username = \" + username\n    return db.execute(query)",
    "filename": "users.py",
    "panelists": [
      {"key": "claude", "model": "claude-haiku-4-5", "role": "security"},
      {"key": "gpt",    "model": "gpt-4o-mini",       "role": "bugs"}
    ]
  }'

Full reference (fields, limits, error codes, the exact parsing regex) is in documentation.ai.md — written to be pasted straight into an agent's context.

Would genuinely like feedback from anyone who's solved the "structured output across heterogeneous LLM providers" problem differently — curious if there's a cleaner approach we're missing.

DEV Community: Mehmet Adem Şengül

We didn't build a code-review agent. Here's why.

We didn't build a code-review agent. Here's why.

The tempting wrong answer

What we built instead

The part that was still a real problem: structured output across providers

Making it resumable, because money is involved

Trying it