Since April, most of Deckgauge's code has been written by AI coding agents — 4,700 commits total, around 1,450 of them with AI co-author trailers. Every merge is human-reviewed, and the review process we ended up with is not the one we started with. This is what changed.
Deckgauge is a self-hosted engineering intelligence platform — one board over Jira, GitHub, GitLab and Azure DevOps, DORA metrics, per-engineer workload. It's a measurement product, and that matters for this story, because a measurement product has a special relationship with wrongness.
Agents don't write bad code. They write plausible code.
The first assumption I brought to AI-assisted development was that review would be about catching sloppy code. It isn't. The agents never wrote ugly code. They wrote code that looked right and was wrong about the data. That's much harder to review than a syntax error.
A syntax error fails loudly at the first run. A plausible calculation passes the build, passes the type check, sometimes even passes a test — and is wrong about what the number means. In a metrics product that is the worst kind of wrong, because it's confident.
Review stopped being line-by-line
We stopped reading diffs line by line and started reading them assumption by assumption. The middle of an agent diff is usually fine. The edges are where it lies to you: the data contract at the boundary, the aggregation rule nobody stated, the join that quietly changes what a metric covers.
So the human review became three questions, in order: What does this assume about the data? What happens at the boundary — empty repo, zero deploys, missing tracker? And is the number it produces the number we meant? Everything in between — formatting, structure, naming — the agents handle better than I do, and I stopped spending attention there.
The gates came before the humans
The discipline that actually made this work: no agent task reaches a human until it has passed lint, build, typecheck, and the test suite. Not "should have passed" — a script runs them, and the review doesn't start until they're green. That sounds obvious. It took us a while to enforce it strictly, because early on we were reading diffs on top of red builds and learning nothing.
Once the gates were strict, something shifted: review got faster and safer at the same time, because the human was no longer doing work the machine could do. Humans review intent and math. Machines review everything else.
What I'd tell a team starting this
Start with the gates, not the prompts. The quality of agent output is a process problem, not a prompting problem. If you let agent code reach review without a green gate, you will spend your review capacity on things a linter would have caught in four seconds, and you will miss the assumption errors that actually matter — the ones hiding under plausible, well-typed, well-formatted code.
And if your product measures anything for a living: a confident wrong number is the one bug you can't ship. Build the review process around that sentence.
The whole history is public — the repo is https://github.com/Codpal-Limited/deckgauge, and the commit log is the artifact. You can scroll it and see exactly what this workflow produced: the agent commits, the correction passes, the gates. I think it's the most honest case study we have.
I'm Yaniv — I run Codpal, a CTO-as-a-service company. Deckgauge is self-hosted and source-available; there's a live demo at https://demo.deckgauge.com if you want to see what the agents built before you read what they wrote.
Top comments (0)