DEV Community

Den Soloviov
Den Soloviov

Posted on Edited on

Why I don't let an AI grade your architecture

 Ask an LLM to review a system design and you’ll get a thoughtful answer. Ask it again and you may get a different one. The model isn’t wrong; it just isn’t built to give the same verdict twice.

That’s fine for a brainstorm. It’s a problem for a grade. A grade only teaches you something if every point has a reason attached, and if you can come back tomorrow and get the same number for the same design.

Rules you can argue with
So I built ArchScore around a public rulebook. Ruleset v1.0.0 has 17 rules across three axes: availability, simplicity and cost. Each axis starts at 100, and each rule that fires adds or removes points. A golden test file pins every number, so a design gets the same score every time.

Take AVAIL-1: Postgres without Multi-AZ costs 18 points per instance. It's my judgement that a single-AZ primary is the most expensive default in cloud. And here's where a reasonable engineer would disagree: a read-only analytics replica that can be rebuilt in an hour doesn't need Multi-AZ. The rule still fires. That's the trade: you lose nuance, you gain a grade you can read, reproduce and dispute.

Where the model still helps
There is an AI in the product: an "AI Architect" that asks the questions a staff engineer would ask in review. But it only talks. It can explain a score the engine already produced; it can never change one.

What's weak
Seventeen rules is a small rulebook. Cost uses a generic $60-per-component reference curve. The weights aren't calibrated against real outages yet. Every rule has a "Disagree" button on the rulebook page, and disputes are how the next version gets better.

The rules I mentioned are public: https://archscore.dev/scoring-constitution

Written with help from AI.

Top comments (0)