Ask an LLM to review a system design and you’ll get a thoughtful answer. Ask it again and you may get a different one. The model isn’t wrong; it just isn’t built to give the same verdict twice.
That’s fine for a brainstorm. It’s a problem for a grade. A grade only teaches you something if every point has a reason attached, and if you can come back tomorrow and get the same number for the same design.
Rules you can argue with
So I built ArchScore around a public rulebook. Ruleset v1.0.0 has 17 rules across three axes: availability, simplicity and cost. Each axis starts at 100, and each rule that fires adds or removes points. A golden test file pins every number, so a design gets the same score every time.
Take AVAIL-1: Postgres without Multi-AZ costs 18 points per instance. It's my judgement that a single-AZ primary is the most expensive default in cloud. And here's where a reasonable engineer would disagree: a read-only analytics replica that can be rebuilt in an hour doesn't need Multi-AZ. The rule still fires. That's the trade: you lose nuance, you gain a grade you can read, reproduce and dispute.
Where the model still helps
There is an AI in the product: an "AI Architect" that asks the questions a staff engineer would ask in review. But it only talks. It can explain a score the engine already produced; it can never change one.
What's weak
Seventeen rules is a small rulebook. Cost uses a generic $60-per-component reference curve. The weights aren't calibrated against real outages yet. Every rule has a "Disagree" button on the rulebook page, and disputes are how the next version gets better.
The rules I mentioned are public: https://archscore.dev/scoring-constitution
Written with help from AI.



Top comments (0)