Why some decisions belong in AI—and others belong in five lines of code.
Part 13 findings of an experiment: building an LLM-powered support agent with deterministic boundaries. The companion repo contains the full code.
"Is this order eligible for a refund?" is four rules: delivered, paid, inside the return window, belongs to the customer.
An LLM can answer that. It would probably answer correctly almost every time. The interesting question isn't whether it can — it's what it costs to ask, multiplied by how often you ask.
The number
Refund eligibility gets checked on every refund request, every status enquiry that mentions a return, and every retry. Say 50,000 checks a day for a mid-sized shop.
$ ./gradlew checkCost
One refund-eligibility check, 50,000 times a day
PATH PER CALL PER DAY PER YEAR
Claude Opus 5 $0.004000 $200.00 $73,000.00
Claude Sonnet 5 $0.001600 $80.00 $29,200.00
Claude Haiku 4.5 $0.000800 $40.00 $14,600.00
Java method $7.09e-13 $3.54e-08 $0.000013
Measured: ~70 ns per deterministic check
The model rows are published per-token prices times an estimated prompt: the policy as a system prompt, the order as JSON, the request, a structured verdict back. Call it 500 tokens in, 60 out.
The Java row is RefundEligibility.evaluate measured in a warmed-up loop and costed as rented CPU time. Seventy nanoseconds at $0.036 per vCPU-hour.
The gap is about a billion to one. Not a percentage — a factor with nine zeros. The deterministic check's entire annual compute bill is roughly one thousandth of a cent.
Where the number comes from
Nothing clever, which is the point:
public static double perCall(TokenPrice price, PromptSize prompt) {
return prompt.inputTokens() / PER_MILLION * price.inputPerMillion()
+ prompt.outputTokens() / PER_MILLION * price.outputPerMillion();
}
The prompt estimate lives in a value called PromptSize, not as a literal inside a formula, precisely so you can disagree with my token count and re-run the comparison with yours. Halve it and the cheapest model still costs $7,300 a year. There is no token estimate that makes this a close call.
flowchart LR
D{"How often does this<br/>decision run?"}
D -->|"once per conversation"| AI["AI: intent, retrieval<br/>$0.004 is a bargain"]
D -->|"per request, per retry,<br/>per rule"| SW["Software: policy, eligibility,<br/>risk tiers, scoping"]
AI --> B["The boundary from ADR 001"]
SW --> B
classDef step fill:#eef2f6,stroke:#8fa3b8,color:#24313f
classDef decision fill:#f7f4ec,stroke:#b3a988,color:#24313f
classDef same fill:#ecf2ed,stroke:#93b39d,color:#3d5344
class AI,SW step
class D decision
class B sameThis is not an anti-model argument
$0.004 for a judgement call that genuinely needs judgement is cheap. Reading "the shoes don't fit, can I send them back?" and turning it into a structured request is worth every cent, and no rules engine I'd want to maintain does it as well.
That call happens once per conversation. Eligibility happens per request, per retry, per rule evaluation. Same price tag, wildly different bill.
So cost doesn't tell you "AI expensive, code cheap". It tells you where the boundary from post 1 pays for itself: the components that ended up in domain are exactly the ones that run at high frequency and have a right answer. That wasn't a cost decision when I drew it — it was a correctness decision. The bill just happens to agree.
The same arithmetic runs anywhere cheap-per-unit meets high-volume. A fraud model scoring every transaction, versus a rules pre-filter that rejects the obvious ones first. A vision system inspecting every part on a line, versus a dimension check that catches most defects for free.
Put the expensive judgement where judgement is needed. Let the cheap deterministic thing handle the rest.
What would change my mind
Three things, in rough order of likelihood:
- Volume collapses. At 500 checks a day the model path costs $2. Nobody restructures a system over $2, and correctness would have to carry the argument alone
- The policy stops being rules. Add goodwill exceptions and "use your discretion for loyal customers" and there's nothing left to express as four booleans. Then the model earns its price
- Prices fall three orders of magnitude. The gap becomes something a budget absorbs — though I'd still want the unit test more than I'd want the API call
That last point is the one I'd defend longest. Even at zero cost, I'd keep eligibility in a method: it's testable, it's inspectable in an audit, and it can't have a bad day. Cost isn't the reason for the boundary. It's just the easiest reason to put on a slide.
What decision in your system runs 50,000 times a day, and do you know what each one costs?
Top comments (1)
The "how often does this decision run" question finally resolved two months of my team arguing about where to put a pricing tier lookup. We had the cost per call but nobody had actually multiplied it by call frequency - once someone charted the page loads per hour, the argument was over. We've ended up with the same implicit split: once-per-intent goes to the model, once-per-list-item doesn't. The billion-to-one ratio actually understates it for anything running inside a loop.