The common assumption is that an agent loop makes an existing decision smarter. For regulated eligibility checks on structured data, it mostly makes it more expensive, slower and harder to defend.
Take a lender's debt-to-income check. The core is one line:
eligible = debt / income <= 0.43
(43% is an example, not a regulatory figure.) The application is already typed: income, debts, loan amount from a form. Sometimes there's a handwritten hardship letter attached. The proposal on the table is to retire the rules engine and let an agent decide.
I'd say no, and the reason isn't the token bill first. With illustrative numbers (250,000 decisions a month, 18,000 tokens per decision, $5 per million tokens) the agent costs $0.09 a decision and $22,500 a month to compute a ratio the rule already gets right. Latency goes from milliseconds to seconds because the calls run in sequence. But cost is the criterion I weigh last.
What I weigh first is what an inconsistent answer costs. In lending, you have to show why someone was declined and that the same facts would produce the same decision again. A model predicts likely tokens. Most of the time it gets the arithmetic right. Give it income as "$115,000 / yr" and an ambiguous auto lease, and it can misread units, drop a digit or talk itself into an exemption that doesn't exist. The rule can't.
The adaptability pitch is where I push back hardest. Change an if/else and your tests and CI tell you exactly what moved. Change an English prompt and you can fix rule A, silently break rule B, and not know which applicants were affected.
So the design I'd pick keeps the rule at the core and gives the model one narrow job at the edge:
- Before the model: input validation, PII scrubbing, and the rule checks that don't need the attachment.
- The model: one single-turn extraction of income, debts and a hardship reason from the letter.
- After the model: schema validation (a Pydantic model works), business-rule bounds and sanity checks. Anything that fails goes to a human queue. Only validated numbers reach the same DTI rule as everyone else.
Token spend then follows the paperwork that needs reading, not the number of applications.
It's not free. A tested rules service takes weeks where a prompt takes an afternoon, someone has to staff the review queue, and deterministic code is consistently wrong if it encodes the wrong definition of income. Determinism makes errors reproducible. People still have to check the rule matches the policy.
If you're looking at your own systems:
- If the logic can be written as rules and the input is structured, code decides. A model only reads the unstructured part.
- Make one standing rule in design review: no model output writes to a database or calls an external API without passing deterministic validation.
- If an agent already decides, run the rule in shadow mode on live traffic and understand every disagreement before you cut over.
The full article has the four options compared, the decision flow, the ownership split, the five-step migration and the sources: https://pub.towardsai.net/let-the-llm-read-the-paperwork-not-make-the-decision-e52b71e22032
Top comments (1)
pretty much how we think about searchspot too, the model reads the messy stuff like reviews and fine print and anything that spends money waits for the traveler to confirm. what happens when the hardship letter contradicts the numbers in the structured data, does that just go to a human