Your LLM Isn't Bad At Math. It Was Never Doing Math In The First Place.
In a tutorial, an LLM call looks like a function: pass in text, get back an answer, move on. It's easy to start treating the model like it's evaluating your business logic the same way a function would, deterministically, the same input always producing the same output.
In production, that assumption breaks in a specific, predictable way, and it's worth naming precisely why.
Traditional software runs on deterministic logic: if input A meets condition B, output C happens, every time, with a proof you could write on a whiteboard. An LLM runs on statistical probability: it predicts the most likely next token given the patterns in its training data. Those are two different kinds of "correct." The friction shows up exactly where a team asks the second kind to do the first kind's job.
Below are the places that friction actually shows up in a live system. These are illustrative scenarios based on the shape this failure takes, not one specific incident.
Asking a model to do arithmetic is asking it to guess what arithmetic usually looks like
Say a pipeline extracts line items from a batch of invoices, then asks the same model call to also compute the subtotal, apply tax, and return a total. On short, simple invoices, it's right almost every time, because short simple math is heavily represented in training data and easy to pattern-match. On a 40-line invoice with mixed tax rates and a rounding rule, it starts being right most of the time, which is a different and much worse thing than right. Nothing throws an error. The number is just plausible instead of correct, and "plausible instead of correct" is invisible until someone reconciles the books.
A statistically-applied business rule is not the same rule as a logically-applied one
Say the business rule is "refunds are allowed within 30 days of purchase." Put that rule in a prompt and ask the model to decide eligibility case by case, and it will get the easy cases right: a purchase from six months ago, obviously no; a purchase from yesterday, obviously yes. Where it gets interesting is the boundary: day 29, day 30, day 31, across time zones, with a purchase timestamp in one format and today's date passed in another. A deterministic date comparison gets this right every single time by construction. A model is producing its best guess at what "a refund decision near the boundary" looks like, based on how those decisions were phrased in its training data, and that's a meaningfully different operation even when it happens to output the correct answer nine times out of ten.
The output still looks like a normal response, which is exactly the problem
This is what makes the failure mode dangerous rather than just annoying: a wrong statistical answer doesn't look different from a right one. It's the same JSON shape, the same confident tone, the same absence of an exception. There's no signal in the response itself that tells you this was a guess rather than a computation. You find out later, from a customer service escalation or a finance reconciliation, not from anything in your logs.
Where the line actually goes
None of this means don't use the model near your business logic. It means be precise about which half of the job you're handing it.
What the model is good at: turning unstructured input, an email, a contract clause, a support ticket, a scanned form, into structured data. That's genuinely ambiguous work: language is fuzzy, humans phrase the same request ten different ways, and a statistical model that's seen millions of phrasings is the right tool for mapping "hey can I send this back, it's been like a month" into structured fields like intent and stated days since purchase.
What the model should never be the last word on: the validation, the calculation, and the rule enforcement that runs on that structured data once you have it. That's deterministic code's job, on purpose, with a strict schema at the boundary so a malformed or out-of-range extraction fails loudly instead of quietly flowing downstream as a confident-looking guess.
We lean on this split in how Cyclopt's automated checks work: the model interprets unstructured signals in a codebase or a pull request, but the actual rule enforcement, the pass or fail line, runs through deterministic logic against a defined schema, not through the model re-deciding the rule each time. The interpretation layer changes. The rule layer doesn't get to.
The reframe
Statistical vs. logical correctness isn't a model-quality problem that gets solved by a better model. It's an architecture decision, and better models make it easier to ignore, not less necessary to make. Let the model handle the ambiguity. Let your code handle the rules.
Where do you actually draw that line in a system you've shipped? What's the one thing you learned the hard way should never have been the model's call to make?
Top comments (4)
The refund boundary is the cleanest example of this because it's so ordinary. Day 30 in one timezone and day 31 in another isn't a model quality problem at all — it's a rule that was never specified precisely enough to be computed. Asking a model to apply it doesn't degrade the model, it just reveals that the business rule was written as prose.
Where I'd push one step further: the extraction half (ticket text into
intentandstated_days) really is the model's job, but the moment I've tried that in anger the failure shifted to the schema — a customer writing "it's been like a month" maps to 30, 31 or 28 depending on the day they sent it, and the model has no way to know which one your rule means. So we ended up making the model emit a range plus the raw phrase, and letting the deterministic layer decide what to do with an ambiguous span. Did you land on a single scalar for that field, or does your structured output carry the uncertainty forward?Totally agree that a fuzzy phrase like "about a month" breaks a rigid scalar field, which is why we ended up having the model pass through the raw quote alongside the parsed value so downstream code can handle the ambiguity instead of the model guessing.
The best boundary I have found is exactly this: let the model extract the operands and intent, then let code compute and enforce. The structured handoff should keep the source span and confidence for every extracted field, though. A perfect date comparison still gives the wrong refund decision if the model silently extracted the wrong purchase date.
That is a crucial point because even a bulletproof deterministic rule will still fail if the model silently hallucinated the purchase date during extraction, making source spans and confidence scores essential parts of that schema handoff.