Updated: Friday, OCT 16 2026
Zero Mental Math: every number is computed by code, rendered by code, and checked by code. The model writes commentary, and its commentary ships only if a deterministic verifier passes it.
🎯 The problem
I built an MCP server that pulls data from my MT5 terminals on a VPS: trades, positions, averages, technical indicators — basically a stack of financial reports I can query in plain English.
Once it was wired up, the model started inventing things. My favorite: it reported a 16th trade on a day that had 15. In financial reporting, "mostly right" is wrong, so I went through the recent research on tool use, RAG and verification and rebuilt the pipeline around one rule: the model never produces a number, and nothing it writes reaches me without being checked by code.
This post is the design, the reference code, what it guarantees, and — just as important — what it doesn't. I wrote this intro; the rest was drafted with AI help, and every number and output shown below came from running the code in this post.
TL;DR
-
Compute every metric in the tool server (Python,
Decimalfor money). Raw rows never reach the model. - Render the facts block with code. The model doesn't copy numbers into a template — it never touches that block.
- Verify the model's commentary with code: every number must exist in the tool payload, every strategy reference must match an exact lookup table.
- Retry once with the verifier's issue list as feedback. If it still fails, ship the facts without commentary.
- The guarantee is narrow and real: what you read was computed from the data the tool queried, and delivered commentary contains no number or strategy the payload doesn't. It does not cover the model calling the wrong tool, querying the wrong period, or drawing the wrong conclusion from correct numbers.
⚠️ Why models get numbers wrong
Exact arithmetic is still unreliable in 2026. Frontier models win math olympiad medals, yet a February 2026 Google study found that every frontier model it tested without tools — including Claude Opus 4.1, GPT-5 and Gemini 2.5 Pro — loses accuracy on plain two-number integer addition as the operands get longer, with most errors traceable to misaligned digits or mishandled carries [1]. The configuration that held up was a model with code execution, and its remaining errors came from skipping the tool or miscopying the tool's answer. That is this whole post in one result: compute in code, then check the copy.
My failure wasn't arithmetic, it was aggregation in context. The 16th trade came from the model counting rows in a long tool response. Models don't use long inputs uniformly — retrieval accuracy depends on where the relevant item sits in the context [2]. Handing a model 400 deal rows and asking "how many positions?" is asking it to aggregate in its head.
Learned arithmetic is fragile, not impossible. The classic result here is that how numbers are tokenized matters enormously: with explicit position tokens a T5 model learned to add and subtract numbers up to 60 digits, but no model learned rules that extrapolated past the lengths it was trained on [3]. Fragile and length-bound is still not something to put between a trade log and a P&L figure.
Temperature 0 doesn't make an endpoint deterministic — and determinism isn't accuracy anyway. Most inference endpoints are nondeterministic mainly because your request gets batched with whatever else is in flight, batch size changes with load, and common kernels aren't batch-invariant [4]. But even a perfectly deterministic model can repeat the same wrong number every time. What you need is a check, not reproducibility.
🏗️ Architecture
user question (plain English)
│
▼
LLM picks tool + typed params ──────────────── not verified (see "The boundary")
│
▼
L1 tool computes metrics (Python, Decimal) deterministic
L2 payload sealed under a snapshot id deterministic
L3 facts block rendered from payload deterministic
L4 entity table (dict / SQL) ───────────┐ deterministic
│ │
▼ ▼
LLM drafts commentary ──► L5 verify(text, payload, entities) deterministic
▲ │ issues
└──── L6 one retry ◄──────┘ → then fail closed: facts only
Trust flows from deterministic systems to stochastic ones, never the reverse. The model can propose; only code can approve.
🔧 Layer 1: Fat tool server (compute everything)
The MCP tool returns metrics, never raw rows that would require the model to count, sum or divide.
from decimal import ROUND_HALF_UP, Decimal
CENT = Decimal("0.01")
def q(x: Decimal) -> Decimal:
return x.quantize(CENT, rounding=ROUND_HALF_UP)
def position_summary(pnls: list[Decimal]) -> dict:
if not pnls:
raise ValueError("no closed positions in range")
wins = [p for p in pnls if p > 0]
losses = [p for p in pnls if p < 0] # break-even trades count toward the total only
gross_win, gross_loss = sum(wins, Decimal(0)), -sum(losses, Decimal(0))
n = len(pnls)
return {
"total_positions": n,
"total_wins": len(wins),
"total_losses": len(losses),
"win_rate_pct": q(Decimal(100 * len(wins)) / n),
"total_pl": q(gross_win - gross_loss),
"avg_win": q(gross_win / len(wins)) if wins else None,
"avg_loss": q(-gross_loss / len(losses)) if losses else None,
"profit_factor": q(gross_win / gross_loss) if gross_loss else None,
"expectancy": q((gross_win - gross_loss) / n),
}
A few details that matter more than they look:
-
Decimal, notfloat, for money. Rounding is explicit and identical everywhere. -
Undefined is
None, not a made-up number. Profit factor with zero losing trades is undefined; the renderer printsn/a. -
Echo the resolved query. The payload carries
{"from": "2025-11-01", "to": "2025-11-30", "timezone": "UTC", ...}so the reader sees exactly what was queried. -
Make the payload a contract. MCP tools declare an
inputSchema, and since the 2025-06-18 spec can declare anoutputSchemaand return typedstructuredContent[5]. Use enums for accounts and symbols and a date format for ranges, so a malformed call fails at the boundary instead of producing a plausible report.
This is program-aided reasoning [6] in production form, and the tool-use framing comes from Toolformer [7] and ReAct [8]: delegate what models do poorly to code.
Testing: unit-test each tool against a fixture whose values you checked by hand. A deterministic function needs one correct run per case, not repeated runs — rerunning it only proves it's deterministic, not that it's right.
📝 Layer 2: Provenance (seal the payload)
Every payload gets a snapshot id, the facts block displays it, and the verifier checks commentary against that exact payload.
import hashlib
import hmac
import json
def snapshot_id(payload: dict, key: bytes) -> str:
canonical = json.dumps(payload, sort_keys=True, separators=(",", ":"), default=str)
return hmac.new(key, canonical.encode(), hashlib.sha256).hexdigest()
def snapshot_ok(payload: dict, key: bytes) -> bool:
body = {k: v for k, v in payload.items() if k != "snapshot"}
return hmac.compare_digest(snapshot_id(body, key), payload["snapshot"])
Use the right tool for the job:
- Same process? If the verifier runs inside the server that produced the payload, the id is just a lookup key — a UUID works.
- Crossing a trust boundary? If the payload passes through a client, a queue or another machine before verification, use HMAC-SHA256 [9] over canonical JSON, with the key loaded from the environment. Changing any metric invalidates the seal.
- Not CRC32. CRC32 detects accidental corruption; it's linear, and collisions are easy to construct deliberately, so it offers no tamper resistance.
- Never ask the model to echo a checksum as proof. A model can append a correct tag to a response with a wrong number in the middle. The tag proves the model saw the payload, not that it used it correctly.
Citations alone don't establish accuracy either: a human audit of commercial generative search engines found only about half of generated sentences were fully supported by their citations, and about a quarter of citations didn't support the sentence they were attached to [10]. Provenance is useful because it tells the verifier what to compare against — the comparison itself is Layer 5.
📄 Layer 3: Deterministic rendering
The facts block is rendered by Python. The model doesn't fill template slots with copied citations; it never edits this block at all.
def money(x: Decimal | None) -> str:
if x is None:
return "n/a"
return f"-${-x:,.2f}" if x < 0 else f"${x:,.2f}"
def render_facts(p: dict, strategies: dict[int, str]) -> str:
s, lots, period = p["summary"], p["lots"], p["query"]
change = lots["largest_change"]
pf = s["profit_factor"] if s["profit_factor"] is not None else "n/a (no losing trades)"
return "\n".join([
f"## Performance — {strategies[p['magic']]} [magic {p['magic']}]",
f"Period: {period['from']} → {period['to']} ({period['timezone']}) · "
f"snapshot `{p['snapshot'][:12]}`",
"",
f"- Positions: {s['total_positions']} ({s['total_wins']} wins, {s['total_losses']} losses)",
f"- Win rate: {s['win_rate_pct']}%",
f"- Total P&L: {money(s['total_pl'])}",
f"- Average win / loss: {money(s['avg_win'])} / {money(s['avg_loss'])}",
f"- Profit factor: {pf}",
f"- Expectancy per position: {money(s['expectancy'])}",
f"- Lots, last {len(lots['recent'])}: {', '.join(map(str, lots['recent']))} "
f"→ current {lots['current']}, mean {lots['mean']}",
f"- Largest lot change: {change['from']} → {change['to']} "
f"({change['pct']:+}%) on {change['date']}",
])
Output for the sample month used throughout this post (29 positions on XAUUSD):
## Performance — Goldfish Scalper [magic 106]
Period: 2025-11-01 → 2025-11-30 (UTC) · snapshot `00a395f9721c`
- Positions: 29 (19 wins, 10 losses)
- Win rate: 65.52%
- Total P&L: $956.30
- Average win / loss: $88.63 / -$72.76
- Profit factor: 2.31
- Expectancy per position: $32.98
- Lots, last 5: 0.05, 0.05, 0.05, 0.08, 0.07 → current 0.07, mean 0.06
- Largest lot change: 0.05 → 0.08 (+60.00%) on 2025-11-24
Structure is not semantics. If you want structured output from the model itself — say, JSON with a commentary field — use constrained decoding. The idea goes back to lexically constrained beam search [11]; today it means JSON-schema-constrained generation, which constrains the shape of the output (and frameworks differ in how much of JSON Schema they actually support) but says nothing about whether its contents are true [12]. Strict format constraints can also measurably degrade a model's reasoning [13]. Constrain format where you need to parse; verify content separately.
🗄️ Layer 4: Exact entity grounding
Not every hallucination is numeric. The model can report the right win rate and attribute it to the wrong strategy.
A magic number is an exact integer key, so it gets an exact lookup:
STRATEGIES = {106: "Goldfish Scalper", 214: "Dark Dione"} # or a SQL table
The renderer always writes strategies as Name [magic N], and the commentary prompt requires the same form, which gives the verifier something unambiguous to check.
Vector search is the wrong tool for exact IDs: embedding similarity is approximate by design and adds retrieval noise and tokens to a lookup that a dict answers exactly. Retrieval-augmented generation [14] earns its place for fuzzy, prose-shaped knowledge — trading rules, playbooks, notes — never for ID-to-name mapping.
✅ Layer 5: Deterministic verifier
This is the gate. It runs on every draft and needs no model.
import re
NUMBER = re.compile(r"(?<![\w.])-?\$?\d[\d,]*(?:\.\d+)?%?")
HEDGE = re.compile(r"(?:~|\b(?:approx(?:imately)?|about|around|roughly|nearly|almost)\s+)\$?\d", re.I)
TAG = re.compile(r"\[magic (\d+)\]")
def leaves(obj):
if isinstance(obj, dict):
obj = list(obj.values())
if isinstance(obj, (list, tuple)):
for v in obj:
yield from leaves(v)
else:
yield obj
def verify(text: str, payload: dict, strategies: dict[int, str]) -> list[str]:
issues = []
# Entities: every [magic N] tag must sit right after N's name, and no other strategy may appear.
for m in TAG.finditer(text):
name = strategies.get(int(m.group(1)))
if name is None:
issues.append(f"unknown magic number {m.group(1)}")
elif not text[:m.start()].rstrip().endswith(name):
issues.append(f"[magic {m.group(1)}] belongs to {name!r}")
for magic, name in strategies.items():
if name in text and magic != payload["magic"]:
issues.append(f"{name!r} is not part of this report")
# Numbers: after removing tags and payload strings (dates, symbols), every number must be in the payload.
prose = TAG.sub("", text)
for s in (v for v in leaves(payload) if isinstance(v, str)):
prose = prose.replace(s, "")
allowed = {abs(Decimal(v)) for v in leaves(payload) if isinstance(v, (int, Decimal))}
for token in NUMBER.findall(prose):
value = abs(Decimal(token.replace("$", "").replace(",", "").rstrip("%")))
if value not in allowed:
issues.append(f"unsupported number: {token}")
if HEDGE.search(prose):
issues.append("approximation language: report exact figures or none")
return issues
A bad draft:
Goldfish Scalper [magic 106] had a strong month: roughly 70% of trades won and P&L reached $1,350.00. Dark Dione [magic 106] sized up 0.05 → 0.08 lots.
- [magic 106] belongs to 'Goldfish Scalper'
- 'Dark Dione' is not part of this report
- unsupported number: 70%
- unsupported number: $1,350.00
- approximation language: report exact figures or none
A good draft passes with zero issues:
Goldfish Scalper [magic 106] won 19 of 29 positions. The average win of $88.63 is larger than the average loss of $72.76, which is why expectancy stays positive at $32.98. Size rose 60.00% on 2025-11-24 and is currently 0.07 lots, above the 0.06 mean.
It fails closed. A false positive withholds commentary; it never alters a number.
Know its limit. It checks that every number exists in the payload, not that it's attached to the right label — "win rate 72.76%" would pass, because 72.76 is the average loss. If you need pairing covered too, use strict mode: the model writes {summary.avg_win} and code writes the digits.
SLOT = re.compile(r"\{([\w.]+)\}")
def fill_slots(commentary: str, payload: dict) -> str:
if NUMBER.search(SLOT.sub("", TAG.sub("", commentary))):
raise ValueError("literal number in strict-mode commentary")
def value(m):
node = payload
for key in m.group(1).split("."):
node = node[key] # unknown slot -> KeyError -> reject the draft
return str(node)
return SLOT.sub(value, commentary)
In strict mode the model is never in a position to type a digit, so there is nothing to miscopy.
Where a model judge still helps
Some claims aren't numeric: "risk management changed this week," "the edge looks like it's decaying." No regex checks those. If you add an LLM judge for them, keep it advisory and keep it away from the numbers:
- Use a different model family from the generator. LLM judges carry known biases, including favoring their own outputs [15]; that self-preference tracks how well a model recognizes its own text [16].
- Don't assume checking is easier than writing. Verification reliability depends on problem difficulty and generator strength, and errors from strong generators are harder for verifiers to catch [17]. LM judges and reward models still fall well short of a perfect verifier; combining several of them narrows that gap [18].
For numeric claims, comparing against the payload is the perfect verifier. Use it.
🔄 Layer 6: Bounded repair, fail closed
def build_report(payload, strategies, draft_commentary, max_attempts=2) -> dict:
facts = render_facts(payload, strategies) # the model never edits this block
issues: list[str] = []
for attempt in range(1, max_attempts + 1):
commentary = draft_commentary(facts, issues) # the only model call
issues = verify(commentary, payload, strategies)
if not issues:
return {"report": f"{facts}\n\n{commentary}",
"verified": True, "attempts": attempt}
return {"report": f"{facts}\n\n_Commentary withheld: failed verification._",
"verified": False, "attempts": max_attempts, "issues": issues}
With the bad draft first and the good draft second: {"verified": true, "attempts": 2}. With a model that keeps failing, you still get every number — the report just ends in _Commentary withheld: failed verification._
Why it's shaped this way:
- The feedback is external ground truth. The retry prompt carries the verifier's exact issue list. Without external feedback, models struggle to correct their own reasoning and sometimes get worse [19]. Iterative refinement can improve outputs [20], but nothing guarantees convergence, so the loop is capped.
- Reject, don't patch. Swapping the right number into a sentence written around a wrong one can produce a fluent, wrong conclusion ("best month ever, 65.52% win rate"). Regenerate instead, and if a fix is truly mechanical, it belongs in strict mode, where code inserts the values from the start.
- The failure mode is "less commentary," never "wrong numbers."
🚨 Trends and dispersion: computed, not narrated
A single average hides the shape of the data — the same summary statistics can describe completely different datasets [21]. "Average 0.06 lots" hides that position size jumped recently, which is exactly the risk signal I care about.
So the server computes the trend, and the renderer prints the recent raw values:
def lot_trend(fills: list[tuple[str, Decimal]]) -> dict:
if len(fills) < 2:
raise ValueError("need at least two fills to describe a trend")
lots = [lot for _, lot in fills]
steps = [(fills[i][0], lots[i - 1], lots[i]) for i in range(1, len(lots))]
date, before, after = max(steps, key=lambda s: abs(s[2] - s[1]))
return {
"recent": lots,
"current": lots[-1],
"mean": q(sum(lots, Decimal(0)) / len(lots)),
"largest_change": {"date": date, "from": before, "to": after,
"pct": q(100 * (after - before) / before)},
}
- Lots, last 5: 0.05, 0.05, 0.05, 0.08, 0.07 → current 0.07, mean 0.06
- Largest lot change: 0.05 → 0.08 (+60.00%) on 2025-11-24
The model never transcribes the list and never calculates the change. Keep the raw window short (5–10 values); long lists belong in a table or a chart, rendered by code.
🧱 The boundary: what this guarantees and what it doesn't
Guaranteed, as long as the report pipeline is the only path to the reader:
- Every number in the facts block was computed by the server from the data the tool queried.
- Every number in delivered commentary exists in that payload, and every strategy reference matches the lookup table. In strict mode, every number is also attached to the metric the model named.
Not guaranteed:
- The right tool and parameters. The model can query October instead of November, or the wrong account or symbol. Surveys of agent hallucinations treat these as their own classes — tool-selection and tool-calling hallucinations [22]. Typed input schemas narrow the space, and echoing the resolved period in the facts block makes the error visible to the reader.
- Number–label pairing in free prose, unless you use strict mode.
- Interpretation. Correct numbers can support a wrong conclusion. That's what the advisory judge, or you, are for.
-
Text outside the gate. In an interactive agent, the agent's own free-form replies don't pass through
verify(). Treat the server-rendered report as the record and the chat around it as unverified. - Upstream data. If the terminal's history is incomplete, the server faithfully computes the wrong thing. Coverage checks — deals without matching positions, gaps in the range — belong in the tool.
🤔 So why use an LLM at all?
The facts block doesn't need one. The model earns its place in two jobs: turning plain-English questions into typed tool calls — I run everything through Claude Code, and it works like an on-the-fly query language for my trading data — and pointing out patterns worth a second look. Both are outside the guarantee above, which is exactly why the numbers live somewhere else.
📏 Measuring it
The number that matters isn't whether the tools are right — unit tests cover that. It's how often a delivered response contains an unsupported number or entity, and verify() measures that for free.
-
Baseline: hand the model raw rows and ask the same questions. Run every model draft through
verify()and count drafts with any issue. -
Pipeline: same questions, same model, through
build_report. Count delivered responses with issues (should be zero by construction) and, separately, how often commentary was withheld — that's the cost of failing closed. - Report counts with an interval, not "100%." Zero failures in n runs bounds the true failure rate at roughly 3/n (95% confidence). Ten clean runs are consistent with a true failure rate as high as 31% (two-sided 95% Clopper–Pearson interval [23]). Showing a failure rate below 1% at 95% confidence with zero observed failures takes 299 runs; below 0.1% takes 2,995.
📊 Summary
| Layer | Technique | What it prevents | Deterministic? |
|---|---|---|---|
| 1 | Server-side metrics, Decimal, typed schemas |
Arithmetic and counting errors | Yes |
| 2 | Snapshot id (HMAC across trust boundaries) | Checking against the wrong or altered payload | Yes |
| 3 | Code-rendered facts block | Paraphrased, rounded or reformatted numbers | Yes |
| 4 | Exact entity table + Name [magic N] tags |
Wrong strategy attribution | Yes |
| 5 |
verify(): numbers and entities vs payload |
Unsupported numbers and entities in commentary | Yes |
| 5b | Optional cross-family judge | Unsupported interpretations (advisory only) | No |
| 6 | One retry with the issue list, then fail closed | Shipping unverified commentary | Yes |
Python computes → Python renders → model comments → Python checks → you read
At no point does a model decide a number, and at no point does a model approve one.
📚 References
- Bie, A., Dick, T., Kulesza, A., Raghavan, P., Raman, V., Vassilvitskii, S. (2026). "AI-rithmetic." arXiv:2602.10416
- Liu, N. F., et al. (2023). "Lost in the Middle: How Language Models Use Long Contexts." TACL. arXiv:2307.03172
- Nogueira, R., Jiang, Z., Lin, J. (2021). "Investigating the Limitations of Transformers with Simple Arithmetic Tasks." arXiv:2102.13019
- He, H., and Thinking Machines Lab (2025). "Defeating Nondeterminism in LLM Inference." thinkingmachines.ai
- Model Context Protocol Specification (2025-06-18). "Tools" —
inputSchema,outputSchema,structuredContent. modelcontextprotocol.io - Gao, L., et al. (2022). "PAL: Program-aided Language Models." arXiv:2211.10435
- Schick, T., et al. (2023). "Toolformer: Language Models Can Teach Themselves to Use Tools." arXiv:2302.04761
- Yao, S., et al. (2022). "ReAct: Synergizing Reasoning and Acting in Language Models." ICLR 2023. arXiv:2210.03629
- Krawczyk, H., Bellare, M., Canetti, R. (1997). "HMAC: Keyed-Hashing for Message Authentication." RFC 2104
- Liu, N. F., Zhang, T., Liang, P. (2023). "Evaluating Verifiability in Generative Search Engines." Findings of EMNLP 2023. arXiv:2304.09848
- Hokamp, C., Liu, Q. (2017). "Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search." ACL 2017. arXiv:1704.07138
- Geng, S., et al. (2025). "JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models." arXiv:2501.10868
- Tam, Z. R., et al. (2024). "Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models." arXiv:2408.02442
- Lewis, P., et al. (2020). "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." NeurIPS 2020. arXiv:2005.11401
- Zheng, L., et al. (2023). "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." NeurIPS 2023 Datasets and Benchmarks. arXiv:2306.05685
- Panickssery, A., Bowman, S. R., Feng, S. (2024). "LLM Evaluators Recognize and Favor Their Own Generations." NeurIPS 2024. arXiv:2404.13076
- Zhou, Y., et al. (2025). "Variation in Verification: Understanding Verification Dynamics in Large Language Models." arXiv:2509.17995
- Saad-Falcon, J., et al. (2025). "Shrinking the Generation-Verification Gap with Weak Verifiers" (Weaver). arXiv:2506.18203
- Huang, J., et al. (2024). "Large Language Models Cannot Self-Correct Reasoning Yet." ICLR 2024. arXiv:2310.01798
- Madaan, A., et al. (2023). "Self-Refine: Iterative Refinement with Self-Feedback." arXiv:2303.17651
- Anscombe, F. J. (1973). "Graphs in Statistical Analysis." The American Statistician, 27(1), 17–21.
- Lin, X., et al. (2025). "LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directions." arXiv:2509.18970
- Clopper, C. J., Pearson, E. S. (1934). "The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial." Biometrika, 26(4), 404–413.
Top comments (3)
This is the move. LLMs should be glorified UI for deterministic backends, not freelance accountants. Let Python do math, let the model copy receipts. Anything else is just vibes pretending to be accuracy.
This is really good. I appreciate the human written part in the beginning (it made me willing to read the AI generated text that followed). I like how this adds actual engineering and goes beyond giving some tools to an agent and hoping for the best.
What I don't understand from the example is why a LLM is needed here at all... it looks like this is calculated by code and have everything the app needs. Everything after seems like statically formatting this info (using templates). What am I missing?
good point, there really is barely any need for the LLM I just like manage everything i do through claude-code-cli
The workflow is fast, and i can actually quickly view the data i need in different ways
I like that it has emergent insights on what I am look at as well , has picked up on patterns in the past that were useful and I used.
LIke an on the fly Graphql where the query language is plan english