Amin Parva

Posted on Aug 10

Your deterministic engine ran correctly on the wrong input

#ai #python #llm #opensource

Your deterministic engine ran correctly on the wrong input.

Source says 450000. Model emits 45000. JSON Schema still says “number.” The tool allowlist still says “ok.” Your deterministic engine — rules DAG, ledger, calculator, claims/tax pack, anything that must not invent — runs correctly on the wrong world.

That’s the bleed. Deterministic systems don’t “hallucinate.” They fail correctly on bad inputs. Schema and allowlists never asked whether each argument was bound to a verbatim evidence span.

Old way

Most stacks treat tool calling as:

Expose the tool
Validate JSON shape
Call the engine

Useful. Incomplete.

What they usually can’t do:

Bind a field to a verbatim evidence span
Refuse execution when the proposal doesn’t match the document
Hand the engine a signed authorize-before-execute artifact — not a soft 0.87 confidence

Equation:

Deterministic Engine + Unverified Probabilistic Input
= Deterministic Wrong Answer

New way

Put a tool-argument gate between proposal and execution:

Capture evidence — what the document actually said (spans)
Ground — tie each field to a verbatim span
Decide — allow · send to human · or refuse
Sign — Ed25519 ParameterManifest
Enforce — fail-closed hard check (enforce_group3_boundary) before the engine runs

Probabilistic extractors and agents may still propose. They don’t get to authorize.

That’s PrismManifest (Apache-2.0, prismmanifest==0.3.4) — orchestrator-agnostic. LangGraph, CrewAI, custom Python, C++ engines, your existing DAG. ChorusGraph is optional, not required.

Is / is not

Is	Is not
Trust boundary before a deterministic engine runs	An LLM / full OCR product
Evidence → signed manifest → hard check	Prompt-injection firewall (that’s a different product class)
Fail-closed	“0.87 confidence” as permission to execute

Money paths are a sharp case; the same gap shows up anywhere untrusted args enter deterministic compute.

Where the guarantee sits (don’t blur this)

Stage	Today (0.3.4)	Guarantee
Span resolve / digit binding	Python	Field ↔ evidence span
Plausibility (integer micro-units)	Python	Exact money math — no FP drift
Quorum / gate status	Python	allow / human / refuse
Signed FlatBuffer enforce	C++ (+ Python twin)	Artifact valid before engine

C++ protects the signed ParameterManifest at the final boundary. Span binding and plausibility stay in Python today. CUDA is optional acceleration — not the security thesis.

Minimal shape

from prismmanifest import KeyRing, PrismManifestPipeline, enforce_group3_boundary, GateDecision
from prismmanifest.router import DocumentPackage, IntentRouter

keyring = KeyRing.generate(key_id="local-dev-ed25519")
package = DocumentPackage(
    doc_id="1040.txt",
    pages=["Form 1040 Tax Year 2024\nLine 11 AGI: $450,000.00\n"],
    form_type="IRS_FORM_1040",
    tax_year=2024,
)
routed = IntentRouter().run(package)
pipeline = PrismManifestPipeline(keyring)
result = pipeline.run_on_evidence(evidence=routed.evidence, extraction=routed.extraction)
gate = enforce_group3_boundary(
    result.manifest,
    public_keys=keyring,
    expected_dag_id="capital_gains_v3",
)
if gate.decision is GateDecision.ACCEPT:
    # only then may your deterministic engine run
    pass

Install:

pip install "prismmanifest==0.3.4"
# CLI: prismmanifest-gate
# Optional: pip install "prismmanifest[kms-azure]"

Try it on the live site (no upload)

Fixed fixture: Form 1040 · tax year 2024 · Line 11 AGI $450,000. Not your documents.

Digit Drop Lab (10-second punch)

https://www.insightits.com/products/digit-drop-lab.html

Action	AI proposes	Live badge
Drop a digit	`$45,000`	Blocked — engine does not run · wrong by $60,750 vs document math
Correct extract	`$450,000`	Allowed — engine may run

Full interactive demo

https://www.insightits.com/products/prismmanifest-demo.html

Digit-drop → Blocked
Correct extract → Allowed
Needs human → Needs human — engine waits
Custom → enter your own proposed dollar against the same evidence

Pattern tabs (underwriting · insurance claims · accounts payable) show the same money-boundary pattern. Live ACCEPT/REJECT still binds Form 1040 Line 11 AGI $450,000 — those tabs do not run a live DTI / settlement / ERP engine in the browser demo.

Landing: https://www.insightits.com/products/prismmanifest.html

Does the gate actually catch the digit drop?

Pilot OSS vendor harnesses (not a live customer fax SLA):

Planted money errors: critical FA 0/11
FinancePackBench text: 100/100 · FA rate 0
FinancePackBench PDF: 20/20 · FA rate 0
500-pack Py × C++ × CUDA decision + signing-hash parity: 511 checks · 0 mismatches

Production bar remains your corpora through the same gate.

Honesty

Not “beat GPT on OCR accuracy.” Extraction quality ≠ execution authorization.
Not who-may-speak (injection) and not answer-grounding — different layers.
Soft confidence is not a substitute for enforce.
Demo = fixed sample packs · Pilot OSS.

Closing

Deterministic engines will keep doing math correctly. The systems problem is authorizing what they’re allowed to compute on.

Propose freely → sign a ParameterManifest → enforce before deterministic compute.

If your stack still feeds model args straight into an engine, drop a comment with where that crossing happens — or reply MANIFEST and I’ll send an async boundary one-pager (no call).

DEV Community