A 12-person humanitarian NGO carries exactly the same strict-liability sanctions exposure as JPMorgan, and cannot hire anyone to manage it. I built an agent that does — and the first real run against Gemini quarantined 438 of 536 counterparties.
Not one of them was a bad verdict.
(I created this piece for the purposes of entering the All Things Agentic Hackathon.)
Repo: https://github.com/edycutjong/interdict · 3-minute demo: https://youtu.be/C1VFGSwS7w4
What quarantine costs
Interdict re-screens an NGO's whole payment book whenever Treasury updates the OFAC sanctions list. A true hit gets held — money stops. A lookalike gets cleared with a written reason. When the model's answer can't be trusted, the counterparty goes to quarantine, which is a terminal state: a human compliance officer is told the system could not safely decide, and the money stays frozen until they rule.
That's expensive by design. Quarantine is supposed to be rare and it's supposed to mean something.
So when 438 of 536 landed there, my first assumption was that the adjudicator had gone haywire. It hadn't. Free-tier Gemini allows five requests a minute. Every call after the first twenty-one came back 429 RESOURCE_EXHAUSTED, and my code did this:
try:
verdict = adjudicator.adjudicate(context)
except Exception as exc:
quarantine(match_id, "PARSE_ERROR", {"error": str(exc)})
A rate limit is not a parse error. But except Exception doesn't know that, so 438 transient network conditions were filed as suspected model-integrity failures.
Why that's worse than the rate limit
The rate limit costs thirty seconds of waiting. The bug costs the escalation queue.
Quarantine only works if a human reads it. Put 438 entries in there that needed nothing but patience, and the one entry that genuinely needs a person — a near-identical name where the model's rationale doesn't hold up — is buried underneath them. The queue stops being a signal and becomes noise, and the operator learns to skim it. That's the actual failure, and it would have survived into production looking like a working system.
The insight that fixed it is boring and, I think, general:
"The model was wrong" and "the model did not answer" are different failures. One is fixed by a human reading the evidence. The other is fixed by waiting.
Conflating them means you cannot triage. So I stopped conflating them.
Two failure classes, not one
First, the adjudicator owns its own backoff, and only retries things that are actually transient:
_TRANSIENT_MARKERS = ("RESOURCE_EXHAUSTED", "429", "503",
"UNAVAILABLE", "DEADLINE_EXCEEDED")
def _is_transient(exc: Exception) -> bool:
return any(m in str(exc) for m in _TRANSIENT_MARKERS)
def _retry_delay(exc: Exception, attempt: int) -> float:
"""Seconds to wait. Prefers the server's own hint over our guess."""
m = re.search(r"retry in (\d+(?:\.\d+)?)s", str(exc))
if m:
return min(float(m.group(1)) + 1.0, 120.0)
return min(DEFAULT_BACKOFF_S * (2 ** (attempt - 1)), 120.0)
That retry in Ns hint matters more than the exponential fallback. The server knows when it will serve you again; guessing is strictly worse than reading. Five attempts, honouring the hint, and only then does it give up — as a distinct exception type:
except Exception as exc:
if not _is_transient(exc):
raise # a bad answer is not a slow answer
if attempt == MAX_TRANSIENT_RETRIES:
raise TransientAdjudicationError(
f"model unreachable after {attempt} attempts: {exc}") from exc
time.sleep(_retry_delay(exc, attempt))
Then the orchestrator — the only component allowed to write a decision — routes on that type:
try:
verdict = adjudicator.adjudicate(context, feedback=feedback)
except TransientAdjudicationError as exc:
# Still quarantine -- money must never move on a decision that was never
# made -- but say so accurately.
_quarantine(conn, match_id, "ADJUDICATOR_UNAVAILABLE", {
"counterparty_id": counterparty_id, "error": str(exc)[:500],
"attempt": attempt, "retryable": True,
})
except Exception as exc:
# A model failure must never become a silent CLEAR.
_quarantine(conn, match_id, "PARSE_ERROR", {
"counterparty_id": counterparty_id, "error": str(exc)[:500],
"attempt": attempt, "retryable": False,
})
Note what did not change: both paths still quarantine, and money still stops in both. The safety property is identical. What changed is that the row now carries retryable: true or retryable: false, so an operator can tell at a glance which pile is which — and the retryable pile drains itself on the next pass without anyone touching it.
The distinction is worth more than the retry. If I'd only added backoff, the 438 would have shrunk but the category error would still be there, waiting for the next outage.
What I'd take to the next agent system
Three things, in order of how much they cost me:
A bare except around a model call is a category error, not a style problem. Model calls fail in at least two ways that demand opposite responses. Any handler that can't distinguish them will eventually make the wrong one, and it will do so quietly.
Let the failure type carry the triage. retryable: true|false in the payload is what makes the queue readable. The alternative is an operator reading 438 stack traces to work out which ones matter.
Escalation is a budget. Every entry you send to a human spends attention you'll need later. I now treat "should this really escalate?" as a design question with a cost attached, the same way I'd treat a database write.
What this doesn't do
It's a hackathon build, and it's specific about what it isn't:
- Nothing runs on Google Cloud compute. Cloud Firestore holds the audit trail; the agents, Postgres and the independent oracle run on a laptop. The free tier doesn't extend to Cloud Run and I had no billing account.
- The model has never issued a CLEAR in the graded book. A contradicting date of birth cuts a lookalike below the adjudication threshold before the model is ever consulted, so the adjudicator is exercised on confirmation, not on discrimination. The grade should be read with that in mind.
- Decision quality is a 101-row stratified sample, not the full 536-row book — free-tier quota, again.
- The payment book is synthetic and labelled everywhere it appears. The OFAC data is real: the 08/07/2026 publication, 19,199 records, archived by content hash.
The screening numbers, for what they're worth: top-1 0.995 against an independent oracle's 0.840, measured on 400 names deliberately perturbed so none appear on the list verbatim. Screening the seeded book verbatim scores 1.000, which is a string-equality test wearing a costume, so I don't report it.
Everything above reproduces with make reproduce. If the escalation-budget idea is useful to you, that's the part I'd steal.
I created this piece of content for the purposes of entering the All Things Agentic Hackathon.
Repo: https://github.com/edycutjong/interdict · Demo: https://youtu.be/C1VFGSwS7w4
Top comments (2)
This is a really strong example of why reliability engineering around AI agents matters just as much as model accuracy.
The distinction between “the model was wrong” and “the model did not answer” is especially important. Treating both as the same failure can create operational problems even when the underlying safety policy is correct.
I also like the idea of making the failure type part of the decision payload. retryable: true/false turns an otherwise noisy escalation queue into something an operator can actually prioritize. The principle of treating human escalation as an attention budget is applicable well beyond compliance systems.
Two questions I’d be interested in:
Would you consider adding a circuit breaker or adaptive rate limiter so the system can proactively reduce requests before hitting provider quotas?
How would you approach preserving the same failure classification and audit guarantees when the system has multiple model providers or fallback models?
The separation between safety decisions and infrastructure failures is a pattern I think many production agent systems could benefit from.
Can we talk about our programming continousely?