DEV Community: AgentBasics

LLM JSON output truncated? A deterministic repair ladder

AgentBasics — Thu, 12 Nov 2026 16:09:36 +0000

You prompted for JSON. The model returned something that is 95% JSON. json.loads throws. Now what?

The two common reactions are both wrong. Reacting with "the API errored" wastes a perfectly good response. Reacting with a pile of regex hacks silently corrupts data — a greedy re.sub that strips a comma inside a string, a naive brace-matcher that trips over a quote in a value. The fix is a repair ladder: a deterministic sequence of progressively more aggressive fixes, stopping at the first that parses, and — this is the part people skip — reporting which rung saved you.

What actually breaks LLM JSON

Markdown fences: triple-backtick json blocks wrapped around valid JSON. Harmless, constant.
Prose around the object: "Sure! Here's the JSON:" then the object, then "Let me know!" Your parser needs the first balanced {...}, string-aware.
Trailing commas: {"a": 1,} — JSON forbids them, JavaScript habits produce them.
Python literals: True/None instead of true/null, single-quoted keys.
Truncation mid-string: max_tokens fires in the middle of a value. The output is not recoverable in full — and pretending otherwise is how agents invent the missing half.

The ladder

def repair_json(text):
    rungs = [
        ("clean",      lambda s: s),
        ("fence",      strip_code_fences),        # strip markdown fences
        ("extract",    first_balanced_span),      # string-aware brace matcher
        ("trailing",   remove_trailing_commas),   # ,} -> }
        ("pyliterals", normalize_true_none),      # True -> true, None -> null
        ("quotes",     single_to_double_quotes),
        ("truncation", close_open_strings_and_brackets),
    ]
    for name, fix in rungs:
        try:
            return json.loads(fix(text)), RepairReport(rung=name)
        except (json.JSONDecodeError, NoBalancedSpan):
            continue
    return None, RepairReport(rung="hopeless")

Two design details matter:

1. The brace matcher must be string-aware. Counting { and } naively breaks the moment a value contains a brace. Track quote state and escapes while scanning — ten lines, no dependencies.

2. The truncation rung is a last resort, and it must announce itself. Closing open strings and brackets turns {"items": [1, 2, 3 into something that parses — but you recovered 3 of maybe 60 items. That's why the function returns the report: rung="truncation" is your signal to log it, retry with a higher token cap, or tell the user the result is partial. A repair that silently pretends nothing happened is just hallucination with extra steps.

What this looks like in practice

obj, report = repair_json(raw_llm_output)
# -> obj = {"items": [1, 2, 3]}, report.rung = "truncation"

That's a real run from our test suite: fence stripped, trailing comma inside the array dropped, the truncated string closed — and the report says the answer came from the desperate rung, so the caller decides whether to trust it. A clean parse reports rung="clean"; over a week of production runs the rung distribution tells you whether your prompts or your token caps are the real problem.

This is the json_repair module in AgentOps Autopilot (MIT, stdlib only). Pair it with the retry pattern from the previous post: a truncated response should count as a retryable failure, not a success.

Next in the series: timeout handling so 40-step runs stop dying at step 41.

Python Retry with Exponential Backoff and Jitter (stdlib only)

AgentBasics — Thu, 12 Nov 2026 16:08:37 +0000

Most retry code I see in agent projects looks like this:

for i in range(3):
    try:
        return call_api()
    except Exception:
        time.sleep(2)
raise RuntimeError("gave up")

It survives a demo. In production it fails in four different ways, and each one is its own failure mode. Here's what's wrong and the stdlib-only pattern we ship instead.

Problem 1: it retries permanent errors

A 401 (bad key) and a 429 (rate-limited) are not the same failure. Retrying a 401 three times burns 6 seconds and a support ticket. Classify first:

RETRYABLE_EXCEPTIONS = (TimeoutError, ConnectionError, OSError)
RETRYABLE_STATUS = {408, 409, 425, 429, 500, 502, 503, 504}

Anything outside those lists should fail on the first attempt. In an agent loop, a permanent error you keep retrying becomes the retry spiral: the same failure five times in a row until the runtime hard-stops the turn.

Problem 2: fixed sleep creates a thundering herd

When an API wobbles, every client that went down at the same moment comes back at the same moment — fixed 2-second sleeps synchronize the retry wave and knock the service down again. The fix is exponential backoff plus jitter:

delay = min(cap, base * (2 ** attempt))   # 1.5, 3, 6, 12... capped
delay *= random.uniform(0.75, 1.25)       # jitter desynchronizes clients
time.sleep(delay)

The cap matters too. Uncapped exponential backoff means your fifth retry sleeps four minutes — through your agent's entire turn budget.

Problem 3: no total budget

This is the one that kills AI agents specifically. If each attempt gets its own timeout and each retry its own sleep, total worst-case time is timeout × (retries+1) + sleeps. A 45-second timeout with 3 retries can eat 4+ minutes — and agent platforms kill turns at 5. The retry loop needs a shared deadline across all attempts, so the last attempt gets whatever time is left, not a fresh allowance:

start = time.monotonic()
deadline = start + budget_s          # ONE budget for the whole call

for attempt in range(retries + 1):
    remaining = deadline - time.monotonic()
    if remaining <= 0:
        break                        # budget exhausted, fail honestly
    try:
        value = run_with_timeout(fn, min(timeout, remaining))
        ...
    except RETRYABLE_EXCEPTIONS:
        if time.monotonic() < deadline - backoff_base:
            time.sleep(jittered_delay)  # only sleep if budget survives it

Problem 4: "success" that isn't

The API returned 200 with 10 of 60 results. The call "succeeded" and your agent built on garbage. A retry helper for agents should take a semantic validator, and a rejected result should count as a retryable failure:

res = resilient_call(lambda: my_llm(prompt),
                     name="llm",
                     timeout=45, retries=3,
                     validate=lambda x: len(x["results"]) == expected)

The whole thing, assembled

Put the four pieces together — classification, jittered capped backoff, shared budget, semantic validation — and you get a wrapper that returns a structured result instead of either a value or an exception:

res = resilient_call(fn, name="llm", timeout=45, retries=3,
                     budget_s=120, validate=shape_check)
if res.ok:
    use(res.value)
else:
    # res.failure = {kind, name, attempts, last_error, elapsed_s}
    journal.record(res.failure)   # "same failure 5x today" becomes a query

The structured failure record is the quiet win: your agent can reason about why a call died ("429 three times, budget exhausted") instead of hallucinating a plausible answer over stack-trace soup.

This is exactly the resilient_call module in AgentOps Autopilot (MIT, stdlib only, 24/24 tests + a chaos gauntlet that injects each failure kind). If you'd rather paste it than rebuild it, the repo is free; the paid tiers add the templates and the runbook.

Next in the series: what to do when the LLM's JSON is truncated mid-string.

We asked the internet what agent failure mode we were missing. It shipped 24 hours later.

AgentBasics — Wed, 11 Nov 2026 22:30:16 +0000

Yesterday our launch thread ended with a genuine question: what's the sixth killer we're missing? We'd been writing about the five we keep hitting in production agents — wall-clock death at step 41, silent API truncation ("success" with 10 of 60 results), JSON cut mid-string by max_tokens, the double-charge after a timeout, and context death by a thousand tokens.

One reply stopped us cold. It came from someone who clearly runs agents against real work, and it was one line:

"the failure mode is partial success with a confident summary"

Read that again, because it's nastier than the other five. Every failure we'd built defenses against errors somehow — an exception, a timeout, a bad shape. This one doesn't. The call returned. The run completed. Eleven steps were planned, four left receipts, and then the model wrote a fluent, confident wrap-up of everything it "found." Your retry logic never fires because nothing failed. Your monitoring stays green. The user gets a beautifully written report of work that didn't happen.

We call it completion ≠ correctness. And the honest answer to "do our tools catch it?" was: no. Not one of them.

So today it shipped.

The defense: receipts, not vibes

The new module is receipt_audit.py (MIT, stdlib-only, ~120 lines). The idea is boring and that's why it works:

Plan. Before the run, declare the steps you intend to take.
Receipts. Every step that actually executes leaves a receipt — including partial ones (detail="4/11 rows").
Audit. Before the final summary is allowed to reach the user, diff planned steps against receipts.
Enforce. If the run is partial but the summary sounds complete, the summary gets rewritten out loud:

PARTIAL (4/11 receipts; not done: post_report, verify_upload).
Found the issues! Everything is in place.

The tone detector is deliberately crude — it flags summaries that say "all/every/done/successfully" when the receipt diff says otherwise, and it stays quiet when the summary already admitted partiality. The point isn't to catch every lie; it's to make pretending-competent structurally expensive.

The verdict also feeds straight into the failure journal as a partial_success record, so "same run went half-done 5 times this week" becomes a query instead of a vibe.

What this felt like as a builder

The loop — publish the failure taxonomy, get named a hole, ship the module, credit the person who named it by name in the README and changelog — took about 24 hours end to end. Two things surprised us:

The community audits better than we do. We shipped five modules around failures we'd personally bled on. A stranger named the sixth in one sentence because they'd bled on a different one. Your failure taxonomy is whatever your scars are; other people's scars are the roadmap.
"Replies shape the release" is only marketing if you don't ship. Say it once, ship once, in public, with credit — and it stops being a slogan and starts being a contract.

If your agent's final answer is a paragraph, that paragraph is currently unaudited. Diff it against its receipts.

The sixth killer: partial success with a confident summary. Named in our comments by @ChanYop. Now a module, credited in the changelog.

What's the seventh? The repo is free (MIT, zero dependencies):
https://github.com/AgentBasicsHQ/agentops-autopilot

The 5 failure modes killing your AI agent (and the 10-line fix for each)

AgentBasics — Sun, 08 Nov 2026 19:33:37 +0000

Every agent that calls external APIs meets the same five killers. Not model failures — plumbing failures. And they're the same five, in every language, for every framework, because they come from the network and the runtime, not the LLM.I got burned by all five in production (a 300-second runtime timeout that killed a 40-step run at step 41; an API that silently truncated "all 60 results" to 10 with no error; JSON cut mid-string by max_tokens; the same ValueError retried five times until a hard stop). So I extracted the fixes into five zero-dependency Python modules. MIT core: https://github.com/AgentBasicsHQ/agentops-autopilot## 1. The wall-clock killer (dies at step 41)The story: your agent runs fine for 40 steps, then the platform's 300s turn limit fires. Everything is lost. No partial work, no status, nothing.The fix: a turn budget shared across every call, with a reserve, and a ship-partial policy.

pythonfrom turn_budget import TurnBudgetbudget = TurnBudget(total_s=280, reserve_s=20)for step in plan: if not budget.can_afford(estimate_s=15): ship_partial(status="ran out of time", done=completed, remaining=plan[i:]) break do(step)

The rule that matters: reserve_s. Without a reserve you discover the deadline inside the last call, which is how you get zero output instead of 80% output.## 2. Silent API truncation ("all results" → 10 results, no error)The story: you ask an API for everything. It returns a page. No error, no "more available" flag you bothered to check. Your agent reports a confident, wrong number.The fix: semantic validation inside the retry wrapper — validate shape and count, not just "it returned".

pythonfrom resilient_call import resilient_callr = resilient_call( lambda: fetch_page(cursor), name="fetch_all", validate=lambda x: len(x["items"]) > 0 and "cursor" in x, retries=3, timeout=30,)

A validate= that encodes your expectation ("non-empty", "has next cursor", "count matches the header") turns a silent wrong-answer into a loud retryable failure. This one check is worth the whole module.## 3. Truncated JSON (max_tokens cuts mid-string)The story: the model returns 95% of a JSON document and stops. json.loads blows up, your retry loop re-asks the whole thing, you pay double, and sometimes the model truncates at exactly the same spot.The fix: a deterministic repair ladder — fence-strip, trailing-comma removal, Python-literal conversion, then bracket-closure for truncated docs — with an honesty report telling you which rung fixed it.

pythonfrom json_repair import repair_jsonobj, report = repair_json(llm_output)if report.rung == "truncation": # recovered what exists; re-ask ONLY the missing scope log.warning("truncated at %s; partial recovered", report.detail)

strict_json=True in resilient_call catches the same failure one level earlier: if your call returns a string that isn't valid JSON, the call fails instead of returning the broken string three steps downstream.## 4. The retry spiral (same failure, five times, then a hard stop)The story: a call fails. The retry logic retries. It fails identically — it's a ValueError, not a network flake — and burns your whole turn before the platform kills the run.The fix: two parts. (a) Retry an allowlist of transient classes, never a blanket except. (b) An anti-spin counter: if this exact (kind, step) failure has happened 3 times today, stop and escalate.

pythonfrom failure_journal import FailureJournaljournal = FailureJournal("logs/failures.jsonl")# inside the failure branch:if journal.same_failure_count_today(r.failure["kind"], step_name) >= 3: raise SystemExit(f"anti-spin: {step_name} failed 3x today — stop and report")

Bonus: journal.top_failures() gives you a ranked list of what actually breaks your agent, which is how you prioritize fixes instead of guessing.## 5. Silent context exhaustion (amnesia before the crash)The story: the context window fills gradually. Your agent doesn't crash — it forgets. Early instructions quietly vanish, and it hallucinates continuity. The crash is loud; the amnesia is silent, and it's worse.The fix: a pressure gauge with graduated actions, checked between steps.

pythonfrom context_pressure import ContextPressurecp = ContextPressure(model_window=128_000, system_overhead=3_000)cp.add("tool", tool_output) # every message goes throughmatch cp.action(): # leading indicator, not post-mortem case "prune_tool_results": cp.prune_tool_results(keep_last=5) case "summarize_oldest_turns": summarize_and_replace(oldest=8) case _: checkpoint_and_restart() # hard_stop_ship_partial

Pruning tool results (usually 70% of your tokens) buys back most of the window for free. Do it before the API returns a context error, not after.---## The boring synthesisNone of these are model problems. They're the same failure classes that network programmers solved decades ago — timeouts, budgets, backoff, validation, logging — re-applied to a system where the "network call" is an LLM and the failure modes include silent semantic truncation.All five modules are stdlib-only, Python 3.10+, 18-check test suite: https://github.com/AgentBasicsHQ/agentops-autopilot (MIT core).If you want the production templates (agent loop, cron pipeline, gateway wrapper, eval harness) + the 12-failure-mode runbook, that's the paid tier on the site — but the free core is genuinely useful on its own, and I'd rather you test it first.What failure mode am I missing? Genuinely asking — the runbook is a living doc and the next addition comes from comments.