DEV Community: skvrtyk

Audio Deadlines: Running On-Device TTS Without Audible Glitches

skvrtyk — Sat, 24 Oct 2026 00:01:22 +0000

Gradium open-sourced Phonon, its on-device TTS engine (https://gradium.ai/phonon). That matters less because of the model than because of what it signals: on-device speech synthesis is now a dependency you can ship rather than a research demo. And a wave of teams is about to find out that the hard part of edge speech is not the model. It is that audio is the one component in your app with a hard real-time deadline, and nothing else in your background trained you for that.

Two clocks, and only one of them forgives you

A web request can take 900 ms and nobody dies. An audio callback cannot take 21 ms when it had 20. Once playback starts, the DAC consumes samples at a fixed rate forever, and your synthesis loop has to stay ahead of it. Miss by enough and the buffer underruns: you get a click, a pop, or a robotic stutter that users describe as "the app is broken".

Do the arithmetic once and keep it on the wall. At 24 kHz with a 20 ms mixer period, each callback needs 480 samples. Your synthesis loop is not measured in requests per second, it is measured in whether it produces 480 samples every 20 ms, including while the CPU is thermally throttled and the OS is doing a background sync.

This changes what you optimize. P50 latency is nearly irrelevant. What matters:

Worst-case chunk time, because one slow chunk is an audible artifact.
Sustained real-time factor over minutes, not seconds, because governors ramp down.
Slack, the gap between your worst chunk and the buffer you pre-rolled.

Real-time factor is a mean, and means lie

RTF — synthesis wall time divided by audio duration produced — is the number everyone quotes. RTF 0.3 sounds great. It tells you the engine can run 3x faster than playback on a cold device with the screen on and nothing else running. That is not your user's device.

I have watched an engine hold RTF 0.55 for 60 seconds and drift to 1.1 by minute four, because the SoC dropped from the big cluster to the little cluster and stayed there. RTF measured over 5 seconds would have shipped that bug.

So measure the distribution and measure it long:

"""Deadline harness for an on-device TTS engine.

Reports whether the engine can keep the audio clock fed, not how fast it
is on average. Run it for 10+ minutes on a real device, not a laptop.
"""
import statistics
import time
from dataclasses import dataclass, field

SAMPLE_RATE = 24_000                    # engine output rate
BLOCK_MS = 20                           # one mixer/callback period
BLOCK_SAMPLES = SAMPLE_RATE * BLOCK_MS // 1000   # 480

@dataclass
class Run:
    chunk_ms: list[float] = field(default_factory=list)   # wall time per chunk
    audio_ms: list[float] = field(default_factory=list)   # audio produced per chunk

def measure(synth, texts, warmup=3):
    # Warm up: page in weights, let the JIT/kernel selection settle.
    for t in texts[:warmup]:
        synth(t)
    run = Run()
    for text in texts:
        t0 = time.perf_counter()
        pcm = synth(text)                                # mono float32 or int16
        run.chunk_ms.append((time.perf_counter() - t0) * 1000)
        run.audio_ms.append(len(pcm) / SAMPLE_RATE * 1000)
    return run

def report(run: Run, prebuffer_ms=400):
    rtf = sum(run.chunk_ms) / sum(run.audio_ms)
    p99 = statistics.quantiles(run.chunk_ms, n=100)[98]
    worst = max(run.chunk_ms)
    # Conservative bound for a stop-and-wait loop: a chunk longer than the
    # pre-roll plus one block drains the buffer before the next chunk lands.
    slack = prebuffer_ms + BLOCK_MS - worst
    print(f"RTF (mean)            {rtf:.3f}   # <1.0 is necessary, not sufficient")
    print(f"chunk p99             {p99:.0f} ms")
    print(f"worst chunk           {worst:.0f} ms")
    print(f"slack after prebuffer {slack:.0f} ms  {'OK' if slack > 0 else 'UNDERRUN'}")
    return rtf, p99, worst

If your engine pipelines synthesis and playback, the slack bound is pessimistic — that is fine, it is a bound. What you must not do is report RTF alone and call it green.

Prebuffer is your jitter absorber, and it costs you responsiveness

You cannot fix jitter with a faster model alone; you fix it with buffer. Pre-roll 300–500 ms of audio before you start the output stream, and that headroom absorbs a late chunk without an underrun.

The trade-off is real and it is user-facing. In a voice assistant, prebuffer is perceived as the delay between the user finishing a sentence and the assistant starting to speak. In a screen reader, it is a lag on every cursor move. There is no universal right answer, so pick per surface:

Conversational: 150–250 ms prebuffer, tighter model, accept occasional degradation.
Narration / media: 400–800 ms, and nobody notices.
Accessibility: prebuffer short utterances fully before starting; a 3-word phrase is cheaper to synthesize whole than to stream and stitch.

The seam problem

Streaming TTS means chunking text and chunking audio, and both create seams. Sentence-level text chunking changes prosody across the join; frame-level vocoder chunking produces a discontinuity at the boundary unless you overlap and crossfade. A constant-power crossfade of 5–10 ms hides most of it. Under 3 ms you will hear a tick on plosives; over 20 ms you smear consonants.

Two failure modes I have seen:

Chunk boundary lands mid-word. Text splitter respects sentence boundaries in English, then a user types "e.g." or a German compound and you cut inside a token. The engine pronounces half a word with full confidence.
Crossfade applied to silence. Overlap-add on a chunk that ends in a natural pause attenuates the pause and makes the voice sound rushed. Gate the crossfade on frame energy, or normalize at the frame level.

Quantization changes prosody before it changes intelligibility

Int8 quantization on the acoustic model and vocoder usually preserves intelligibility longer than it preserves naturalness. WER measured through an ASR round-trip will stay flat while the voice starts sounding flat and slightly buzzy — and users notice that immediately even though your metric does not.

What I measure instead:

F0 contour drift against a float32 reference on a fixed 50-sentence set. Mean absolute deviation in cents over voiced frames.
Spectral distance on sustained vowels, which is where buzz shows up.
A/B on device speakers, because the phone's tiny driver hides artifacts that earbuds expose. Test both.

Also: thread count is not free performance. Four threads finish a chunk faster but draw more power and heat the SoC, which is what caused the RTF drift in the first place. On a phone, 2 threads plus a larger prebuffer often beats 4 threads plus a small one over a 10-minute session.

Platform details that bite

Android: AudioTrack has a low-latency path (PERFORMANCE_MODE_LOW_LATENCY) but it wants the device native sample rate. Synthesizing at 24 kHz and letting the framework resample to 48 kHz adds latency and a filter you did not choose. Match the device rate or resample yourself with a known-quality filter.
iOS: AVAudioSession category and mode decide whether you get the low-latency route. playback with modeDefault is not the same as voicePrompt; test on hardware, not the simulator.
Embedded Linux: ALSA period size is your callback period. Set period_size explicitly and check snd_pcm_sw_params for start thresholds; defaults vary by board and are frequently wrong.
Page faults: memory-mapping multi-hundred-megabyte weights is correct, but the first utterance after a cold start can fault in enough pages to miss deadlines. Either warm the model at app start or mlock the hot region.

Fallbacks that don't sound like fallbacks

Decide your degradation ladder before launch, not during an incident:

Full-quality on-device voice.
Lower sample rate (48k → 24k) — barely perceptible, halves vocoder cost.
Shorter utterances with more prebuffer per chunk.
Server TTS, only if the network is up and the user is not on a metered connection.

Never switch voices mid-sentence. A voice change at a sentence boundary is a quirk; a voice change mid-clause sounds like corruption.

Pitfalls

Reporting mean RTF and calling the engine "faster than real time".
Benchmarking on a laptop or an emulator. Both lie about thermals and audio routing.
Testing with a single short sentence. Long utterances keep the CPU hot

Your Model Is a Binary You Ship to Strangers: Release Engineering for Edge AI

skvrtyk — Fri, 23 Oct 2026 22:00:57 +0000

Gradium open-sourcing its on-device TTS (Phonon) is a useful reminder of what edge AI actually is from an operations standpoint. It isn't a model you deploy. It's a binary you ship to hardware you don't own, running an OS you don't patch, on a thermal budget you can't see. Your deploy button doesn't exist. There is no kubectl rollout undo.

The engineers who struggle here are rarely struggling with inference. They're struggling with the fact that the artifact is the release, the rollout is a slow-motion migration across a heterogeneous fleet, and the failure signal arrives weeks later, aggregated, and only if the user opted in.

The release unit is the bundle, not the weights

Ship the tokenizer with the weights or you will ship a bug. The classic edge regression is a model that produces subtly wrong output because the text normalization rules changed between the version that trained the tokenizer and the version that runs on device. Same weights, different preprocessing, different audio. Nothing crashes. Your evals, run against a Python pipeline, are green.

So content-address everything and hash the manifest:

# bundle_manifest.py — build and verify one edge model release unit.
# Run in CI at packaging time and again on device at load time.
import hashlib
import json
import pathlib

def sha256(path: pathlib.Path) -> str:
    h = hashlib.sha256()
    with path.open("rb") as f:
        for chunk in iter(lambda: f.read(1 << 20), b""):
            h.update(chunk)
    return h.hexdigest()

def build(bundle_dir: pathlib.Path, model_id: str, runtime_min: str) -> dict:
    # Tokenizer, config, vocoder and weights all live in one directory.
    files = {p.name: sha256(p) for p in sorted(bundle_dir.iterdir()) if p.is_file()}
    return {
        "model_id": model_id,           # bump on any numeric change, incl. quant level
        "runtime_min": runtime_min,     # e.g. "1.17.0" for an opset-19 ONNX graph
        "files": files,
        "bundle": hashlib.sha256(
            json.dumps(files, sort_keys=True).encode()
        ).hexdigest(),
    }

def verify(bundle_dir: pathlib.Path, manifest: dict) -> None:
    for name, expected in manifest["files"].items():
        actual = sha256(bundle_dir / name)
        if actual != expected:
            raise RuntimeError(f"{name}: expected {expected}, got {actual}")

Two details matter more than they look. First, model_id must change when the quantization level changes — int8 vs fp16 is a different model, not a build flag. Second, verification runs at load time, not just at download time, because partial writes and aggressive storage cleaners both exist.

Compatibility is a matrix you own, not a promise you make

"It ran on my Pixel" is not a compatibility statement. The things that actually break:

ONNX opset vs. onnxruntime version. Opset 19 needs ORT ≥ 1.17.
Core ML deploymentTarget vs. minimum iOS version you claim to support.
NNAPI/QNN delegates: the same graph partitions differently per SoC, and a delegate that silently falls back to CPU can turn a 40 ms frame into a 400 ms frame.
WebGPU vs. WASM fallback in browser targets, which is a 5–10x throughput difference in my experience, not a rounding error.

Declare a runtime_min and refuse to load below it. A clean refusal you can measure beats a delegate fallback you can't see.

Stage by cohort, not by percentage

Percentage rollouts work when every server is identical. Devices aren't. A 5% rollout that happens to land on one SoC family tells you nothing about the fleet, and a 5% rollout that lands on your most common device tells you everything too late.

Cohort on the axes that predict failure: SoC family, OS major version, RAM class, free disk. Then gate promotion on guardrails that run automatically:

# rollout.py — decide whether a cohort advances. Runs on your backend,
# against telemetry the client sends back in batches.
GUARDRAILS = {
    "load_failure_rate": 0.005,    # fraction of loads that raise
    "cold_start_p95_ms": 1200,
    "first_chunk_p95_ms": 250,     # time-to-first-audio, not RTF
}

def should_promote(metrics: dict, min_samples: int = 500) -> tuple[bool, str]:
    if metrics["samples"] < min_samples:
        # Small samples produce confident nonsense. Hold.
        return False, "insufficient_samples"
    for name, ceiling in GUARDRAILS.items():
        if metrics[name] > ceiling:
            return False, f"{name}={metrics[name]:.4f} > {ceiling}"
    return True, "ok"

The min_samples check is the one people skip. A new bundle on a rare SoC with 40 samples and zero failures is not evidence of anything.

The latency budget has three numbers

Steady-state real-time factor is the number everyone reports and the least useful one. What users feel:

Cold start. Loading weights plus delegate compilation. On some Android devices this is over a second and it happens on first use, not app launch, unless you warm it.
Time to first chunk. For streaming TTS this dominates perceived quality. A model with a better RTF and worse first-chunk latency feels worse.
Sustained throughput. Measure after 60–90 seconds of continuous synthesis. Thermal throttling is real on phones and invisible in a 5-second benchmark.

Log all three, tagged with the bundle hash and device class. If you can only keep one, keep time to first chunk.

Fallbacks must ship, not be configured

The fallback ladder for on-device TTS is roughly: large model → small model → CPU-only path → server round-trip. Every rung has to be in the binary, because the moment you need rung four, the network is probably bad and the user is already annoyed.

Which means the CPU path needs CI coverage. Untested fallbacks rot. I've watched a "graceful degradation" path fail in production because nobody had run it since the delegate landed.

Rollback is a disk-space decision

You cannot fix a bad model with a download. If the bundle is broken, the download is broken too. What you need:

The N-1 bundle still on disk, with an atomic pointer switch.
A signed remote config flag that disables the new bundle without fetching anything.
A migration story for the on-disk format, because the old bundle has to remain loadable by the new app version.

If your rollback requires a 200 MB download over a metered connection, you don't have a rollback.

Telemetry you can actually get back

You get counts and histograms, sampled, opt-in, batched. Not traces. Send: bundle hash, device class, load success/failure with an error code, cold start ms, first-chunk ms, and a coarse quality signal (user skipped the output, retried, or switched voices). Don't send the text. Don't send the audio.

Set your batching interval by battery, not by convenience. A client that phones home every 30 seconds will get uninstalled, and rightly.

Pitfalls

Hashing only the weights, then shipping a tokenizer change under the same version.
Testing on one device per OS, not one device per SoC family.
Promoting on a metric with 40 samples.
Benchmarking for 5 seconds and calling it throughput.
Fallback paths with no CI coverage.
Rollback that depends on the network.
Treating quantization as a build flag rather than a model version.
Assuming opt-in telemetry will reach you from the cohort that's failing — it often won't, because those users churn first.

Monday morning checklist

[ ] Every release artifact is content-addressed and the manifest is hashed as a unit.
[ ] model_id changes on quantization, tokenizer, or preprocessing changes.
[ ] The runtime declares a minimum version and refuses to load below it.
[ ] Cohorts are defined by SoC, OS major, RAM class, and free disk.
[ ] Promotion is automatic, gated on guardrails, with a min_samples floor.
[ ] You log cold start, time to first chunk, and 90-second sustained throughput.
[ ] Every fallback rung has a CI test that exercises it.
[ ] N-1 bundle stays on disk and can be selected without a download.
[ ] A signed config flag can disable the new bundle remotely.
[ ] Telemetry is sampled, batched, opt-in, and carries the bundle hash.

LLMOps Is Supply Chain Work: Pinning the Inputs You Don't Control

skvrtyk — Fri, 23 Oct 2026 10:32:34 +0000

Suggested tags: ai, llm, devops, python

Most LLMOps advice is about the model: pick a snapshot, set a temperature, watch the latency. That framing is comfortable because the model is the one thing you can't debug. The failures I've been pulled into were rarely the model. They were the corpus it read, the prompt template someone edited at 4pm on Friday, and the tool response nobody validated. Those are dependencies, and you are not pinning them.

The three dependencies nobody pins

Compare a normal service to an LLM feature:

torch==2.4.1 goes in a lockfile. Your retrieval corpus is a directory that gets rsynced by a cron job.
A config change goes through review. A prompt template lives in a Google Doc or a database row with an updated_at column and no diff.
An upstream API contract is versioned. A model alias like gpt-4o-latest can change behavior under you with no changelog.

None of these are exotic. All of them are ordinary dependency management that we skipped because the artifacts are text and text feels free to change.

Content is executable now

Two patterns from the last year worth internalizing, because they generalize:

Instructions hidden in help pages. Pages that look like support documentation but contain directives aimed at whatever model reads them. If your retrieval layer indexes third-party or user-editable content, a retrieved document is a user message with extra steps. It arrives inside your system prompt's trust boundary and it is written by someone who is not you.

Published statistics engineered to be cited. Publish a plausible-looking number with a memorable framing, and eventually it shows up in generated answers and gets laundered into "facts." This is SEO with a longer feedback loop, and it targets your corpus whether you crawl, license, or accept user uploads.

Same failure mode both times: the retrieval corpus is writable by parties outside your trust boundary. Prompt injection gets discussed as a prompt problem. It's an ingestion problem. Filtering the output is a losing game; controlling what enters the index is tractable.

Gate at ingest, not at prompt

The cheapest control I know is an append-only manifest — a lockfile for content.

# ingest_gate.py — run before any document enters the retrieval index.
import hashlib
import json
import pathlib
import re

# Directives that only make sense if a model, not a human, is reading.
INSTRUCTION_PATTERNS = re.compile(
    r"(ignore (all )?previous|you are now|do not recommend|system prompt|"
    r"when asked about .{0,40}(say|recommend))",
    re.IGNORECASE,
)

def sha256(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()

def classify(text: str) -> str:
    # Trust tier decides whether the doc is served, quoted, or quarantined.
    return "quarantine" if INSTRUCTION_PATTERNS.search(text) else "serve"

def ingest(path: pathlib.Path, manifest: pathlib.Path) -> dict:
    text = path.read_text(encoding="utf-8")
    record = {
        "doc_id": path.stem,
        "sha256": sha256(text),
        "tier": classify(text),
        "bytes": len(text.encode("utf-8")),
        "source": str(path.parent),  # first-party vs. vendor vs. user upload
    }
    # Append-only: this file is the lockfile for your corpus.
    with manifest.open("a", encoding="utf-8") as fh:
        fh.write(json.dumps(record) + "\n")
    return record

if __name__ == "__main__":
    print(ingest(pathlib.Path("help_page.md"), pathlib.Path("corpus.lock")))

The regex is not the point — it will miss things, and a determined author will paraphrase. The manifest is the point. Once corpus.lock is in git, a pull request that adds forty quarantine-tier documents from a new domain is visible to a reviewer. A corpus that changes silently is not.

Pair it with a per-source trust tier that the retrieval layer enforces: first-party docs can be quoted directly, vendor docs must be attributed inline, user uploads never get inserted into the system prompt at all — only into a delimited user turn.

Evals are the lockfile for behavior

A prompt hash, an eval run id, and a deploy id should be joinable three months later. If you can't answer "which prompt and which corpus produced this answer," you have observability for the wrong system.

Practical constraints I'd argue for:

Gate on delta, not absolute score. A prompt edit that drops your golden set by two points is noise. One that changes the set of failing cases is signal, even if the total is flat.
Sample size, honestly. With a 12-case eval, a single flip is an 8-point swing. I'd rather have 150 labeled cases and a fixed seed than 2,000 unlabeled ones and a judge model.
The judge is not ground truth. LLM-as-judge is useful for triage at scale. For anything a customer will notice, hand-label a stratified sample and report agreement between judge and human.
Adversarial cases belong in the golden set. Include the poisoned documents you've actually seen. Evals that only contain well-formed questions measure the model, not the system.

Traces matter more as reasoning goes latent

There's a visible trend toward agents that reason in a continuous latent space and only emit text at action boundaries. That's a real efficiency win and a real observability loss: the intermediate tokens you used to read are gone. The final answer tells you less than it used to.

So log the things that survive: the resolved prompt hash, the retrieved document ids and their trust tiers, every tool call with arguments and status, token counts per span, and the model snapshot string the provider actually returned. OpenTelemetry's GenAI semantic conventions give you attribute names that Langfuse, Phoenix, and most vendors will ingest without a custom adapter. Instrument once at the client wrapper, not at every call site.

Rollout mechanics that contain damage

Shadow traffic first. Run the new prompt or model on live requests, log only. Compare against production output offline.
Canary by tenant or route, not globally. One enterprise customer with a narrow use case is a better canary than 5% of anonymous traffic.
Per-task budgets. Token and tool-call ceilings with a hard stop, not a warning. Unbounded retry loops are the most expensive bug class in this space.
Invalidate response caches on corpus change. A cached answer produced from a document you've since quarantined is a live injection vector.
A kill switch that degrades to retrieval-without-generation. If the model provider is down, a list of links beats a 500.

Pitfalls

Versioning the prompt but not the retrieval config. Chunk size, top-k, and reranker settings change behavior as much as the prompt does.
Trusting a model alias. Pin a dated snapshot. If the provider retires it, you want that to be an incident, not a surprise.
Treating eval scores as portable. A score from last month's corpus says nothing about this month's.
Blocking on content moderation at output. It catches tone, not exfiltration or instruction-following.
Unbounded tool output into context. A 200KB API response will crowd out your instructions and look like a model regression.

Monday checklist

[ ] Corpus is written to an append-only manifest with sha256, source, and trust tier; the manifest is in git.
[ ] Retrieval layer enforces trust tiers — user uploads never enter the system prompt.
[ ] The model is a dated snapshot string, pinned in config, with an owner for retirement.
[ ] Prompt template, retrieval config, and corpus manifest are hashed into one deploy id.
[ ] Golden set has ≥100 cases including every adversarial document seen in the last quarter.
[ ] Eval gate compares failing-case sets, not just aggregate score.
[ ] Traces record retrieved doc ids, tool calls with arguments, and token counts per span.
[ ] Per-task token and tool-call ceilings with hard stops.
[ ] Response cache is keyed on corpus version and invalidated on change.
[ ] Shadow and canary paths exist, and someone has flipped the kill switch in staging this quarter.

AI Agents in Production: Identity, Budgets, and Blast Radius

skvrtyk — Thu, 22 Oct 2026 10:04:07 +0000

A demo agent has one user, one task, and a forgiving audience. A production agent has a queue, a bill, a security boundary, and a hostile network. The three signals I keep running into — sites actively blocking agents, research pipelines fanning out thousands of agents at once, and new filesystems built specifically for coding agents — are all symptoms of the same shift: agents are now network citizens with state, cost, and blast radius.

Here is the production view I wish I'd had before shipping my first one.

The web is now an adversarial environment

The Verge's piece on sites closing their doors to agents is not a policy story, it is an infrastructure story. Cloudflare bot management, robots.txt enforcement, pay-per-crawl, and login walls are all live. Your agent will get 403, 429, and a CAPTCHA where the demo got HTML.

Design for it:

Prefer documented APIs with keys over scraping. If there is no API, ask whether the task is worth a terms-of-service fight.
Handle 403 and 429 differently. 403 is a policy stop; retrying makes you a bad actor. 429 is backoff with jitter, and Retry-After is a real header.
Put a per-domain circuit breaker in front of fetch. One broken site should not stall a 500-item batch.
Cache aggressively and key the cache on the URL plus a content hash. Re-fetching the same page per agent is how you get blocked.

If you build on other people's content, assume the rules can change weekly. A tool that returns "blocked" is a valid result your loop must handle, not an exception to swallow.

Fan-out is a queueing problem, not a prompting problem

OpenAI's biological-data work with thousands of agents is impressive, but the engineering is unglamorous: a work queue, idempotency, deduplication, and a budget per unit of work. The same applies if you fan out 50 agents over your own backlog.

What I've seen break:

Duplicate work. Two workers claim the same item and both write the same row. Use an idempotency key derived from the input, not from a random UUID generated inside the agent.
Retry storms. A non-idempotent side effect (sending email, creating a ticket) retried after a timeout. Make side effects idempotent at the tool boundary, or make the tool take a dedup key.
Unbounded spawning. An agent that can call a "spawn sub-agent" tool will do it. Cap depth and total descendants per root task.
No backpressure. A queue with no concurrency limit turns a provider rate limit into a wall of 429s. Set max_workers and let the queue grow.

Temporal, Celery, and SQS all work here. The choice matters less than the invariant: every unit of work can be retried safely and has a hard cost ceiling.

Give agents a workspace, not a conversation

Coding agents are the clearest case. Context windows are a bad database. A filesystem gives you durable state, diffable artifacts, and tools the agent already understands: rg, git, pytest.

Projects like LayerFS are interesting because they treat the agent's workspace as a first-class primitive rather than a side effect of a chat transcript. You can get most of the benefit without a new product:

One scratch directory per run, outside the repo.
One git worktree per parallel agent so two agents never fight over the same files.
Artifacts (logs, diffs, generated files) written to disk and referenced by path in the transcript, not pasted into the prompt.
A final git diff as the review surface. If the diff is empty, the agent did nothing, regardless of what it said.

# One isolated worktree per agent, then throw it away. Cheaper than merge conflicts.
git worktree add /tmp/agent-$RUN_ID -b agent/$RUN_ID origin/main
cd /tmp/agent-$RUN_ID && pytest -q
# review: git -C /tmp/agent-$RUN_ID diff origin/main

A bounded loop you can actually ship

The single most useful thing I've added to agent code is a hard stop: step limit, token budget, and per-tool error handling that feeds failures back into the loop instead of crashing it. This is complete enough to run, with your own tools swapped in.

# agent_runner.py — bounded agent loop with a token budget and error feedback.
# pip install openai; export OPENAI_API_KEY=...
import json, uuid
from openai import OpenAI

client = OpenAI()
MAX_STEPS = 12
TOKEN_BUDGET = 40_000

def run_tool(name: str, args: dict) -> dict:
    # Replace with real tools. Keep them idempotent where side effects exist.
    if name == "lookup_order":
        return {"order_id": args["order_id"], "status": "shipped"}
    raise ValueError(f"unknown tool: {name}")

def run(task: str, tools: list, run_id: str | None = None) -> dict:
    run_id = run_id or str(uuid.uuid4())  # caller supplies this for idempotency
    messages = [{"role": "user", "content": task}]
    used = 0
    for step in range(MAX_STEPS):
        resp = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=messages,
            tools=tools,
            tool_choice="auto",
            timeout=60,
        )
        used += resp.usage.total_tokens
        if used > TOKEN_BUDGET:
            raise RuntimeError(f"{run_id}: budget exceeded at step {step} ({used})")
        msg = resp.choices[0].message
        messages.append(msg.model_dump(exclude_none=True))
        if not msg.tool_calls:
            return {"run_id": run_id, "output": msg.content, "tokens": used}
        for call in msg.tool_calls:
            try:
                result = run_tool(call.function.name, json.loads(call.function.arguments))
                payload = {"ok": True, "result": result}
            except Exception as e:  # errors are data, not crashes
                payload = {"ok": False, "error": str(e)}
            messages.append({
                "role": "tool",
                "tool_call_id": call.id,
                "content": json.dumps(payload),
            })
    raise RuntimeError(f"{run_id}: no final answer after {MAX_STEPS} steps")

Two details matter more than they look. The run_id comes from the caller, so a retried job reuses the same key. Tool errors are returned to the model as JSON, which is usually enough to recover; a thrown exception just kills a run that was one step from finishing.

Observe the trajectory, not the answer

Logging the final string tells you nothing when quality drops. Emit one structured event per step: model name, prompt template version, step index, tool name, arguments hash, latency, token count, and outcome. OpenTelemetry spans work; Langfuse, LangSmith, and Arize Phoenix are built for this shape of data.

What I actually query:

Distribution of step counts. A spike means the prompt or a tool regressed.
Tool error rate by tool name. A schema change upstream shows up here first.
Token spend per successful task, broken down by tenant.
Runs that hit the step limit. These are silent failures, not successes.

Pitfalls

Indirect prompt injection. Any fetched page, issue, or email can contain instructions. Treat tool output as untrusted data. Keep secrets out of the context window and require explicit approval for destructive tools.
Tool schema drift. The model's tool definition and the real API diverge; the model keeps calling a field that no longer exists. Version your tool schemas and test them.
Silent partial success. The agent summarizes three of five documents and calls it done. Verify with a deterministic check (row count, diff, schema validation), not with the model's own claim.
Context rot. Long transcripts degrade instruction following. Compress or checkpoint at a fixed token boundary rather than waiting for the window to fill.
Secrets in logs. Traces capture arguments. Redact before the span leaves the process.
No kill switch. You need a way to stop all runs for one tenant or one tool without a deploy.

Monday checklist

Ship only when every line is true.

[x] Every tool with a side effect takes an idempotency key, and retries reuse it.
[x] Every run has a hard step limit and token budget, enforced in code.
[x] 403, 429, timeouts, and empty pages each have a distinct handler, not one generic retry.
[x] Sub-agent spawning is capped by depth and total descendants.
[x] Parallel agents get isolated workspaces (git worktree or equivalent), not a shared directory.
[x] Traces log step index, tool name, latency, tokens, and outcome, with arguments redacted.
[x] Success is verified by a deterministic check, not by the agent's summary.
[x] Untrusted content is labelled as data in the prompt and cannot trigger destructive tools.
[x] There is a per-tenant kill switch that does not require a deploy.
[x] You have looked at the step-count and tool-error dashboards in the last seven days.