<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: skvrtyk</title>
    <description>The latest articles on DEV Community by skvrtyk (@dima_kruglov_608741d9a416).</description>
    <link>https://dev.to/dima_kruglov_608741d9a416</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4200170%2F2fa276f2-d4e3-4468-b4b1-b259b9482a84.png</url>
      <title>DEV Community: skvrtyk</title>
      <link>https://dev.to/dima_kruglov_608741d9a416</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/dima_kruglov_608741d9a416"/>
    <language>en</language>
    <item>
      <title>The Degradation Ladder: LLMOps for When the Model Is Slow, Wrong, or Down</title>
      <dc:creator>skvrtyk</dc:creator>
      <pubDate>Sat, 31 Oct 2026 23:36:58 +0000</pubDate>
      <link>https://dev.to/dima_kruglov_608741d9a416/the-degradation-ladder-llmops-for-when-the-model-is-slow-wrong-or-down-3fm3</link>
      <guid>https://dev.to/dima_kruglov_608741d9a416/the-degradation-ladder-llmops-for-when-the-model-is-slow-wrong-or-down-3fm3</guid>
      <description>&lt;p&gt;Most LLM features ship with one path: prompt in, frontier model out. That path is the most expensive, slowest and least available of everything you could have built. When it degrades — and it will, because you're calling someone else's service over a network — the feature either hangs, returns something that fails a validator downstream, or throws a 500 at the user.&lt;/p&gt;

&lt;p&gt;The fix isn't a better prompt. It's an ordered ladder of tiers, each with its own budget, plus a gate between them and one log line per request recording which tier answered. That's the part of LLMOps that survives contact with production.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three failures, not one
&lt;/h2&gt;

&lt;p&gt;Group your incidents by cause before you design anything:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Latency.&lt;/strong&gt; p50 is 800ms, p99 is 30s. On a voice interface users hear dead air and hang up; on a form they click again and you double your traffic. Your provider's tail is your tail.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Quality.&lt;/strong&gt; HTTP 200, plausible output, wrong answer. Signalled by a downstream &lt;code&gt;KeyError&lt;/code&gt;, a broken citation, or nothing at all until a customer complains.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Availability.&lt;/strong&gt; 429s, 503s, a regional outage, an expired key, or a pinned model version being deprecated out from under you.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Each needs a different response. A timeout doesn't want a retry, it wants a shorter path. A 429 wants a retry with jitter. A quality failure wants &lt;em&gt;escalation&lt;/em&gt;, not repetition — re-running the same tier with the same prompt returns the same plausible garbage.&lt;/p&gt;

&lt;h2&gt;
  
  
  Retries are a load amplifier
&lt;/h2&gt;

&lt;p&gt;Most SDKs default to 2–3 retries. With no jitter, at p99 every client retries in the same window and you DDoS the provider that was already struggling; you burn quota, escalate rate limits, and turn a slow minute into a 20-minute outage. I've watched this happen with a 40-line background job.&lt;/p&gt;

&lt;p&gt;Two rules I now apply everywhere:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Retry at most once at the same tier, only for 429/503/timeout, only with jitter, and only if the remaining deadline allows a full attempt.&lt;/li&gt;
&lt;li&gt;Enforce the deadline at the transport layer — &lt;code&gt;timeout=&lt;/code&gt; on the HTTP client — not just with &lt;code&gt;asyncio.wait_for&lt;/code&gt;. Cancelling the coroutine doesn't reliably close the socket; the provider still finishes the request, still bills it, still counts it against your rate limit.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  The ladder
&lt;/h2&gt;

&lt;p&gt;Order tiers ascending by cost and latency, not by quality. For a typical text feature:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Exact cache&lt;/strong&gt;, keyed on a hash of &lt;code&gt;(model_id, prompt_version, normalized_input)&lt;/code&gt;. Free, instant. On support-style traffic this is usually the biggest single cost lever you have.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Task-specific model.&lt;/strong&gt; A small classifier or extractor you trained or fine-tuned. For a narrow, well-defined task this isn't a compromise — a task-specific model can beat a frontier model on accuracy, latency and cost simultaneously. Don't take that on faith; measure it on 200 labelled examples from your own traffic.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Small general model.&lt;/strong&gt; Fast, cheap, fine for the majority of requests that are easy.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Frontier model.&lt;/strong&gt; Entered only when the gate rejects what came before.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Static fallback.&lt;/strong&gt; A canned answer, a form, a human queue. This tier must exist and must be exercised in staging, or your graceful degradation is a &lt;code&gt;TypeError&lt;/code&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;For voice or streaming, reorder: time-to-first-token dominates everything, so a tier that takes 3s before the first token is unusable no matter how good it is. You may jump straight to the frontier model with a hard cutoff and a shorter spoken answer instead.&lt;/p&gt;

&lt;h2&gt;
  
  
  The gate is a function, not a vibe
&lt;/h2&gt;

&lt;p&gt;The gate decides "good enough to return." It's a schema check, an enum membership check, a length bound, or a cheap rule like "every document ID in the output exists in the retrieved set." Without one, you only escalate on exceptions — and degraded models rarely throw. They return confident nonsense that passes &lt;code&gt;if text:&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Show me the ladder
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;
python
import time, httpx

STATIC_FALLBACK = "I can't answer that
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>devops</category>
      <category>python</category>
    </item>
    <item>
      <title>The Agent Loop Is a Runtime: Budgets, Cancellation, and Idempotency</title>
      <dc:creator>skvrtyk</dc:creator>
      <pubDate>Sat, 31 Oct 2026 00:00:56 +0000</pubDate>
      <link>https://dev.to/dima_kruglov_608741d9a416/the-agent-loop-is-a-runtime-budgets-cancellation-and-idempotency-3heo</link>
      <guid>https://dev.to/dima_kruglov_608741d9a416/the-agent-loop-is-a-runtime-budgets-cancellation-and-idempotency-3heo</guid>
      <description>&lt;p&gt;You have tool calling working. The demo terminates, the JSON parses, the model picks &lt;code&gt;search_flights&lt;/code&gt; instead of &lt;code&gt;book_flight&lt;/code&gt;. Then it goes into production and the incidents are not about model quality at all. They are about the loop: a run that quietly spun for forty minutes, a refund issued twice, a cancel button that stopped the UI but not the worker, a bill that doubled because every step resends the entire history.&lt;/p&gt;

&lt;p&gt;Treat the loop as a runtime you own, with the same concerns as any scheduler: admission control, deadlines, cooperative cancellation, and at-most-once side effects. The model is a component inside that runtime, not the runtime itself.&lt;/p&gt;

&lt;h2&gt;
  
  
  Four budgets, not one
&lt;/h2&gt;

&lt;p&gt;An agent run needs at least four independent limits:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Steps&lt;/strong&gt; — the outer &lt;code&gt;for&lt;/code&gt; loop. Cheap to check, catches the obvious spin.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Wall clock&lt;/strong&gt; — a deadline, not a per-step timeout. Users care about the run, not the step.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tool calls&lt;/strong&gt; — separate from steps, because one step can fan out.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Money or tokens&lt;/strong&gt; — the one that actually stops a runaway, since cost per step grows with history.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The last point matters and is easy to miss: because you resend the conversation each step, per-step input tokens grow roughly linearly with step count, so total run cost grows faster than linearly. A step budget of 12 is not "12 cheap calls"; step 12 is the most expensive call in the run. Cap on cumulative tokens or dollars if you want a real ceiling.&lt;/p&gt;

&lt;p&gt;Which budget binds first is workload-dependent, and you should measure it rather than guess. Log steps-to-completion, tool-calls-to-completion and wall-clock-to-completion per task type, then set your caps near the p99 of &lt;em&gt;successful&lt;/em&gt; runs. If a large fraction of runs hit the cap, the task is out of scope for the agent — tightening or loosening the number will not fix it.&lt;/p&gt;

&lt;p&gt;The trade-off is honest: a hard cap converts a slow success into a fast failure. That is usually right for an interactive request and usually wrong for a batch job where you would rather finish in ten minutes than fail in ninety seconds. Make it a parameter, not a constant.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cancellation has to reach the tool
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;asyncio.Task.cancel()&lt;/code&gt;, a &lt;code&gt;threading.Event&lt;/code&gt;, a context flag — all of them cancel &lt;em&gt;your&lt;/em&gt; loop. None of them cancel &lt;code&gt;subprocess.run&lt;/code&gt; without a timeout, an HTTP client with no read timeout, or a Postgres query with no &lt;code&gt;statement_timeout&lt;/code&gt;. Your cancel becomes a lie the moment the run is inside a tool.&lt;/p&gt;

&lt;p&gt;Two things fix this. First, propagate the deadline, not the duration: each tool gets the &lt;em&gt;remaining&lt;/em&gt; time and computes its own timeouts from it. Second, every blocking call gets an explicit timeout, including the model call, and a timeout is a returned error to the model rather than a crash.&lt;/p&gt;

&lt;p&gt;And accept that after a cancel, the run is in an unknown state. Whatever the tool was doing may have half-completed. Do not pretend otherwise — put a reconciliation path behind anything that mutates.&lt;/p&gt;

&lt;h2&gt;
  
  
  Side effects need a key the model cannot choose
&lt;/h2&gt;

&lt;p&gt;Agents retry constantly. The model re-emits a tool call after a truncated response; your SDK retries a 500; a worker restarts mid-step. Any of these can replay a write. The fix is the same one you would apply to a webhook handler: an idempotency key derived deterministically from the call, stored in the same transaction as the effect, checked before execution.&lt;/p&gt;

&lt;p&gt;Derive it from &lt;code&gt;(run_id, tool_name, canonicalized_args)&lt;/code&gt; and hash it. Not from the model's prose, not from a timestamp, not from anything the model picks. For third-party APIs that do not support idempotency keys, record intent before the call and reconcile after — the outbox pattern, unchanged from the last decade of distributed systems.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Bounded agent runner: deadline, step budget, idempotent side effects.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;hashlib&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;

&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;BudgetExceeded&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;Exception&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="k"&gt;pass&lt;/span&gt;
&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Cancelled&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;Exception&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="k"&gt;pass&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;idem_key&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tool&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="c1"&gt;# Canonical JSON so key order can't produce two keys for the same call.
&lt;/span&gt;    &lt;span class="c1"&gt;# Never include a timestamp, a retry counter, or free-text from the model.
&lt;/span&gt;    &lt;span class="n"&gt;payload&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;run&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;run_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;tool&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;tool&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;args&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
                         &lt;span class="n"&gt;sort_keys&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;separators&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;,&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;hashlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sha256&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;()).&lt;/span&gt;&lt;span class="nf"&gt;hexdigest&lt;/span&gt;&lt;span class="p"&gt;()[:&lt;/span&gt;&lt;span class="mi"&gt;32&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;run_agent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;task&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;llm_step&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tools&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ledger&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;run_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
              &lt;span class="n"&gt;max_steps&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;deadline_s&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;90.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cancel&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;deadline&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;deadline_s&lt;/span&gt;
    &lt;span class="n"&gt;messages&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;role&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;user&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;content&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;task&lt;/span&gt;&lt;span class="p"&gt;}]&lt;/span&gt;

    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;step&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;max_steps&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;cancel&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
            &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;Cancelled&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;run &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;run_id&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; cancelled at step &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;step&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;remaining&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;deadline&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;remaining&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;BudgetExceeded&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;deadline hit at step &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;step&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

        &lt;span class="c1"&gt;# Reserve the final step for an answer. A loop that spends its whole
&lt;/span&gt;        &lt;span class="c1"&gt;# budget on tool calls returns nothing usable.
&lt;/span&gt;        &lt;span class="n"&gt;action&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;llm_step&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;remaining_s&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;remaining&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                          &lt;span class="n"&gt;force_final&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;step&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;max_steps&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;action&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;final&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;action&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;content&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

        &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;action&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;tool&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;action&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;args&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="n"&gt;key&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;idem_key&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;ledger&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;                      &lt;span class="c1"&gt;# replay: reuse recorded outcome
&lt;/span&gt;            &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;role&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;tool&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;content&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;ledger&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;]})&lt;/span&gt;
            &lt;span class="k"&gt;continue&lt;/span&gt;

        &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;                                   &lt;span class="c1"&gt;# tool gets REMAINING time
&lt;/span&gt;            &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;tools&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;](&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout_s&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;0.1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;deadline&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;()))&lt;/span&gt;
        &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nb"&gt;TimeoutError&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;exc&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;error&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; timed out: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;exc&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;   &lt;span class="c1"&gt;# feed back, don't raise
&lt;/span&gt;        &lt;span class="n"&gt;ledger&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;                   &lt;span class="c1"&gt;# same tx as the side effect
&lt;/span&gt;        &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;role&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;tool&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;content&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;)})&lt;/span&gt;

    &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;BudgetExceeded&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;step budget exhausted for run &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;run_id&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Wire &lt;code&gt;ledger&lt;/code&gt; to Redis or a table with the key as primary key, and make the write atomic with the effect where you can.&lt;/p&gt;

&lt;h2&gt;
  
  
  Traces that answer the on-call question
&lt;/h2&gt;

&lt;p&gt;The question at 2am is never "what was the prompt". It is "why did this run call &lt;code&gt;refund&lt;/code&gt; twice" and "where did the $4 go". One span per step, with: step index, model, input/output tokens, latency, tool name, arg hash, result hash, outcome. Log the hashes, not the payloads, in the hot path; put payloads in object storage keyed by the same hash.&lt;/p&gt;

&lt;p&gt;With that, duplicate side effects become a five-minute query: group spans by arg hash within a run and look for repeats.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pitfalls
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Retries multiplying across layers.&lt;/strong&gt; SDK retry × your HTTP retry × agent retry = up to 27 attempts on the same write. Pick one layer to retry and disable the others.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Per-step timeouts instead of a run deadline.&lt;/strong&gt; Twelve steps at 30s each is a six-minute request.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Catching &lt;code&gt;Exception&lt;/code&gt; (or &lt;code&gt;BaseException&lt;/code&gt;) around the loop.&lt;/strong&gt; Cancellation and deadline exits get swallowed and the loop continues.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Idempotency keys built from a timestamp or a UUID generated at call time.&lt;/strong&gt; Guarantees duplicates instead of preventing them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Unbounded message history.&lt;/strong&gt; Cost per step climbs, and at some point the tool outputs crowd out the instructions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cancel implemented only in the UI.&lt;/strong&gt; The worker keeps going, and so does the bill.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Monday morning checklist
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;[ ] Each run has an explicit &lt;code&gt;max_steps&lt;/code&gt;, wall-clock deadline, tool-call cap and token/dollar cap.&lt;/li&gt;
&lt;li&gt;[ ] The final step is reserved for an answer, not a tool call.&lt;/li&gt;
&lt;li&gt;[ ] Every blocking call — model, HTTP, subprocess, DB — has a timeout derived from the &lt;em&gt;remaining&lt;/em&gt; run budget.&lt;/li&gt;
&lt;li&gt;[ ] Cancellation is checked between steps and reaches in-flight tools.&lt;/li&gt;
&lt;li&gt;[ ] Every mutating tool call carries a deterministic&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>devops</category>
      <category>python</category>
    </item>
    <item>
      <title>Your Agent's Tool Layer Is a Public API. Design It Like One.</title>
      <dc:creator>skvrtyk</dc:creator>
      <pubDate>Fri, 30 Oct 2026 22:00:49 +0000</pubDate>
      <link>https://dev.to/dima_kruglov_608741d9a416/your-agents-tool-layer-is-a-public-api-design-it-like-one-57nl</link>
      <guid>https://dev.to/dima_kruglov_608741d9a416/your-agents-tool-layer-is-a-public-api-design-it-like-one-57nl</guid>
      <description>&lt;p&gt;The first agent I put in front of real users had eleven tools and a system prompt that said "ask for confirmation before doing anything destructive." It worked in staging for six weeks. On day two in production it issued the same refund three times because the payment API returned a 502, the model decided to retry, and my wrapper happily forwarded the retry.&lt;/p&gt;

&lt;p&gt;Nothing about that failure was an LLM problem. It was a missing idempotency key — the same bug you'd catch in code review on any HTTP client. The difference is that with an agent, the retry decision is made by a component you can't read, at a frequency you can't predict, with arguments you didn't hardcode.&lt;/p&gt;

&lt;p&gt;That's the framing I want to push: the model is a caller. Your tools are the API. Every property you demand of a public API — versioning, timeouts, idempotency, scoped credentials, rate limits — applies here, except the caller is adversarial-by-accident rather than adversarial-by-intent.&lt;/p&gt;

&lt;h2&gt;
  
  
  Classify tools by side effect, not by domain
&lt;/h2&gt;

&lt;p&gt;Most tool registries I've seen are grouped by vendor: &lt;code&gt;stripe.*&lt;/code&gt;, &lt;code&gt;github.*&lt;/code&gt;, &lt;code&gt;s3.*&lt;/code&gt;. That grouping tells you nothing about risk. What you actually need is a three-way classification on every tool:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;read&lt;/strong&gt; — no external state change. Safe to retry, safe to parallelize, safe to run in a loop.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;write&lt;/strong&gt; — creates or mutates state, but can be made idempotent. Needs a key.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;destructive&lt;/strong&gt; — deletes, revokes, sends to a human, moves money. Needs a key &lt;em&gt;and&lt;/em&gt; a gate that is not the model.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The classification isn't documentation, it's enforcement. &lt;code&gt;read&lt;/code&gt; tools are the only ones that get relaxed timeouts and unlimited retries. &lt;code&gt;destructive&lt;/code&gt; tools either require an out-of-band approval step or run only inside a dry-run harness. If you can't classify a tool into one of the three, you don't understand it well enough to expose it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Idempotency keys belong to the caller, not the tool
&lt;/h2&gt;

&lt;p&gt;The failure I opened with has a known fix, and the fix is not "tell the model not to retry." Models retry. They retry because a tool result got truncated, because the JSON didn't parse, because a 429 looked like a transient error, because you upgraded the model. Assume every tool call will be issued at least twice.&lt;/p&gt;

&lt;p&gt;So derive the key yourself, from the run and the arguments, and pass it downstream:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;hashlib&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;dataclasses&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;dataclass&lt;/span&gt;

&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Budget&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Hard stop per run. Not a metric — an exception.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;__init__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_calls&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_seconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;120&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;calls&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;max_calls&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;max_calls&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;deadline&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;max_seconds&lt;/span&gt;

    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;charge&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;calls&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;max_calls&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;deadline&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="c1"&gt;# Raise, don't return a string: the runtime converts it into a
&lt;/span&gt;            &lt;span class="c1"&gt;# tool error the model can see, and the loop terminates.
&lt;/span&gt;            &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;RuntimeError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;tool budget exhausted&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;calls&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;

&lt;span class="nd"&gt;@dataclass&lt;/span&gt;
&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;ToolSpec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;
    &lt;span class="n"&gt;fn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;callable&lt;/span&gt;
    &lt;span class="n"&gt;side_effect&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;          &lt;span class="c1"&gt;# "read" | "write" | "destructive"
&lt;/span&gt;    &lt;span class="n"&gt;timeout_s&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;5.0&lt;/span&gt;

&lt;span class="n"&gt;REGISTRY&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ToolSpec&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;tool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;side_effect&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout_s&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;5.0&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;wrap&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fn&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;REGISTRY&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ToolSpec&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fn&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;side_effect&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout_s&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fn&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;wrap&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_stable_key&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="c1"&gt;# Sorted JSON so dict ordering from the model can't change the key.
&lt;/span&gt;    &lt;span class="n"&gt;blob&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sort_keys&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;separators&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;,&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;hashlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sha256&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;run_id&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;:&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;:&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;blob&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;()).&lt;/span&gt;&lt;span class="nf"&gt;hexdigest&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="nd"&gt;@tool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;refund.create&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;side_effect&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;write&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout_s&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;8.0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;create_refund&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;order_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;amount_cents&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;idempotency_key&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="c1"&gt;# Downstream API dedupes on idempotency_key; a second call is a no-op.
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;payments&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;refund&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;order_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;amount_cents&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;idempotency_key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;idempotency_key&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;budget&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Budget&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;budget&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;charge&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;spec&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;REGISTRY&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;side_effect&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;read&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;idempotency_key&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;args&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;idempotency_key&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;_stable_key&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;)}&lt;/span&gt;
    &lt;span class="c1"&gt;# Production: run in a worker with spec.timeout_s and kill it on overrun.
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two details matter. The key is derived from &lt;code&gt;run_id&lt;/code&gt;, not from a timestamp, so a retry inside the same run is a genuine no-op — but a &lt;em&gt;new&lt;/em&gt; run gets a new key, which is what you want when a human re-asks. And the key is hashed from sorted JSON, so a model that emits &lt;code&gt;{"a":1,"b":2}&lt;/code&gt; then &lt;code&gt;{"b":2,"a":1}&lt;/code&gt; doesn't sneak past.&lt;/p&gt;

&lt;p&gt;The place this breaks: never let the model supply &lt;code&gt;idempotency_key&lt;/code&gt; itself. If it can choose the key, it can choose a new one on retry, and you're back to triple refunds with extra steps.&lt;/p&gt;

&lt;h2&gt;
  
  
  Budgets are a hard stop, not a dashboard
&lt;/h2&gt;

&lt;p&gt;Token budgets get all the attention because the invoice is visible. The budget that actually protects you is &lt;em&gt;actions per run&lt;/em&gt;. A tool call that takes 80ms and costs zero tokens can still send an email to a customer, file a legal ticket, or spin up a $4/hour VM. Agents in a loop will do this cheerfully — the failure mode isn't malice, it's a model that thinks the last call "didn't work" and tries a slightly different argument.&lt;/p&gt;

&lt;p&gt;Three counters, all enforced in the runtime, not in the prompt:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Tool calls per run&lt;/strong&gt; — I default to 20 for interactive agents, 100 for batch jobs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Wall clock per run&lt;/strong&gt; — a deadline checked before each call. Long tail latency compounds when the model fans out.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Mutating calls per run&lt;/strong&gt; — a separate, much smaller cap (often 1) for &lt;code&gt;destructive&lt;/code&gt; tools.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;When a budget trips, surface it to the model as a tool error and let it finish. Killing the process mid-generation loses the trace you need to debug it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Dry-run is a product feature
&lt;/h2&gt;

&lt;p&gt;The most useful thing you can build for a &lt;code&gt;destructive&lt;/code&gt; tool is a simulation mode where the tool returns the same shape of result without doing the thing. That's the same idea behind firmware simulators for agents that touch device state: you don't want to discover the argument schema was wrong on a real fleet.&lt;/p&gt;

&lt;p&gt;Dry-run is cheap for some tools (return a synthesized response) and expensive for others (you need a shadow database). The trade-off is worth stating explicitly per tool. What breaks is dry-run drift: the simulated path shares code with production except for one &lt;code&gt;if dry_run:&lt;/code&gt; early return, and six months later the simulated branch is missing a validation the real one added. If you can, make dry-run a &lt;em&gt;credential&lt;/em&gt; — point the tool at a sandbox endpoint with read-only keys rather than branching inside your code.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pitfalls
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Timeouts that cancel the wrapper, not the work.&lt;/strong&gt; &lt;code&gt;asyncio.wait_for&lt;/code&gt; on an HTTP call is fine; a thread doing &lt;code&gt;subprocess.run&lt;/code&gt; will keep going after you stop awaiting it. Kill the process group.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Truncated tool output.&lt;/strong&gt; If you clip a 40KB JSON response to 2KB, the model sees malformed JSON, retries, and you get duplicate writes. Return a summary plus a handle, not a truncated blob.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tool results as injection vector.&lt;/strong&gt; A fetched web page or a ticket body is untrusted input. If a &lt;code&gt;read&lt;/code&gt; tool's output can steer the model into a &lt;code&gt;write&lt;/code&gt; tool, your classification earned nothing. Strip or fence external text.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Per-run credentials that outlive the run.&lt;/strong&gt; Issue short-lived tokens per run; a service account with write access across the whole org is a blast radius you can't budget.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Counting retries as one call.&lt;/strong&gt; The whole point of the per-run cap is that the model's internal retry loop is invisible to you. Charge on every dispatch, including retries.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Monday checklist
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;[ ] Every registered tool has a &lt;code&gt;side_effect&lt;/code&gt; field, and the registry refuses to load a tool without one.&lt;/li&gt;
&lt;li&gt;[ ] &lt;code&gt;write&lt;/code&gt; and &lt;code&gt;destructive&lt;/code&gt; tools require an &lt;code&gt;idempotency_key&lt;/code&gt;; the runtime derives it from &lt;code&gt;run_id&lt;/code&gt; + tool + sorted args, never from the model.&lt;/li&gt;
&lt;li&gt;[ ] Per-run caps exist for tool calls, wall clock, and destructive actions, and they throw rather than log.&lt;/li&gt;
&lt;li&gt;[ ] Every tool has an explicit timeout, and the timeout cancels the underlying work.&lt;/li&gt;
&lt;li&gt;[ ] Credentials are per-run and short-lived; scope them to the smallest set of endpoints the tool actually calls.&lt;/li&gt;
&lt;li&gt;[ ] At least one &lt;code&gt;destructive&lt;/code&gt; tool has a dry-run path pointed at a sandbox endpoint, not an &lt;code&gt;if&lt;/code&gt; branch.&lt;/li&gt;
&lt;li&gt;[ ] Tool output is size-capped with a summary-plus-handle pattern, and external text is fenced before it reaches the model.&lt;/li&gt;
&lt;li&gt;[ ] You can answer "what stops this run from doing the thing twice?" without mentioning the system prompt.&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>devops</category>
      <category>python</category>
    </item>
    <item>
      <title>Treat a Model Upgrade Like a Schema Migration</title>
      <dc:creator>skvrtyk</dc:creator>
      <pubDate>Thu, 29 Oct 2026 04:51:18 +0000</pubDate>
      <link>https://dev.to/dima_kruglov_608741d9a416/treat-a-model-upgrade-like-a-schema-migration-49gm</link>
      <guid>https://dev.to/dima_kruglov_608741d9a416/treat-a-model-upgrade-like-a-schema-migration-49gm</guid>
      <description>&lt;p&gt;The most common LLMOps incident I've seen is not a prompt bug or a jailbreak. It's a one-line diff.&lt;/p&gt;

&lt;p&gt;Someone bumps &lt;code&gt;gpt-4o&lt;/code&gt; to &lt;code&gt;gpt-4o-2024-08-06&lt;/code&gt; (or &lt;code&gt;claude-3-5-sonnet-latest&lt;/code&gt; to a newer snapshot), CI passes because nothing asserts on model output shape, and three days later a downstream consumer crashes on &lt;code&gt;priority: null&lt;/code&gt; where it used to get &lt;code&gt;"normal"&lt;/code&gt;. Nobody changed the prompt. Nobody changed the code. The provider changed the weights, or the alias started pointing somewhere else.&lt;/p&gt;

&lt;p&gt;If you come from backend work, you already have the discipline for this: you don't swap a column type in production in one deploy. You expand, migrate, contract. Apply that to models.&lt;/p&gt;

&lt;h2&gt;
  
  
  The version string is a pointer, not a contract
&lt;/h2&gt;

&lt;p&gt;Most providers give you two things: a floating alias (&lt;code&gt;-latest&lt;/code&gt;, or a bare family name) and a dated snapshot. The alias is a pointer that moves. The snapshot is a pin that stays put until the provider retires it — and retirements are announced, scheduled, and then they happen whether or not you upgraded.&lt;/p&gt;

&lt;p&gt;So the pin belongs in config, not scattered across call sites:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;One registry mapping a logical task name (&lt;code&gt;extract-ticket&lt;/code&gt;, &lt;code&gt;summarize-thread&lt;/code&gt;) to a dated snapshot.&lt;/li&gt;
&lt;li&gt;The registry versioned in git, so a model change is a reviewed diff with a revert button.&lt;/li&gt;
&lt;li&gt;Prompt hash and output schema version in the same record. If you can't express "model X + prompt hash Y + schema v2" as a single deployable tuple, you don't have a rollback story.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The floating alias is fine in a notebook. In production it means a third party can ship a behavior change to you on a Tuesday.&lt;/p&gt;

&lt;h2&gt;
  
  
  Structured output is the schema, and it drifts
&lt;/h2&gt;

&lt;p&gt;If your call site does &lt;code&gt;json.loads(resp.content)&lt;/code&gt; and hands the dict downstream, the model's output is your API contract, with all the usual problems plus a few extra:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Enums grow.&lt;/strong&gt; A new snapshot adds &lt;code&gt;"urgent"&lt;/code&gt; to a field you validate as &lt;code&gt;low|normal|high&lt;/code&gt;. Strict schema validation turns that into a hard failure; lenient validation turns it into the wrong branch.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Nulls move.&lt;/strong&gt; &lt;code&gt;null&lt;/code&gt;, &lt;code&gt;"null"&lt;/code&gt;, &lt;code&gt;"N/A"&lt;/code&gt;, or a missing key. All four mean "absent", and each one breaks a different consumer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Numbers become strings.&lt;/strong&gt; &lt;code&gt;"3"&lt;/code&gt; instead of &lt;code&gt;3&lt;/code&gt;. Pydantic in lax mode coerces it, which hides the shift until something concatenates.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Refusals are valid.&lt;/strong&gt; A safety refusal is a 200 with prose. Your parser fails, your retry loop retries, and you pay for three refusals.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Truncation is silent unless you check &lt;code&gt;finish_reason&lt;/code&gt;.&lt;/strong&gt; A response cut off at &lt;code&gt;max_tokens&lt;/code&gt; can still be near-valid. Always assert &lt;code&gt;finish_reason == "stop"&lt;/code&gt; before parsing, and treat &lt;code&gt;"length"&lt;/code&gt; as an error, not a retry-forever.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The fix is the same expand/contract you'd use for a JSON API. During the expand phase, the reader accepts both shapes and logs which one it saw. Once the logs show the old shape is gone, you delete the branch.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# pins.py — the model version lives in exactly one place, and readers are tolerant.
&lt;/span&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;dataclasses&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;dataclass&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;hashlib&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;logging&lt;/span&gt;

&lt;span class="n"&gt;log&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getLogger&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;__name__&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="nd"&gt;@dataclass&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;frozen&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;ModelPin&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;task&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;            &lt;span class="c1"&gt;# logical name, e.g. "extract-ticket"
&lt;/span&gt;    &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;           &lt;span class="c1"&gt;# dated snapshot; never "-latest"
&lt;/span&gt;    &lt;span class="n"&gt;schema_version&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;
    &lt;span class="n"&gt;prompt_sha&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;

&lt;span class="n"&gt;PINS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;extract-ticket&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;ModelPin&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;extract-ticket&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gpt-4o-2024-08-06&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;9f3c1a&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;cache_key&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pin&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;ModelPin&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;user_input&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="c1"&gt;# Cache on the pin, not just the input. Otherwise a rollback serves
&lt;/span&gt;    &lt;span class="c1"&gt;# answers produced by the model you just rolled back from.
&lt;/span&gt;    &lt;span class="n"&gt;raw&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;pin&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;pin&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;schema_version&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;pin&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;prompt_sha&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;user_input&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;hashlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sha256&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;()).&lt;/span&gt;&lt;span class="nf"&gt;hexdigest&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;read_ticket&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;resp_content&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;finish_reason&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;finish_reason&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;stop&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;non-stop finish: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;finish_reason&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# "length" == truncated
&lt;/span&gt;    &lt;span class="n"&gt;payload&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;resp_content&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="c1"&gt;# Expand phase: accept v1 and v2, log which one showed up, contract later.
&lt;/span&gt;    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;schema_version&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;info&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ticket_schema_seen&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;extra&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;shape&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;v2&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;title&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;summary&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;priority&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;priority&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;level&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]}&lt;/span&gt;
    &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;info&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ticket_schema_seen&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;extra&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;shape&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;v1&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;title&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;title&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;priority&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;priority&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;normal&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ship the tolerant reader &lt;em&gt;before&lt;/em&gt; you change the pin. That ordering is the whole trick, and it's the step people skip on Friday afternoons.&lt;/p&gt;

&lt;h2&gt;
  
  
  Embeddings: never mix generations in one index
&lt;/h2&gt;

&lt;p&gt;This one is nastier than output drift because it fails silently and partially. Embedding models change dimensionality, normalization, and neighborhood structure between versions. Vectors from two generations are not comparable, even if the dimensions happen to match. Cosine similarity still returns a number, so your search still returns results — just wrong ones, with no error.&lt;/p&gt;

&lt;p&gt;Version the index in its name: &lt;code&gt;docs_v2_&amp;lt;model&amp;gt;_&amp;lt;dims&amp;gt;&lt;/code&gt;. Write new documents to both old and new during the backfill, flip reads when the backfill completes, then drop the old collection. Budget for the full re-embedding cost up front; for a large corpus that's the dominant cost of the upgrade, not the chat tokens.&lt;/p&gt;

&lt;h2&gt;
  
  
  Shadow traffic: measure the right thing
&lt;/h2&gt;

&lt;p&gt;Before the pin moves for real, dual-run. Send a sample of live (or replayed) inputs to the candidate and score both outputs. Two things matter here:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Sample size.&lt;/strong&gt; Small samples tell you nothing about a 2% change in failure rate. Decide the minimum sample before you look, or you'll stop at the first run that looks good.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What you measure.&lt;/strong&gt; Per-call latency is misleading. Measure tokens per &lt;em&gt;resolved task&lt;/em&gt;, because newer models often get more verbose — a cheaper per-token price with 40% more output tokens and more retries can be a net loss. Track task success, retry rate, output token ratio, and schema-violation rate together.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you use an LLM judge, calibrate it against a hand-labeled set first and report its agreement rate. An uncalibrated judge is a random number generator with good manners.&lt;/p&gt;

&lt;h2&gt;
  
  
  Rollback is not undo
&lt;/h2&gt;

&lt;p&gt;Flipping the pin back does not un-send the email, un-issue the refund, or un-create the Jira ticket. Model changes are usually paired with tool calls, and tool calls have side effects.&lt;/p&gt;

&lt;p&gt;Before you allow a pin change in a write path:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Idempotency keys on every side-effecting tool call, keyed on the logical operation, not the attempt.&lt;/li&gt;
&lt;li&gt;An outbox or dry-run mode you can route a candidate model into, so shadow traffic never writes.&lt;/li&gt;
&lt;li&gt;A kill switch that stops new work, separate from the pin, so you can freeze the queue while you decide.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Rolling back the model and rolling back the effects are two different runbooks. Write both.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pitfalls
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Caching on input only.&lt;/strong&gt; After a rollback, you serve the old model's cached answers and conclude the rollback didn't work.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Retrying on parse failures.&lt;/strong&gt; Refusals and truncations are deterministic-ish; three retries triple the bill and change nothing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Contracting too early.&lt;/strong&gt; Dropping the v1 reader the day the new pin ships guarantees an incident on the first straggler.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Testing against live providers in CI.&lt;/strong&gt; Flaky, slow, and it re-introduces the floating alias you just removed. Record fixtures from the pinned snapshot instead.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Only measuring the happy path.&lt;/strong&gt; The interesting deltas are in refusal rate, schema violations, and tool-argument validity.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Monday checklist
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;[ ] Every model call resolves through a single registry with dated snapshots. No &lt;code&gt;-latest&lt;/code&gt; outside notebooks.&lt;/li&gt;
&lt;li&gt;[ ] The registry records model, prompt hash, and schema version together, and it's in git.&lt;/li&gt;
&lt;li&gt;[ ] &lt;code&gt;finish_reason&lt;/code&gt; is asserted before parsing anywhere output is consumed.&lt;/li&gt;
&lt;li&gt;[ ] Output readers accept the previous schema version and log which shape they saw.&lt;/li&gt;
&lt;li&gt;[ ] Cache keys include the model pin and schema version.&lt;/li&gt;
&lt;li&gt;[ ] Embedding collections are named with model and dimension; re-embedding cost is estimated.&lt;/li&gt;
&lt;li&gt;[ ] Shadow traffic runs on a pre-declared sample size, with tokens-per-task and schema-violation rate in the report.&lt;/li&gt;
&lt;li&gt;[ ] Every side-effecting tool call has an idempotency key and a dry-run path.&lt;/li&gt;
&lt;li&gt;[ ] You have a written rollback runbook with two sections: revert the pin, repair the effects.&lt;/li&gt;
&lt;li&gt;[ ] CI runs contract tests against recorded fixtures from the pinned snapshot, not the live API.&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>devops</category>
      <category>python</category>
    </item>
    <item>
      <title>Your On-Device Model Is a Deploy Artifact, Not a Library</title>
      <dc:creator>skvrtyk</dc:creator>
      <pubDate>Tue, 27 Oct 2026 04:34:03 +0000</pubDate>
      <link>https://dev.to/dima_kruglov_608741d9a416/your-on-device-model-is-a-deploy-artifact-not-a-library-3cll</link>
      <guid>https://dev.to/dima_kruglov_608741d9a416/your-on-device-model-is-a-deploy-artifact-not-a-library-3cll</guid>
      <description>&lt;p&gt;Bundling a quantized model into an app is a weekend. Keeping 200,000 installs on a compatible &lt;code&gt;(app, runtime, weights)&lt;/code&gt; triple for a year is the actual work. The model is not a library you link once; it is a 1.5–2 GB artifact that ships on a schedule you don't control, to hardware you've never held, over networks that drop mid-transfer.&lt;/p&gt;

&lt;p&gt;I've watched more edge-AI incidents come from artifact lifecycle than from model quality.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>devops</category>
      <category>python</category>
    </item>
    <item>
      <title>Context Assembly Is a Build Step: Versioning the Prompt You Actually Send</title>
      <dc:creator>skvrtyk</dc:creator>
      <pubDate>Mon, 26 Oct 2026 00:01:04 +0000</pubDate>
      <link>https://dev.to/dima_kruglov_608741d9a416/context-assembly-is-a-build-step-versioning-the-prompt-you-actually-send-18od</link>
      <guid>https://dev.to/dima_kruglov_608741d9a416/context-assembly-is-a-build-step-versioning-the-prompt-you-actually-send-18od</guid>
      <description>&lt;p&gt;You review a 30-line template diff. Production sends 40,000 tokens assembled at request time from six sources, four of which changed this week and none of which appear in the pull request. The diff you approved is not the change you shipped.&lt;/p&gt;

&lt;p&gt;This is the part of LLMOps that keeps biting teams after they have evals, tracing, and a gateway in place. The template is versioned in git. The context is assembled by code paths nobody treats as release artifacts.&lt;/p&gt;

&lt;h2&gt;
  
  
  The prompt is a build output
&lt;/h2&gt;

&lt;p&gt;List what actually goes into a request for a typical RAG-plus-tools system:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The system template (in git, reviewed).&lt;/li&gt;
&lt;li&gt;Top-k retrieved chunks from an index that gets rebuilt on its own schedule.&lt;/li&gt;
&lt;li&gt;Tool JSON schemas, whose descriptions are prompt text written by whoever last touched the tool.&lt;/li&gt;
&lt;li&gt;A conversation summary produced by another model call.&lt;/li&gt;
&lt;li&gt;A user profile or tenant config blob.&lt;/li&gt;
&lt;li&gt;Few-shot examples chosen at runtime by a selector.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Each is a dependency with its own version and its own deploy cadence. The retrieved chunks change when someone reindexes. The tool schema changes when a backend engineer renames a parameter and rewrites the description. Neither triggers a review of the prompt.&lt;/p&gt;

&lt;p&gt;So when quality drops on Tuesday and no code shipped, there is no diff to look at. That is the failure this whole post is about.&lt;/p&gt;

&lt;h2&gt;
  
  
  Fingerprint everything the model sees
&lt;/h2&gt;

&lt;p&gt;Compute a content hash over the inputs that can change model behaviour, and log it with every trace:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;model id and version&lt;/li&gt;
&lt;li&gt;template version&lt;/li&gt;
&lt;li&gt;retrieval snapshot id (the index build, not the query)&lt;/li&gt;
&lt;li&gt;the fully rendered prompt&lt;/li&gt;
&lt;li&gt;decoding parameters that matter&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Store fingerprints always. Store full rendered prompts sampled or redacted, depending on your data rules — full-text retention is a compliance decision, not an observability one. The fingerprint is small, cheap, and answers "what changed" during an incident in one query.&lt;/p&gt;

&lt;p&gt;The trade-off: a fingerprint that includes a timestamp, a request id, or a per-user greeting has near-zero collision rate and zero diagnostic value. Hash the parts that affect generation, not the metadata around it. I have watched a team spend a day on "why is our prompt hash changing every request" when the answer was a locale string.&lt;/p&gt;

&lt;h2&gt;
  
  
  A small assembler with provenance and eviction
&lt;/h2&gt;

&lt;p&gt;You do not need a framework for this. You need three properties: every section carries a trust level, every section carries an eviction priority, and over-budget on a pinned section is a hard error rather than a silent trim.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;dataclasses&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;dataclass&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;field&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;hashlib&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;sha256&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;

&lt;span class="nd"&gt;@dataclass&lt;/span&gt;
&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Section&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;
    &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;
    &lt;span class="n"&gt;trusted&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;bool&lt;/span&gt;      &lt;span class="c1"&gt;# False for retrieval, tool output, user uploads
&lt;/span&gt;    &lt;span class="n"&gt;priority&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;      &lt;span class="c1"&gt;# higher survives truncation
&lt;/span&gt;    &lt;span class="n"&gt;pinned&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;bool&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;  &lt;span class="c1"&gt;# never evict; over budget means fail loudly
&lt;/span&gt;
&lt;span class="n"&gt;INSTRUCTION_LIKE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;(ignore (all )?(previous|prior) instructions|you are now|system prompt)&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;IGNORECASE&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;screen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="c1"&gt;# Cheap heuristic, not a guarantee. Log hits and route to a stricter path.
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;INSTRUCTION_LIKE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;[filtered]&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;count_tokens&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="c1"&gt;# Swap in tiktoken/the provider counter. One counter per model family.
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;//&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;

&lt;span class="nd"&gt;@dataclass&lt;/span&gt;
&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Assembled&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;
    &lt;span class="n"&gt;fingerprint&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;
    &lt;span class="n"&gt;dropped&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;default_factory&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;assemble&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sections&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;budget&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;model_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;template_version&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;snapshot_id&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;ordered&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sections&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pinned&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;priority&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;reverse&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;kept&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;dropped&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;used&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[],&lt;/span&gt; &lt;span class="p"&gt;[],&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;ordered&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;&amp;lt;&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; trust=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;high&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;trusted&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;low&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;&amp;gt;&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;trusted&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="nf"&gt;screen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;&amp;gt;&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="n"&gt;cost&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;count_tokens&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;used&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;cost&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;budget&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pinned&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pinned section &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; exceeds budget&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="n"&gt;dropped&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;continue&lt;/span&gt;
        &lt;span class="n"&gt;kept&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="n"&gt;used&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;cost&lt;/span&gt;
    &lt;span class="c1"&gt;# Render in original order so layout is stable across requests.
&lt;/span&gt;    &lt;span class="n"&gt;pos&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;id&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sections&lt;/span&gt;&lt;span class="p"&gt;)}&lt;/span&gt;
    &lt;span class="n"&gt;kept&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sort&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;pair&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;pos&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nf"&gt;id&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pair&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;])])&lt;/span&gt;
    &lt;span class="n"&gt;prompt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\n\n&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;kept&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sha256&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;part&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;template_version&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;snapshot_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;part&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nc"&gt;Assembled&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;hexdigest&lt;/span&gt;&lt;span class="p"&gt;()[:&lt;/span&gt;&lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;dropped&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Usage: build once per request, log the fingerprint beside the trace id.
&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;assemble&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;sections&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="nc"&gt;Section&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;question&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;why was my claim denied?&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;trusted&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;priority&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;pinned&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="nc"&gt;Section&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;policy_chunk&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;retrieved_text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;trusted&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;priority&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="nc"&gt;Section&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;profile&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tenant_blob&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;trusted&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;priority&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;],&lt;/span&gt;
    &lt;span class="n"&gt;budget&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;12_000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;model_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;claude-sonnet-4&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;template_version&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;support-v14&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;snapshot_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;index_build_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two details in there are load-bearing. Sections render in original insertion order rather than priority order, because reordering the same content changes model behaviour and makes your eval deltas unattributable. And &lt;code&gt;pinned&lt;/code&gt; turns budget pressure into an exception instead of a degraded&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>devops</category>
      <category>python</category>
    </item>
    <item>
      <title>Your Eval Suite Has a Flake Rate: Gating LLM Changes on Paired, Powered Tests</title>
      <dc:creator>skvrtyk</dc:creator>
      <pubDate>Sun, 25 Oct 2026 22:00:49 +0000</pubDate>
      <link>https://dev.to/dima_kruglov_608741d9a416/your-eval-suite-has-a-flake-rate-gating-llm-changes-on-paired-powered-tests-j7p</link>
      <guid>https://dev.to/dima_kruglov_608741d9a416/your-eval-suite-has-a-flake-rate-gating-llm-changes-on-paired-powered-tests-j7p</guid>
      <description>&lt;p&gt;I have watched the same release go wrong three times. Someone tunes a prompt or swaps a model, the eval average moves from 0.81 to 0.83, the PR gets approved, and two weeks later a specific intent is failing in production. The regression was real. The improvement was not. Nobody did anything careless — the eval reported a number that looked like a measurement and was actually a coin flip.&lt;/p&gt;

&lt;p&gt;The unglamorous part of LLMOps is not prompt management or tracing. It is that most eval suites cannot detect the changes teams ship on. Fixing that is a half-day of statistics and one CI job.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three kinds of noise, only one of which is your change
&lt;/h2&gt;

&lt;p&gt;Before you compare two configurations, know what is moving:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Example sampling noise.&lt;/strong&gt; Your golden set has 200 examples. Each one is worth 0.5 points. A "3-point gain" can be six examples flipping, several of which are borderline judgments.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Judge noise.&lt;/strong&gt; If an LLM grades the outputs and you left temperature above 0, the same output can pass on Tuesday and fail on Wednesday. Order matters too: ask a judge "which is better, A or B" and then "which is better, B or A" and you will find disagreement.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Task noise.&lt;/strong&gt; Sampling temperature, tool timeouts, and retrieval index drift all change outputs without any code change on your side.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;You cannot remove all three. You can remove the first one almost entirely by comparing &lt;strong&gt;paired&lt;/strong&gt; results: run baseline and candidate on the &lt;em&gt;same&lt;/em&gt; examples and analyze the per-example difference. Example difficulty cancels out. A prompt that fixes one hard case and breaks another now shows up as one +1 and one -1, not as an aggregate you have to reason about.&lt;/p&gt;

&lt;h2&gt;
  
  
  Do the arithmetic before you build the gate
&lt;/h2&gt;

&lt;p&gt;For a paired comparison of pass/fail outcomes, the standard error of the mean difference depends on the &lt;strong&gt;disagreement rate&lt;/strong&gt; — how often the two configs disagree on the same example — not on the accuracy.&lt;/p&gt;

&lt;p&gt;If 15% of examples flip between your two configurations and you have 200 paired examples:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;SE = sqrt(0.15 × 0.85 / 200) ≈ 2.5 percentage points&lt;/li&gt;
&lt;li&gt;Minimum detectable effect at 80% power, α=0.05 ≈ 2.8 × SE ≈ 7 points&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So on that setup, a 2-point bump is invisible. To see a 5-point change under the same assumptions you need roughly 400 paired examples; to see 3 points, about 1,100. These are arithmetic, not benchmarks — plug in your own disagreement rate, which is usually higher than you expect for prompt changes.&lt;/p&gt;

&lt;p&gt;Two consequences follow. First, stop gating on raw averages. Second, if your golden set is 200 examples and you cannot grow it, your gate should be "no regression beyond X" rather than "improvement required."&lt;/p&gt;

&lt;h2&gt;
  
  
  The gate itself
&lt;/h2&gt;

&lt;p&gt;The whole thing is about thirty lines of stdlib Python. Bootstrap the paired differences, look at the interval, and ship only when the interval clears zero.&lt;/p&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;
python
"""paired_gate.py — decide whether an LLM change is real or noise.
Run: python paired_gate.py    Deps: none (stdlib only)."""
import random

def paired_bootstrap(baseline, candidate, iters=5000, seed=7):
    """Scores aligned on the SAME example ids, 0/1 or 0..1.
    Returns (mean delta, 2.5th pct, 97.5th pct)."""
    diffs = [c - b for b, c in zip(baseline, candidate)]
    n = len(diffs)
    rng = random.Random(seed)  # fixed seed: the gate must be reproducible
    means = sorted(
        sum(diffs[rng.randrange(n)] for _ in range(n)) / n
        for _ in range(iters)
    )
    return sum(diffs) / n, means[int(0.025 * iters)], means[int(0.975 * iters)]

def required_n(disagreement_rate, mde, z_alpha=1.96, z_power=0.84):
    """Paired binary comparison: examples needed per side."""
    se = mde / (z_alpha + z_power)
    p = disagreement_rate
    return int((p * (1 - p)) / se**2) + 1

if __name__ == "__main__":
    # 200 paired examples; candidate looks "+3 points" better.
    baseline  = [1] * 160 + [0] * 40
    candidate = [1] * 166 + [0] * 34
    delta, lo, hi = paired_bootstrap(baseline, candidate)
    print(f"delta
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>devops</category>
      <category>python</category>
    </item>
    <item>
      <title>Replay or It Didn't Happen: Making Agent Runs Reproducible</title>
      <dc:creator>skvrtyk</dc:creator>
      <pubDate>Sun, 25 Oct 2026 08:24:09 +0000</pubDate>
      <link>https://dev.to/dima_kruglov_608741d9a416/replay-or-it-didnt-happen-making-agent-runs-reproducible-2lp2</link>
      <guid>https://dev.to/dima_kruglov_608741d9a416/replay-or-it-didnt-happen-making-agent-runs-reproducible-2lp2</guid>
      <description>&lt;p&gt;An agent fails in production. You have the transcript: the model's reasoning, the tool names, the arguments, the outputs. You read it and the bug is obvious — the agent called &lt;code&gt;search_orders&lt;/code&gt; with the wrong customer ID. You fix the prompt, redeploy, and the failure never reproduces. Six weeks later it happens again, with a different tool.&lt;/p&gt;

&lt;p&gt;The transcript told you what happened. It did not let you make it happen again. Those are different artifacts, and most agent stacks only ship the first one.&lt;/p&gt;

&lt;h2&gt;
  
  
  The transcript is not the trace
&lt;/h2&gt;

&lt;p&gt;The transcript is what the model said. The trace is everything the process observed: tool arguments before dispatch, raw responses before parsing, latency, retry counts, and the exact model/prompt/tool-schema versions in play. You need the second to reproduce, and you need it recorded at the boundary the model does not control.&lt;/p&gt;

&lt;p&gt;That boundary is the tool function. Record there and you get three things at once: a deterministic stub for replay, an input for offline evals, and a diffable record of what your agent actually did versus what the prompt claims it does.&lt;/p&gt;

&lt;h2&gt;
  
  
  Record at the call site, not in the framework
&lt;/h2&gt;

&lt;p&gt;Framework-level tracing (LangSmith, Langfuse, OpenTelemetry spans) is good for observability but usually records post-processed results. If your tool returns a dataclass and the recorder serializes it after your parser ran, you have lost the ability to test the parser.&lt;/p&gt;

&lt;p&gt;Here is a dependency-free record/replay layer. It is short enough to read in one sitting and complete enough to run.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# replay.py — record/replay at the tool boundary. No dependencies.
&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hashlib&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;functools&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;pathlib&lt;/span&gt;

&lt;span class="n"&gt;TRACE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;pathlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;traces/run.jsonl&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;MODE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;record&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;  &lt;span class="c1"&gt;# "record" | "replay" | "strict-replay"
&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_key&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tool_name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="c1"&gt;# sort_keys: dict ordering must not change the hash.
&lt;/span&gt;    &lt;span class="n"&gt;payload&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;t&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;tool_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;a&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="n"&gt;sort_keys&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;hashlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sha256&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;()).&lt;/span&gt;&lt;span class="nf"&gt;hexdigest&lt;/span&gt;&lt;span class="p"&gt;()[:&lt;/span&gt;&lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_load&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;TRACE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exists&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt;
    &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;TRACE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read_text&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;splitlines&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
    &lt;span class="c1"&gt;# Last write wins: a retried call records twice, replay uses the final result.
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;key&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;result&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;  &lt;span class="c1"&gt;# cache this per run in prod
&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;tool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fn&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="nd"&gt;@functools.wraps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fn&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;wrapper&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;kwargs&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;recorded&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;_key&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;__name__&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kwargs&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nf"&gt;_load&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;MODE&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;record&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;fn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;kwargs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;TRACE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;a&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;key&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;tool&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;fn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;__name__&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                                    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;args&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;kwargs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;result&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;

        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;recorded&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="c1"&gt;# Strict mode is the point: an unrecorded call means the agent
&lt;/span&gt;            &lt;span class="c1"&gt;# took a path your trace never covered. Fail loudly.
&lt;/span&gt;            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;MODE&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;strict-replay&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;RuntimeError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;unrecorded call: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;fn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;__name__&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;(&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;kwargs&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;)&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;fn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;kwargs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;recorded&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;wrapper&lt;/span&gt;

&lt;span class="nd"&gt;@tool&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;http_get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;urllib.request&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;  &lt;span class="c1"&gt;# real I/O; only reached in record mode
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Write a header line into &lt;code&gt;run.jsonl&lt;/code&gt; before any tool calls: prompt hash, model ID and version, tool schema hash, and a timestamp. Without it you cannot tell whether a replay divergence is a bug or a prompt change, and you will spend a day finding out the hard way.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three replay modes and what each one hides
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Record&lt;/strong&gt; executes for real. Everything downstream is a side effect you now own — see the pitfalls.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Strict replay&lt;/strong&gt; fails on any unrecorded call. This is what you want in CI: it tells you the agent explored a new branch, which is either the fix you shipped or the regression you just introduced.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Loose replay&lt;/strong&gt; falls through to live I/O for misses. Useful against staging, dangerous against production, and it makes tests silently non-deterministic when a cache expires. I have seen a "passing" eval suite that was quietly hitting a live API for 30% of its calls.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Default to strict. Opt into loose per test, with the miss count asserted.&lt;/p&gt;

&lt;h2&gt;
  
  
  Normalizing keys without lying
&lt;/h2&gt;

&lt;p&gt;The hash is only as good as the arguments. A few things break it:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Timestamps and UUIDs in args.&lt;/strong&gt; Agents love &lt;code&gt;since=datetime.now()&lt;/code&gt;. Every replay is a miss. Inject a frozen clock instead of stripping the field — stripping it hides the ordering bug you were trying to catch.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Unordered collections.&lt;/strong&gt; &lt;code&gt;json.dumps(sort_keys=True)&lt;/code&gt; sorts dict keys, not lists. Sort set-derived lists before they reach the tool.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Floats and serialization drift.&lt;/strong&gt; &lt;code&gt;0.1 + 0.2&lt;/code&gt; versus &lt;code&gt;0.3&lt;/code&gt; produces different hashes across languages and JSON encoders. Round at the tool signature, not in the recorder.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you normalize by deletion, write down what you deleted in the trace header. Six months later that comment is the only thing standing between you and a "why does replay pass but prod fail" afternoon.&lt;/p&gt;

&lt;h2&gt;
  
  
  What replay cannot fix
&lt;/h2&gt;

&lt;p&gt;Replay makes the &lt;em&gt;tool layer&lt;/em&gt; deterministic. It does nothing for the model. Even with &lt;code&gt;temperature=0&lt;/code&gt;, providers change weights, quantization, and routing behind a stable model name. Your recorded run is a snapshot of one sample from a distribution, and replaying it proves the code around the model still works — not that the model will behave the same tomorrow.&lt;/p&gt;

&lt;p&gt;That gap is why you still need fresh eval runs against the real model on a schedule, seeded with the traces that failed. Replay is the regression harness; live eval is the drift detector. Teams that run only one of them are surprised by a different class of bug every quarter.&lt;/p&gt;

&lt;p&gt;It also does not fix the world moving underneath you. A replayed fetch returns yesterday's page. That site may have since started blocking automated clients in its &lt;code&gt;robots.txt&lt;/code&gt; or terms — a real and growing constraint — and your replay will happily show green. Keep a small, scheduled live-canary set for anything touching third-party endpoints, and treat a replayed response as evidence about your code, never about the remote service.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pitfalls
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Recording side effects as if they were reads.&lt;/strong&gt; A replay that re-runs &lt;code&gt;send_email&lt;/code&gt; in loose mode is an incident, not a test. Split reads from writes and only replay reads; gate writes behind an explicit dry-run flag.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Traces as a PII landfill.&lt;/strong&gt; Raw tool args and responses contain customer data, tokens, and full documents. Redact at write time, set a retention window, and treat &lt;code&gt;traces/&lt;/code&gt; as production data with access control. A trace directory in a public repo is a breach.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Stale stubs masking real breakage.&lt;/strong&gt; A recorded 200 from an API you since migrated away from will keep passing. Version the trace alongside the tool schema hash and invalidate on change.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Unbounded trace files.&lt;/strong&gt; One JSONL per run, rotated, not one global append-only file that grows until &lt;code&gt;_load()&lt;/code&gt; takes 400ms per call.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Recording inside the sandbox without a policy.&lt;/strong&gt; If you run fleets, do it in a runtime where egress and filesystem writes are already constrained (NVIDIA's OpenShell is one example of that shape); a recorder that can write anywhere is a second problem, not a solution.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Monday checklist
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;[x] Trace written at the tool boundary, before parsing, not after.&lt;/li&gt;
&lt;li&gt;[x] Header line per run: prompt hash, model ID + version, tool schema hash.&lt;/li&gt;
&lt;li&gt;[x] &lt;code&gt;strict-replay&lt;/code&gt; is the default in CI; loose mode is opt-in and asserted.&lt;/li&gt;
&lt;li&gt;[x] Frozen clock and seeded IDs injected, not stripped from args.&lt;/li&gt;
&lt;li&gt;[x] Writes are separated from reads; no side-effecting tool is replayable.&lt;/li&gt;
&lt;li&gt;[x] Redaction applied at write time; retention window set; &lt;code&gt;traces/&lt;/code&gt; access-controlled.&lt;/li&gt;
&lt;li&gt;[x] Traces invalidated when the tool schema hash changes.&lt;/li&gt;
&lt;li&gt;[x] A scheduled live-eval run against the current model, seeded from failed traces.&lt;/li&gt;
&lt;li&gt;[x] A small live canary for third-party endpoints that can change their terms.&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>devops</category>
      <category>python</category>
    </item>
    <item>
      <title>Your Agent Is a Guest: Egress Policy for Agent Fleets</title>
      <dc:creator>skvrtyk</dc:creator>
      <pubDate>Sat, 24 Oct 2026 22:00:55 +0000</pubDate>
      <link>https://dev.to/dima_kruglov_608741d9a416/your-agent-is-a-guest-egress-policy-for-agent-fleets-2baa</link>
      <guid>https://dev.to/dima_kruglov_608741d9a416/your-agent-is-a-guest-egress-policy-for-agent-fleets-2baa</guid>
      <description>&lt;p&gt;Most agent code I review is careful about what it does &lt;em&gt;inside&lt;/em&gt; the sandbox and careless about what it does on the way out. Then it hits a site that returns 403 to anything with &lt;code&gt;python-httpx&lt;/code&gt; in the User-Agent, retries forty times, and burns an afternoon.&lt;/p&gt;

&lt;p&gt;The web is getting less tolerant of agents. Publishers are blocking crawlers outright, CDNs fingerprint TLS and headers, and more sites are moving to "pay for the API or go away." That's not a temporary mood — it's the steady state. If your fleet fetches anything from a domain you don't own, egress is a subsystem you have to design, not a &lt;code&gt;client.get()&lt;/code&gt; call you sprinkle through a tool.&lt;/p&gt;

&lt;p&gt;This is the view from the proxy layer: what breaks, what to log, and what to put in front of the agents.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two kinds of fan-out, one shared bottleneck
&lt;/h2&gt;

&lt;p&gt;There are two shapes of agent workload that both look fine in a notebook and both break at fleet scale:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Breadth-first research.&lt;/strong&gt; Thousands of agent instances each chase a slightly different query against public sources. OpenAI's biological-data work is the well-known version of this shape; the scaling problem is dedupe, not compute.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Depth-first tool use.&lt;/strong&gt; One agent loops over a task and calls ten external APIs per step. The scaling problem is that a single badly-behaved loop can generate more traffic than your whole test suite.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In both cases the failure is the same: every agent has its own HTTP client, its own retry policy, and no shared notion of "we have already asked this host this question." A per-agent rate limiter is not a rate limiter. It's a multiplier.&lt;/p&gt;

&lt;p&gt;Put egress in one place — a sidecar, a proxy process, or a single service the agents call — so the rate limiter, the &lt;code&gt;robots.txt&lt;/code&gt; cache, and the dedupe index are shared. Sandboxed runtimes for agent fleets, OpenShell being one of the recent entries, generally give you a network boundary to hang this on. Use the boundary; don't let agents dial out directly.&lt;/p&gt;

&lt;h2&gt;
  
  
  What a block actually looks like
&lt;/h2&gt;

&lt;p&gt;Not all rejections are 403, and the ones that aren't are worse:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Signal&lt;/th&gt;
&lt;th&gt;Meaning&lt;/th&gt;
&lt;th&gt;Right response&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;403&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Policy or fingerprint refusal&lt;/td&gt;
&lt;td&gt;Stop. Do not retry. Escalate to a human.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;429&lt;/code&gt; + &lt;code&gt;Retry-After&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Explicit backpressure&lt;/td&gt;
&lt;td&gt;Honor the header, then resume&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;200&lt;/code&gt; with a challenge page&lt;/td&gt;
&lt;td&gt;Soft block, invisible to status-code logic&lt;/td&gt;
&lt;td&gt;Detect and treat as failure&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Connection reset / TLS error&lt;/td&gt;
&lt;td&gt;Fingerprinting, or genuinely flaky&lt;/td&gt;
&lt;td&gt;Retry twice with jitter, then stop&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The soft block is the one that burns people. Your extractor returns zero rows, the agent concludes "no results," and writes a confident summary of an empty page. Check the parse yield per host — if a host that used to return 40 links per page starts returning 0 with a 200, that's a block, not a content change.&lt;/p&gt;

&lt;p&gt;Rotating IPs or spoofing a browser User-Agent to get past this is an arms race you will lose, and it's also a contract problem: you're evading a technical measure that expresses the operator's preference. The engineering answer is to buy the API, license the data, or drop the source.&lt;/p&gt;

&lt;h2&gt;
  
  
  A gate you can actually drop in
&lt;/h2&gt;

&lt;p&gt;Python 3.10+, &lt;code&gt;httpx&lt;/code&gt;. This is deliberately small: one class, one shared cache, fail-closed on unknown hosts.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;urllib.robotparser&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;collections&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;defaultdict&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;urllib.parse&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;urlparse&lt;/span&gt;

&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;httpx&lt;/span&gt;

&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;EgressGate&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Shared politeness + robots.txt gate. One instance per fleet, not per agent.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;

    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;__init__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;user_agent&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;min_interval_s&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ua&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;user_agent&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;min_interval&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;min_interval_s&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_last_hit&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;defaultdict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_robots&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;tuple&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;robotparser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;RobotFileParser&lt;/span&gt;&lt;span class="p"&gt;]]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;httpx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Client&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;User-Agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;user_agent&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;   &lt;span class="c1"&gt;# identify yourself, always
&lt;/span&gt;            &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;httpx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Timeout&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;10.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;5.0&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
            &lt;span class="n"&gt;follow_redirects&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_robots_for&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;robotparser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;RobotFileParser&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;host&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;urlparse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="n"&gt;netloc&lt;/span&gt;
        &lt;span class="n"&gt;cached&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_robots&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="c1"&gt;# Cache for an hour. Re-fetching robots.txt per request is itself abuse.
&lt;/span&gt;        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;cached&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;cached&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;3600&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;cached&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="n"&gt;rp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;robotparser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;RobotFileParser&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="n"&gt;rp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set_url&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/robots.txt&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;rp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nb"&gt;Exception&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;rp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;disallow_all&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;  &lt;span class="c1"&gt;# fail closed: unknown host, stay out
&lt;/span&gt;        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_robots&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;rp&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;rp&lt;/span&gt;

    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;httpx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Response&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;host&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;urlparse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="n"&gt;netloc&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;_robots_for&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;can_fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ua&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;  &lt;span class="c1"&gt;# policy skip, not an error - log it separately
&lt;/span&gt;
        &lt;span class="n"&gt;wait&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;min_interval&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_last_hit&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;wait&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;wait&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_last_hit&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
            &lt;span class="n"&gt;resp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status_code&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;429&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="c1"&gt;# Retry-After is sometimes an HTTP date; parse defensively in prod.
&lt;/span&gt;                &lt;span class="n"&gt;delay&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Retry-After&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
                &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;min&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;delay&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;60.0&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
                &lt;span class="k"&gt;continue&lt;/span&gt;
            &lt;span class="k"&gt;break&lt;/span&gt;

        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status_code&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;403&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;PermissionError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; refused agent traffic&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two things worth calling out. &lt;code&gt;min_interval&lt;/code&gt; is a &lt;em&gt;global&lt;/em&gt; per-host floor, so two agents racing on the same domain queue behind each other. And the 403 raise is deliberate: a 403 should propagate up and kill the sub-task, not get swallowed by a &lt;code&gt;try/except: pass&lt;/code&gt; three frames down.&lt;/p&gt;

&lt;h2&gt;
  
  
  Dedupe before you fan out
&lt;/h2&gt;

&lt;p&gt;If you're running a thousand agents against a fixed corpus, the cheapest win is not making the request twice. Hash the &lt;code&gt;(method, url, normalized_query)&lt;/code&gt; tuple, claim it in a shared store with an atomic &lt;code&gt;SETNX&lt;/code&gt;, and let the loser wait on the winner's result rather than re-deriving it. The failure mode without this is ugly: 2,000 agents independently discovering the same 40 pages, then 2,000 polite rate limiters all hammering the same host at one request per second.&lt;/p&gt;

&lt;p&gt;Store the fetched payload keyed by URL and content hash, with a TTL tied to how volatile the source is. Don't store bodies you aren't licensed to retain — that's a policy decision your gate should be able to enforce by host, not a judgement call at extraction time.&lt;/p&gt;

&lt;h2&gt;
  
  
  Log the decision, not just the response
&lt;/h2&gt;

&lt;p&gt;Per fetch, emit: host, status, bytes, latency, the policy decision (&lt;code&gt;allowed&lt;/code&gt; / &lt;code&gt;robots_skip&lt;/code&gt; / &lt;code&gt;blocked&lt;/code&gt;), the dedupe key, and the agent ID that asked. Then alert on two ratios: policy-skip rate per host, and parse yield per host. Both going wrong at once is the signature of a soft block. Neither appears in a status-code dashboard.&lt;/p&gt;

&lt;p&gt;Keep the agent ID. When someone asks why your fleet hit their site 8,000 times, "which agent, doing what, under which task" is the only answer that ends the conversation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pitfalls
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Fetching &lt;code&gt;robots.txt&lt;/code&gt; inside the request path.&lt;/strong&gt; You've turned one request into two and added a per-request dependency on a host that may rate-limit it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A rate limiter per agent process.&lt;/strong&gt; Ten agents, ten limiters, ten times the intended rate.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Retrying 403.&lt;/strong&gt; It will not become a 200, and now you've doubled the signal that you're a problem.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treating &lt;code&gt;200&lt;/code&gt; as success.&lt;/strong&gt; Challenge pages, consent walls, and JS shells all return 200 with no content.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ignoring &lt;code&gt;Retry-After&lt;/code&gt; and using fixed 1s backoff.&lt;/strong&gt; Fixed backoff synchronizes your retries into a thundering herd; add jitter.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rotating egress IPs as a fix.&lt;/strong&gt; It works for a week and converts a rate-limit problem into a legal one.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One User-Agent string for everything.&lt;/strong&gt; You lose the ability to explain your traffic, and you can't be allowlisted.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Monday checklist
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;[ ] All agent HTTP traffic goes through one egress component; agents have no direct network access.&lt;/li&gt;
&lt;li&gt;[ ] &lt;code&gt;robots.txt&lt;/code&gt; is fetched once per host and cached with&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>devops</category>
      <category>python</category>
    </item>
    <item>
      <title>Audio Deadlines: Running On-Device TTS Without Audible Glitches</title>
      <dc:creator>skvrtyk</dc:creator>
      <pubDate>Sat, 24 Oct 2026 00:01:22 +0000</pubDate>
      <link>https://dev.to/dima_kruglov_608741d9a416/audio-deadlines-running-on-device-tts-without-audible-glitches-kdi</link>
      <guid>https://dev.to/dima_kruglov_608741d9a416/audio-deadlines-running-on-device-tts-without-audible-glitches-kdi</guid>
      <description>&lt;p&gt;Gradium open-sourced Phonon, its on-device TTS engine (&lt;a href="https://gradium.ai/phonon" rel="noopener noreferrer"&gt;https://gradium.ai/phonon&lt;/a&gt;). That matters less because of the model than because of what it signals: on-device speech synthesis is now a dependency you can ship rather than a research demo. And a wave of teams is about to find out that the hard part of edge speech is not the model. It is that audio is the one component in your app with a hard real-time deadline, and nothing else in your background trained you for that.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two clocks, and only one of them forgives you
&lt;/h2&gt;

&lt;p&gt;A web request can take 900 ms and nobody dies. An audio callback cannot take 21 ms when it had 20. Once playback starts, the DAC consumes samples at a fixed rate forever, and your synthesis loop has to stay ahead of it. Miss by enough and the buffer underruns: you get a click, a pop, or a robotic stutter that users describe as "the app is broken".&lt;/p&gt;

&lt;p&gt;Do the arithmetic once and keep it on the wall. At 24 kHz with a 20 ms mixer period, each callback needs 480 samples. Your synthesis loop is not measured in requests per second, it is measured in whether it produces 480 samples every 20 ms, including while the CPU is thermally throttled and the OS is doing a background sync.&lt;/p&gt;

&lt;p&gt;This changes what you optimize. P50 latency is nearly irrelevant. What matters:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Worst-case chunk time&lt;/strong&gt;, because one slow chunk is an audible artifact.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sustained real-time factor over minutes&lt;/strong&gt;, not seconds, because governors ramp down.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Slack&lt;/strong&gt;, the gap between your worst chunk and the buffer you pre-rolled.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Real-time factor is a mean, and means lie
&lt;/h2&gt;

&lt;p&gt;RTF — synthesis wall time divided by audio duration produced — is the number everyone quotes. RTF 0.3 sounds great. It tells you the engine can run 3x faster than playback on a cold device with the screen on and nothing else running. That is not your user's device.&lt;/p&gt;

&lt;p&gt;I have watched an engine hold RTF 0.55 for 60 seconds and drift to 1.1 by minute four, because the SoC dropped from the big cluster to the little cluster and stayed there. RTF measured over 5 seconds would have shipped that bug.&lt;/p&gt;

&lt;p&gt;So measure the distribution and measure it long:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;Deadline harness for an on-device TTS engine.

Reports whether the engine can keep the audio clock fed, not how fast it
is on average. Run it for 10+ minutes on a real device, not a laptop.
&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;statistics&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;dataclasses&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;dataclass&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;field&lt;/span&gt;

&lt;span class="n"&gt;SAMPLE_RATE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;24_000&lt;/span&gt;                    &lt;span class="c1"&gt;# engine output rate
&lt;/span&gt;&lt;span class="n"&gt;BLOCK_MS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;                           &lt;span class="c1"&gt;# one mixer/callback period
&lt;/span&gt;&lt;span class="n"&gt;BLOCK_SAMPLES&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;SAMPLE_RATE&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;BLOCK_MS&lt;/span&gt; &lt;span class="o"&gt;//&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;   &lt;span class="c1"&gt;# 480
&lt;/span&gt;
&lt;span class="nd"&gt;@dataclass&lt;/span&gt;
&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Run&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;chunk_ms&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;default_factory&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# wall time per chunk
&lt;/span&gt;    &lt;span class="n"&gt;audio_ms&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;float&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;default_factory&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# audio produced per chunk
&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;measure&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;synth&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;texts&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;warmup&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="c1"&gt;# Warm up: page in weights, let the JIT/kernel selection settle.
&lt;/span&gt;    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;texts&lt;/span&gt;&lt;span class="p"&gt;[:&lt;/span&gt;&lt;span class="n"&gt;warmup&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
        &lt;span class="nf"&gt;synth&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Run&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;texts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;t0&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;perf_counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="n"&gt;pcm&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;synth&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;                                &lt;span class="c1"&gt;# mono float32 or int16
&lt;/span&gt;        &lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;chunk_ms&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;perf_counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;t0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;audio_ms&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pcm&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;SAMPLE_RATE&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;run&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;report&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Run&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;prebuffer_ms&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;400&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;rtf&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;chunk_ms&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;audio_ms&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;p99&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;statistics&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;quantiles&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;chunk_ms&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="mi"&gt;98&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;worst&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;chunk_ms&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="c1"&gt;# Conservative bound for a stop-and-wait loop: a chunk longer than the
&lt;/span&gt;    &lt;span class="c1"&gt;# pre-roll plus one block drains the buffer before the next chunk lands.
&lt;/span&gt;    &lt;span class="n"&gt;slack&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;prebuffer_ms&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;BLOCK_MS&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;worst&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;RTF (mean)            &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;rtf&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;   # &amp;lt;1.0 is necessary, not sufficient&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;chunk p99             &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;p99&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; ms&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;worst chunk           &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;worst&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; ms&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;slack after prebuffer &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;slack&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; ms  &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;OK&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;slack&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;UNDERRUN&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;rtf&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;p99&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;worst&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If your engine pipelines synthesis and playback, the slack bound is pessimistic — that is fine, it is a bound. What you must not do is report RTF alone and call it green.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prebuffer is your jitter absorber, and it costs you responsiveness
&lt;/h2&gt;

&lt;p&gt;You cannot fix jitter with a faster model alone; you fix it with buffer. Pre-roll 300–500 ms of audio before you start the output stream, and that headroom absorbs a late chunk without an underrun.&lt;/p&gt;

&lt;p&gt;The trade-off is real and it is user-facing. In a voice assistant, prebuffer is perceived as the delay between the user finishing a sentence and the assistant starting to speak. In a screen reader, it is a lag on every cursor move. There is no universal right answer, so pick per surface:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Conversational&lt;/strong&gt;: 150–250 ms prebuffer, tighter model, accept occasional degradation.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Narration / media&lt;/strong&gt;: 400–800 ms, and nobody notices.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Accessibility&lt;/strong&gt;: prebuffer short utterances fully before starting; a 3-word phrase is cheaper to synthesize whole than to stream and stitch.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The seam problem
&lt;/h2&gt;

&lt;p&gt;Streaming TTS means chunking text and chunking audio, and both create seams. Sentence-level text chunking changes prosody across the join; frame-level vocoder chunking produces a discontinuity at the boundary unless you overlap and crossfade. A constant-power crossfade of 5–10 ms hides most of it. Under 3 ms you will hear a tick on plosives; over 20 ms you smear consonants.&lt;/p&gt;

&lt;p&gt;Two failure modes I have seen:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Chunk boundary lands mid-word.&lt;/strong&gt; Text splitter respects sentence boundaries in English, then a user types "e.g." or a German compound and you cut inside a token. The engine pronounces half a word with full confidence.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Crossfade applied to silence.&lt;/strong&gt; Overlap-add on a chunk that ends in a natural pause attenuates the pause and makes the voice sound rushed. Gate the crossfade on frame energy, or normalize at the frame level.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Quantization changes prosody before it changes intelligibility
&lt;/h2&gt;

&lt;p&gt;Int8 quantization on the acoustic model and vocoder usually preserves intelligibility longer than it preserves naturalness. WER measured through an ASR round-trip will stay flat while the voice starts sounding flat and slightly buzzy — and users notice that immediately even though your metric does not.&lt;/p&gt;

&lt;p&gt;What I measure instead:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;F0 contour drift&lt;/strong&gt; against a float32 reference on a fixed 50-sentence set. Mean absolute deviation in cents over voiced frames.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Spectral distance&lt;/strong&gt; on sustained vowels, which is where buzz shows up.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A/B on device speakers&lt;/strong&gt;, because the phone's tiny driver hides artifacts that earbuds expose. Test both.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Also: thread count is not free performance. Four threads finish a chunk faster but draw more power and heat the SoC, which is what caused the RTF drift in the first place. On a phone, 2 threads plus a larger prebuffer often beats 4 threads plus a small one over a 10-minute session.&lt;/p&gt;

&lt;h2&gt;
  
  
  Platform details that bite
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Android&lt;/strong&gt;: &lt;code&gt;AudioTrack&lt;/code&gt; has a low-latency path (&lt;code&gt;PERFORMANCE_MODE_LOW_LATENCY&lt;/code&gt;) but it wants the device native sample rate. Synthesizing at 24 kHz and letting the framework resample to 48 kHz adds latency and a filter you did not choose. Match the device rate or resample yourself with a known-quality filter.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;iOS&lt;/strong&gt;: &lt;code&gt;AVAudioSession&lt;/code&gt; category and mode decide whether you get the low-latency route. &lt;code&gt;playback&lt;/code&gt; with &lt;code&gt;modeDefault&lt;/code&gt; is not the same as &lt;code&gt;voicePrompt&lt;/code&gt;; test on hardware, not the simulator.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Embedded Linux&lt;/strong&gt;: ALSA period size is your callback period. Set &lt;code&gt;period_size&lt;/code&gt; explicitly and check &lt;code&gt;snd_pcm_sw_params&lt;/code&gt; for start thresholds; defaults vary by board and are frequently wrong.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Page faults&lt;/strong&gt;: memory-mapping multi-hundred-megabyte weights is correct, but the first utterance after a cold start can fault in enough pages to miss deadlines. Either warm the model at app start or &lt;code&gt;mlock&lt;/code&gt; the hot region.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Fallbacks that don't sound like fallbacks
&lt;/h2&gt;

&lt;p&gt;Decide your degradation ladder before launch, not during an incident:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Full-quality on-device voice.&lt;/li&gt;
&lt;li&gt;Lower sample rate (48k → 24k) — barely perceptible, halves vocoder cost.&lt;/li&gt;
&lt;li&gt;Shorter utterances with more prebuffer per chunk.&lt;/li&gt;
&lt;li&gt;Server TTS, only if the network is up and the user is not on a metered connection.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Never switch voices mid-sentence. A voice change at a sentence boundary is a quirk; a voice change mid-clause sounds like corruption.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pitfalls
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Reporting mean RTF and calling the engine "faster than real time".&lt;/li&gt;
&lt;li&gt;Benchmarking on a laptop or an emulator. Both lie about thermals and audio routing.&lt;/li&gt;
&lt;li&gt;Testing with a single short sentence. Long utterances keep the CPU hot&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>devops</category>
      <category>python</category>
    </item>
    <item>
      <title>Your Model Is a Binary You Ship to Strangers: Release Engineering for Edge AI</title>
      <dc:creator>skvrtyk</dc:creator>
      <pubDate>Fri, 23 Oct 2026 22:00:57 +0000</pubDate>
      <link>https://dev.to/dima_kruglov_608741d9a416/your-model-is-a-binary-you-ship-to-strangers-release-engineering-for-edge-ai-1nn3</link>
      <guid>https://dev.to/dima_kruglov_608741d9a416/your-model-is-a-binary-you-ship-to-strangers-release-engineering-for-edge-ai-1nn3</guid>
      <description>&lt;p&gt;Gradium open-sourcing its on-device TTS (&lt;a href="https://gradium.ai/phonon" rel="noopener noreferrer"&gt;Phonon&lt;/a&gt;) is a useful reminder of what edge AI actually is from an operations standpoint. It isn't a model you deploy. It's a binary you ship to hardware you don't own, running an OS you don't patch, on a thermal budget you can't see. Your deploy button doesn't exist. There is no &lt;code&gt;kubectl rollout undo&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The engineers who struggle here are rarely struggling with inference. They're struggling with the fact that the artifact is the release, the rollout is a slow-motion migration across a heterogeneous fleet, and the failure signal arrives weeks later, aggregated, and only if the user opted in.&lt;/p&gt;

&lt;h2&gt;
  
  
  The release unit is the bundle, not the weights
&lt;/h2&gt;

&lt;p&gt;Ship the tokenizer with the weights or you will ship a bug. The classic edge regression is a model that produces subtly wrong output because the text normalization rules changed between the version that trained the tokenizer and the version that runs on device. Same weights, different preprocessing, different audio. Nothing crashes. Your evals, run against a Python pipeline, are green.&lt;/p&gt;

&lt;p&gt;So content-address everything and hash the manifest:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# bundle_manifest.py — build and verify one edge model release unit.
# Run in CI at packaging time and again on device at load time.
&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;hashlib&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;pathlib&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;sha256&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;pathlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;hashlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sha256&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rb&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;chunk&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;iter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sa"&gt;b&lt;/span&gt;&lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
            &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;chunk&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;hexdigest&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;build&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bundle_dir&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;pathlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;model_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;runtime_min&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="c1"&gt;# Tokenizer, config, vocoder and weights all live in one directory.
&lt;/span&gt;    &lt;span class="n"&gt;files&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;sha256&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bundle_dir&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iterdir&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;is_file&lt;/span&gt;&lt;span class="p"&gt;()}&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;model_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;model_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;           &lt;span class="c1"&gt;# bump on any numeric change, incl. quant level
&lt;/span&gt;        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;runtime_min&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;runtime_min&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;     &lt;span class="c1"&gt;# e.g. "1.17.0" for an opset-19 ONNX graph
&lt;/span&gt;        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;files&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;files&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;bundle&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;hashlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sha256&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;files&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sort_keys&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;hexdigest&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;verify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bundle_dir&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;pathlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;manifest&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;expected&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;manifest&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;files&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
        &lt;span class="n"&gt;actual&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sha256&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bundle_dir&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;actual&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="n"&gt;expected&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;RuntimeError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;: expected &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;expected&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;, got &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;actual&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two details matter more than they look. First, &lt;code&gt;model_id&lt;/code&gt; must change when the quantization level changes — int8 vs fp16 is a different model, not a build flag. Second, verification runs at load time, not just at download time, because partial writes and aggressive storage cleaners both exist.&lt;/p&gt;

&lt;h2&gt;
  
  
  Compatibility is a matrix you own, not a promise you make
&lt;/h2&gt;

&lt;p&gt;"It ran on my Pixel" is not a compatibility statement. The things that actually break:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;ONNX opset vs. onnxruntime version. Opset 19 needs ORT ≥ 1.17.&lt;/li&gt;
&lt;li&gt;Core ML &lt;code&gt;deploymentTarget&lt;/code&gt; vs. minimum iOS version you claim to support.&lt;/li&gt;
&lt;li&gt;NNAPI/QNN delegates: the same graph partitions differently per SoC, and a delegate that silently falls back to CPU can turn a 40 ms frame into a 400 ms frame.&lt;/li&gt;
&lt;li&gt;WebGPU vs. WASM fallback in browser targets, which is a 5–10x throughput difference in my experience, not a rounding error.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Declare a &lt;code&gt;runtime_min&lt;/code&gt; and refuse to load below it. A clean refusal you can measure beats a delegate fallback you can't see.&lt;/p&gt;

&lt;h2&gt;
  
  
  Stage by cohort, not by percentage
&lt;/h2&gt;

&lt;p&gt;Percentage rollouts work when every server is identical. Devices aren't. A 5% rollout that happens to land on one SoC family tells you nothing about the fleet, and a 5% rollout that lands on your most common device tells you everything too late.&lt;/p&gt;

&lt;p&gt;Cohort on the axes that predict failure: SoC family, OS major version, RAM class, free disk. Then gate promotion on guardrails that run automatically:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# rollout.py — decide whether a cohort advances. Runs on your backend,
# against telemetry the client sends back in batches.
&lt;/span&gt;&lt;span class="n"&gt;GUARDRAILS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;load_failure_rate&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.005&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;    &lt;span class="c1"&gt;# fraction of loads that raise
&lt;/span&gt;    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cold_start_p95_ms&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;first_chunk_p95_ms&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;250&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;     &lt;span class="c1"&gt;# time-to-first-audio, not RTF
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;should_promote&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;metrics&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;min_samples&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;500&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;tuple&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;metrics&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;samples&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;min_samples&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="c1"&gt;# Small samples produce confident nonsense. Hold.
&lt;/span&gt;        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;insufficient_samples&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ceiling&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;GUARDRAILS&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;metrics&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;ceiling&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;metrics&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; &amp;gt; &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;ceiling&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ok&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;min_samples&lt;/code&gt; check is the one people skip. A new bundle on a rare SoC with 40 samples and zero failures is not evidence of anything.&lt;/p&gt;

&lt;h2&gt;
  
  
  The latency budget has three numbers
&lt;/h2&gt;

&lt;p&gt;Steady-state real-time factor is the number everyone reports and the least useful one. What users feel:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Cold start.&lt;/strong&gt; Loading weights plus delegate compilation. On some Android devices this is over a second and it happens on first use, not app launch, unless you warm it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Time to first chunk.&lt;/strong&gt; For streaming TTS this dominates perceived quality. A model with a better RTF and worse first-chunk latency feels worse.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sustained throughput.&lt;/strong&gt; Measure after 60–90 seconds of continuous synthesis. Thermal throttling is real on phones and invisible in a 5-second benchmark.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Log all three, tagged with the bundle hash and device class. If you can only keep one, keep time to first chunk.&lt;/p&gt;

&lt;h2&gt;
  
  
  Fallbacks must ship, not be configured
&lt;/h2&gt;

&lt;p&gt;The fallback ladder for on-device TTS is roughly: large model → small model → CPU-only path → server round-trip. Every rung has to be in the binary, because the moment you need rung four, the network is probably bad and the user is already annoyed.&lt;/p&gt;

&lt;p&gt;Which means the CPU path needs CI coverage. Untested fallbacks rot. I've watched a "graceful degradation" path fail in production because nobody had run it since the delegate landed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Rollback is a disk-space decision
&lt;/h2&gt;

&lt;p&gt;You cannot fix a bad model with a download. If the bundle is broken, the download is broken too. What you need:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The N-1 bundle still on disk, with an atomic pointer switch.&lt;/li&gt;
&lt;li&gt;A signed remote config flag that disables the new bundle without fetching anything.&lt;/li&gt;
&lt;li&gt;A migration story for the on-disk format, because the old bundle has to remain loadable by the new app version.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If your rollback requires a 200 MB download over a metered connection, you don't have a rollback.&lt;/p&gt;

&lt;h2&gt;
  
  
  Telemetry you can actually get back
&lt;/h2&gt;

&lt;p&gt;You get counts and histograms, sampled, opt-in, batched. Not traces. Send: bundle hash, device class, load success/failure with an error code, cold start ms, first-chunk ms, and a coarse quality signal (user skipped the output, retried, or switched voices). Don't send the text. Don't send the audio.&lt;/p&gt;

&lt;p&gt;Set your batching interval by battery, not by convenience. A client that phones home every 30 seconds will get uninstalled, and rightly.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pitfalls
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Hashing only the weights, then shipping a tokenizer change under the same version.&lt;/li&gt;
&lt;li&gt;Testing on one device per OS, not one device per SoC family.&lt;/li&gt;
&lt;li&gt;Promoting on a metric with 40 samples.&lt;/li&gt;
&lt;li&gt;Benchmarking for 5 seconds and calling it throughput.&lt;/li&gt;
&lt;li&gt;Fallback paths with no CI coverage.&lt;/li&gt;
&lt;li&gt;Rollback that depends on the network.&lt;/li&gt;
&lt;li&gt;Treating quantization as a build flag rather than a model version.&lt;/li&gt;
&lt;li&gt;Assuming opt-in telemetry will reach you from the cohort that's failing — it often won't, because those users churn first.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Monday morning checklist
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;[ ] Every release artifact is content-addressed and the manifest is hashed as a unit.&lt;/li&gt;
&lt;li&gt;[ ] &lt;code&gt;model_id&lt;/code&gt; changes on quantization, tokenizer, or preprocessing changes.&lt;/li&gt;
&lt;li&gt;[ ] The runtime declares a minimum version and refuses to load below it.&lt;/li&gt;
&lt;li&gt;[ ] Cohorts are defined by SoC, OS major, RAM class, and free disk.&lt;/li&gt;
&lt;li&gt;[ ] Promotion is automatic, gated on guardrails, with a &lt;code&gt;min_samples&lt;/code&gt; floor.&lt;/li&gt;
&lt;li&gt;[ ] You log cold start, time to first chunk, and 90-second sustained throughput.&lt;/li&gt;
&lt;li&gt;[ ] Every fallback rung has a CI test that exercises it.&lt;/li&gt;
&lt;li&gt;[ ] N-1 bundle stays on disk and can be selected without a download.&lt;/li&gt;
&lt;li&gt;[ ] A signed config flag can disable the new bundle remotely.&lt;/li&gt;
&lt;li&gt;[ ] Telemetry is sampled, batched, opt-in, and carries the bundle hash.&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>devops</category>
      <category>python</category>
    </item>
    <item>
      <title>LLMOps Is Supply Chain Work: Pinning the Inputs You Don't Control</title>
      <dc:creator>skvrtyk</dc:creator>
      <pubDate>Fri, 23 Oct 2026 10:32:34 +0000</pubDate>
      <link>https://dev.to/dima_kruglov_608741d9a416/llmops-is-supply-chain-work-pinning-the-inputs-you-dont-control-10dj</link>
      <guid>https://dev.to/dima_kruglov_608741d9a416/llmops-is-supply-chain-work-pinning-the-inputs-you-dont-control-10dj</guid>
      <description>&lt;p&gt;&lt;em&gt;Suggested tags: ai, llm, devops, python&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Most LLMOps advice is about the model: pick a snapshot, set a temperature, watch the latency. That framing is comfortable because the model is the one thing you can't debug. The failures I've been pulled into were rarely the model. They were the corpus it read, the prompt template someone edited at 4pm on Friday, and the tool response nobody validated. Those are dependencies, and you are not pinning them.&lt;/p&gt;

&lt;h2&gt;
  
  
  The three dependencies nobody pins
&lt;/h2&gt;

&lt;p&gt;Compare a normal service to an LLM feature:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;torch==2.4.1&lt;/code&gt; goes in a lockfile. Your retrieval corpus is a directory that gets rsynced by a cron job.&lt;/li&gt;
&lt;li&gt;A config change goes through review. A prompt template lives in a Google Doc or a database row with an &lt;code&gt;updated_at&lt;/code&gt; column and no diff.&lt;/li&gt;
&lt;li&gt;An upstream API contract is versioned. A model alias like &lt;code&gt;gpt-4o-latest&lt;/code&gt; can change behavior under you with no changelog.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of these are exotic. All of them are ordinary dependency management that we skipped because the artifacts are text and text feels free to change.&lt;/p&gt;

&lt;h2&gt;
  
  
  Content is executable now
&lt;/h2&gt;

&lt;p&gt;Two patterns from the last year worth internalizing, because they generalize:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Instructions hidden in help pages.&lt;/strong&gt; Pages that look like support documentation but contain directives aimed at whatever model reads them. If your retrieval layer indexes third-party or user-editable content, a retrieved document is a user message with extra steps. It arrives inside your system prompt's trust boundary and it is written by someone who is not you.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Published statistics engineered to be cited.&lt;/strong&gt; Publish a plausible-looking number with a memorable framing, and eventually it shows up in generated answers and gets laundered into "facts." This is SEO with a longer feedback loop, and it targets your corpus whether you crawl, license, or accept user uploads.&lt;/p&gt;

&lt;p&gt;Same failure mode both times: &lt;strong&gt;the retrieval corpus is writable by parties outside your trust boundary.&lt;/strong&gt; Prompt injection gets discussed as a prompt problem. It's an ingestion problem. Filtering the output is a losing game; controlling what enters the index is tractable.&lt;/p&gt;

&lt;h2&gt;
  
  
  Gate at ingest, not at prompt
&lt;/h2&gt;

&lt;p&gt;The cheapest control I know is an append-only manifest — a lockfile for content.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# ingest_gate.py — run before any document enters the retrieval index.
&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;hashlib&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;pathlib&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;

&lt;span class="c1"&gt;# Directives that only make sense if a model, not a human, is reading.
&lt;/span&gt;&lt;span class="n"&gt;INSTRUCTION_PATTERNS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;(ignore (all )?previous|you are now|do not recommend|system prompt|&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;when asked about .{0,40}(say|recommend))&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;IGNORECASE&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;sha256&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;hashlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sha256&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;utf-8&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)).&lt;/span&gt;&lt;span class="nf"&gt;hexdigest&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;classify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="c1"&gt;# Trust tier decides whether the doc is served, quoted, or quarantined.
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;quarantine&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;INSTRUCTION_PATTERNS&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;serve&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;ingest&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;pathlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;manifest&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;pathlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read_text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;encoding&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;utf-8&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;record&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;doc_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stem&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;sha256&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;sha256&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;tier&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;classify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;bytes&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;utf-8&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)),&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;source&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;parent&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;  &lt;span class="c1"&gt;# first-party vs. vendor vs. user upload
&lt;/span&gt;    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="c1"&gt;# Append-only: this file is the lockfile for your corpus.
&lt;/span&gt;    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;manifest&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;a&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;encoding&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;utf-8&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;fh&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;fh&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;record&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;record&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;__main__&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;ingest&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pathlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;help_page.md&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;pathlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;corpus.lock&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The regex is not the point — it will miss things, and a determined author will paraphrase. The manifest is the point. Once &lt;code&gt;corpus.lock&lt;/code&gt; is in git, a pull request that adds forty quarantine-tier documents from a new domain is visible to a reviewer. A corpus that changes silently is not.&lt;/p&gt;

&lt;p&gt;Pair it with a per-source trust tier that the retrieval layer enforces: first-party docs can be quoted directly, vendor docs must be attributed inline, user uploads never get inserted into the system prompt at all — only into a delimited user turn.&lt;/p&gt;

&lt;h2&gt;
  
  
  Evals are the lockfile for behavior
&lt;/h2&gt;

&lt;p&gt;A prompt hash, an eval run id, and a deploy id should be joinable three months later. If you can't answer "which prompt and which corpus produced this answer," you have observability for the wrong system.&lt;/p&gt;

&lt;p&gt;Practical constraints I'd argue for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Gate on delta, not absolute score.&lt;/strong&gt; A prompt edit that drops your golden set by two points is noise. One that changes the &lt;em&gt;set of failing cases&lt;/em&gt; is signal, even if the total is flat.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sample size, honestly.&lt;/strong&gt; With a 12-case eval, a single flip is an 8-point swing. I'd rather have 150 labeled cases and a fixed seed than 2,000 unlabeled ones and a judge model.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The judge is not ground truth.&lt;/strong&gt; LLM-as-judge is useful for triage at scale. For anything a customer will notice, hand-label a stratified sample and report agreement between judge and human.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Adversarial cases belong in the golden set.&lt;/strong&gt; Include the poisoned documents you've actually seen. Evals that only contain well-formed questions measure the model, not the system.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Traces matter more as reasoning goes latent
&lt;/h2&gt;

&lt;p&gt;There's a visible trend toward agents that reason in a continuous latent space and only emit text at action boundaries. That's a real efficiency win and a real observability loss: the intermediate tokens you used to read are gone. The final answer tells you less than it used to.&lt;/p&gt;

&lt;p&gt;So log the things that survive: the resolved prompt hash, the retrieved document ids and their trust tiers, every tool call with arguments and status, token counts per span, and the model snapshot string the provider actually returned. OpenTelemetry's GenAI semantic conventions give you attribute names that Langfuse, Phoenix, and most vendors will ingest without a custom adapter. Instrument once at the client wrapper, not at every call site.&lt;/p&gt;

&lt;h2&gt;
  
  
  Rollout mechanics that contain damage
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Shadow traffic first.&lt;/strong&gt; Run the new prompt or model on live requests, log only. Compare against production output offline.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Canary by tenant or route,&lt;/strong&gt; not globally. One enterprise customer with a narrow use case is a better canary than 5% of anonymous traffic.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Per-task budgets.&lt;/strong&gt; Token and tool-call ceilings with a hard stop, not a warning. Unbounded retry loops are the most expensive bug class in this space.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Invalidate response caches on corpus change.&lt;/strong&gt; A cached answer produced from a document you've since quarantined is a live injection vector.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A kill switch that degrades to retrieval-without-generation.&lt;/strong&gt; If the model provider is down, a list of links beats a 500.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Pitfalls
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Versioning the prompt but not the retrieval config.&lt;/strong&gt; Chunk size, top-k, and reranker settings change behavior as much as the prompt does.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Trusting a model alias.&lt;/strong&gt; Pin a dated snapshot. If the provider retires it, you want that to be an incident, not a surprise.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treating eval scores as portable.&lt;/strong&gt; A score from last month's corpus says nothing about this month's.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Blocking on content moderation at output.&lt;/strong&gt; It catches tone, not exfiltration or instruction-following.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Unbounded tool output into context.&lt;/strong&gt; A 200KB API response will crowd out your instructions and look like a model regression.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Monday checklist
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;[ ] Corpus is written to an append-only manifest with &lt;code&gt;sha256&lt;/code&gt;, source, and trust tier; the manifest is in git.&lt;/li&gt;
&lt;li&gt;[ ] Retrieval layer enforces trust tiers — user uploads never enter the system prompt.&lt;/li&gt;
&lt;li&gt;[ ] The model is a dated snapshot string, pinned in config, with an owner for retirement.&lt;/li&gt;
&lt;li&gt;[ ] Prompt template, retrieval config, and corpus manifest are hashed into one deploy id.&lt;/li&gt;
&lt;li&gt;[ ] Golden set has ≥100 cases including every adversarial document seen in the last quarter.&lt;/li&gt;
&lt;li&gt;[ ] Eval gate compares failing-case sets, not just aggregate score.&lt;/li&gt;
&lt;li&gt;[ ] Traces record retrieved doc ids, tool calls with arguments, and token counts per span.&lt;/li&gt;
&lt;li&gt;[ ] Per-task token and tool-call ceilings with hard stops.&lt;/li&gt;
&lt;li&gt;[ ] Response cache is keyed on corpus version and invalidated on change.&lt;/li&gt;
&lt;li&gt;[ ] Shadow and canary paths exist, and someone has flipped the kill switch in staging this quarter.&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>devops</category>
      <category>python</category>
    </item>
  </channel>
</rss>
