The Expensive Part of an AI Feature Is the Second Call

Last month I watched a summarizer endpoint go from roughly 900 ms p99 to 3.4 s in a single afternoon, and nothing about the model had changed at all. Someone had added a fallback provider because "reliability", which felt responsible on Tuesday and looked like vandalism on the latency chart by Friday. The prompt was identical, the response shape was identical, and the only diff was fourteen lines of retry logic sitting quietly between the route and the client.

Here is where it actually broke, and it was not where the diff was. The gateway handed every attempt a fresh 1.5 s timeout, so two attempts meant a 3 s ceiling and three attempts meant 4.5 s, and no single place in the code knew the request had a budget in the first place. The retry was not the bug. The arithmetic was. Have you ever stared at a p99 chart after adding a fallback and wondered who moved the ceiling? Nobody moved it. It was never there.

So let me take a position instead of hedging: for most AI features, the expensive part is the second call, not the token price. The token price is a number on a monthly invoice that mostly behaves, while retries and fallbacks are a number that appears nowhere, because nobody ever bills you for the call that should not have been made. If you only instrument one thing this quarter, instrument the second call.

A budget that cannot be renegotiated

I want the deadline to be created exactly once, at the edge, and then inherited by every layer underneath it without exception. The trick is that a retry must never receive a fresh timeout from the provider client's defaults, so I make the reservation explicit and I make it the only way to call a model.

# budget.py
import time, uuid
from dataclasses import dataclass, field
from typing import Callable

class BudgetExhausted(Exception):
    def __init__(self, reason: str, remaining_ms: int):
        super().__init__(reason)
        self.reason = reason          # "deadline" | "call_budget"
        self.remaining_ms = remaining_ms

@dataclass
class Budget:
    total_ms: int
    max_calls: int = 2
    clock: Callable[[], float] = time.monotonic
    started_at: float = field(init=False)
    calls: int = 0
    trace_id: str = field(default_factory=lambda: uuid.uuid4().hex)

    def __post_init__(self) -> None:
        self.started_at = self.clock()

    def remaining_ms(self) -> int:
        return int(self.total_ms - (self.clock() - self.started_at) * 1000)

    def claim(self) -> int:
        """Reserve one attempt and return the timeout it is allowed to use."""
        remaining = self.remaining_ms()
        if self.calls >= self.max_calls:
            raise BudgetExhausted("call_budget", remaining)
        if remaining <= 0:
            raise BudgetExhausted("deadline", remaining)
        self.calls += 1
        return remaining

The important line is return remaining, because the second attempt does not get the default timeout back, it gets whatever survived the first attempt. On a slow provider day that is often not much, and a very small timeout that fails fast is infinitely more useful than a generous one that fails late. The injected clock is not decoration either; it is what lets me assert exact numbers in a test instead of sleeping and hoping.

def call_model(prompt: str, budget: Budget, client) -> str:
    timeout_ms = budget.claim()
    return client.complete(prompt=prompt, timeout=timeout_ms / 1000)

Make the failure a status code, not a stack trace

A budget that only exists inside Python is a suggestion, so the route has to translate exhaustion into something the caller can act on. I return 504 when the deadline is gone and 503 when the call allowance is gone, because those are genuinely different situations for whoever is retrying me. Both carry the trace id, so the client can quote it in a bug report instead of describing the weather.

@app.post("/summarize")
async def summarize(req: SummarizeRequest):
    budget = Budget(total_ms=req.deadline_ms, max_calls=2)
    try:
        text = await run_in_threadpool(call_model, req.text, budget, client)
    except BudgetExhausted as exc:
        ledger.write(budget, outcome=exc.reason, remaining_ms=exc.remaining_ms)
        code = 504 if exc.reason == "deadline" else 503
        return JSONResponse(
            {"error": exc.reason, "trace_id": budget.trace_id},
            status_code=code,
            headers={"Retry-After": "1", "X-Trace-Id": budget.trace_id},
        )
    ledger.write(budget, outcome="ok", remaining_ms=budget.remaining_ms())
    return {"summary": text, "trace_id": budget.trace_id, "calls": budget.calls}

Now the test that would have caught the Friday incident, written with the fake clock so the assertion is about arithmetic rather than wall time.

def test_retry_inherits_the_remaining_deadline():
    now = [0.0]
    budget = Budget(total_ms=400, max_calls=2, clock=lambda: now[0])

    first = budget.claim()                       # 400 ms granted
    now[0] += 0.300                              # provider eats 300 ms
    second = budget.claim()                      # must be ~100 ms, not 400

    assert first == 400
    assert second <= 100


def test_a_third_attempt_is_refused_before_it_is_sent():
    budget = Budget(total_ms=10_000, max_calls=2, clock=lambda: 0.0)
    budget.claim(); budget.claim()
    with pytest.raises(BudgetExhausted) as err:
        budget.claim()
    assert err.value.reason == "call_budget"

That second test matters more than it looks, because refusing locally is free and refusing remotely is not. When the call budget is enforced before the network hop, a fallback storm turns into a handful of fast rejections instead of a bill.

The invoice is a lagging indicator

Once each attempt is a row, the retry tax stops being an opinion and becomes a query. I keep the ledger deliberately boring, because a schema I can inspect on a bad day beats a dashboard I have to trust.

CREATE TABLE model_calls (
  trace_id     TEXT NOT NULL,
  attempt      INTEGER NOT NULL,
  provider     TEXT NOT NULL,
  outcome      TEXT NOT NULL,   -- ok | deadline | call_budget | provider_error
  remaining_ms INTEGER NOT NULL,
  created_at   TEXT NOT NULL DEFAULT (datetime('now')),
  PRIMARY KEY (trace_id, attempt)
);

-- calls you paid for and threw away
WITH last_attempt AS (
  SELECT trace_id, MAX(attempt) AS attempt FROM model_calls GROUP BY trace_id
)
SELECT c.provider, c.outcome,
       COUNT(*) AS wasted_calls,
       ROUND(SUM(c.remaining_ms) / 1000.0, 1) AS budget_still_on_the_clock_s
FROM model_calls c
JOIN last_attempt l ON l.trace_id = c.trace_id
WHERE c.attempt < l.attempt AND c.outcome != 'ok'
GROUP BY c.provider, c.outcome
ORDER BY wasted_calls DESC;

When I ran this shape of query against the summarizer, the fallback provider accounted for a third of all attempts and answered almost none of them. That is the number I now put next to cost per request, because cost per request happily averages away the calls that produced nothing.

Where the free tier actually helped

I did not want to tune a timeout budget while a meter was running, so I kept the failure suite pointed at a live endpoint and let the gateway sit on a real host instead of my laptop. MonkeyCode's free model access covered the model calls while I dialed in max_calls and the default deadline, and the free server option let the ledger live in the same network position it occupies in production, which is the only position where a latency number means anything. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The offering is described to me as ten million free tokens plus a free server, and I will be honest that I have verified only what my own runs consumed, not the shape of anyone else's workload. Treat the allowance as a starting budget for the instrumentation work rather than a benchmark, and keep the failure tests you actually care about in your own CI either way.

Limitations, and who should skip this

A budget bounds latency and it does not bound quality, so a tight deadline will occasionally turn a slow good answer into a fast useful error. If your product cannot return a 503 under load, you need a queue with a visible backlog instead of a gate that rejects, and that is a different design. If you stream tokens, claim() once at the start is theatre, because you must re-check the remaining budget between chunks or the deadline only exists in the response headers. The SQLite ledger is single-node and illustrative, and teams already running per-route deadlines in a service mesh should measure there rather than adding a second clock that disagrees with the first one.

Reuse checklist: create the deadline once at the edge and pass it down; reserve every attempt through one function; record provider, attempt, and remaining budget per call; return distinct status codes for deadline versus call allowance; and ask the wasted-call query every week whether the fallback is earning its place.

So which handoff in your stack has no owner for the clock: the gateway, the queue, or the retry buried inside the SDK? Send me the failure state you actually see, ideally the response code and the trace id, and I will happily argue about whose timeout it is.