DEV Community

skvrtyk
skvrtyk

Posted on

The Agent Loop Is a Runtime: Budgets, Cancellation, and Idempotency

You have tool calling working. The demo terminates, the JSON parses, the model picks search_flights instead of book_flight. Then it goes into production and the incidents are not about model quality at all. They are about the loop: a run that quietly spun for forty minutes, a refund issued twice, a cancel button that stopped the UI but not the worker, a bill that doubled because every step resends the entire history.

Treat the loop as a runtime you own, with the same concerns as any scheduler: admission control, deadlines, cooperative cancellation, and at-most-once side effects. The model is a component inside that runtime, not the runtime itself.

Four budgets, not one

An agent run needs at least four independent limits:

  • Steps — the outer for loop. Cheap to check, catches the obvious spin.
  • Wall clock — a deadline, not a per-step timeout. Users care about the run, not the step.
  • Tool calls — separate from steps, because one step can fan out.
  • Money or tokens — the one that actually stops a runaway, since cost per step grows with history.

The last point matters and is easy to miss: because you resend the conversation each step, per-step input tokens grow roughly linearly with step count, so total run cost grows faster than linearly. A step budget of 12 is not "12 cheap calls"; step 12 is the most expensive call in the run. Cap on cumulative tokens or dollars if you want a real ceiling.

Which budget binds first is workload-dependent, and you should measure it rather than guess. Log steps-to-completion, tool-calls-to-completion and wall-clock-to-completion per task type, then set your caps near the p99 of successful runs. If a large fraction of runs hit the cap, the task is out of scope for the agent — tightening or loosening the number will not fix it.

The trade-off is honest: a hard cap converts a slow success into a fast failure. That is usually right for an interactive request and usually wrong for a batch job where you would rather finish in ten minutes than fail in ninety seconds. Make it a parameter, not a constant.

Cancellation has to reach the tool

asyncio.Task.cancel(), a threading.Event, a context flag — all of them cancel your loop. None of them cancel subprocess.run without a timeout, an HTTP client with no read timeout, or a Postgres query with no statement_timeout. Your cancel becomes a lie the moment the run is inside a tool.

Two things fix this. First, propagate the deadline, not the duration: each tool gets the remaining time and computes its own timeouts from it. Second, every blocking call gets an explicit timeout, including the model call, and a timeout is a returned error to the model rather than a crash.

And accept that after a cancel, the run is in an unknown state. Whatever the tool was doing may have half-completed. Do not pretend otherwise — put a reconciliation path behind anything that mutates.

Side effects need a key the model cannot choose

Agents retry constantly. The model re-emits a tool call after a truncated response; your SDK retries a 500; a worker restarts mid-step. Any of these can replay a write. The fix is the same one you would apply to a webhook handler: an idempotency key derived deterministically from the call, stored in the same transaction as the effect, checked before execution.

Derive it from (run_id, tool_name, canonicalized_args) and hash it. Not from the model's prose, not from a timestamp, not from anything the model picks. For third-party APIs that do not support idempotency keys, record intent before the call and reconcile after — the outbox pattern, unchanged from the last decade of distributed systems.

"""Bounded agent runner: deadline, step budget, idempotent side effects."""
import hashlib, json, time

class BudgetExceeded(Exception): pass
class Cancelled(Exception): pass

def idem_key(run_id: str, tool: str, args: dict) -> str:
    # Canonical JSON so key order can't produce two keys for the same call.
    # Never include a timestamp, a retry counter, or free-text from the model.
    payload = json.dumps({"run": run_id, "tool": tool, "args": args},
                         sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(payload.encode()).hexdigest()[:32]

def run_agent(task, llm_step, tools, ledger, *, run_id,
              max_steps=12, deadline_s=90.0, cancel=lambda: False):
    deadline = time.monotonic() + deadline_s
    messages = [{"role": "user", "content": task}]

    for step in range(max_steps):
        if cancel():
            raise Cancelled(f"run {run_id} cancelled at step {step}")
        remaining = deadline - time.monotonic()
        if remaining <= 0:
            raise BudgetExceeded(f"deadline hit at step {step}")

        # Reserve the final step for an answer. A loop that spends its whole
        # budget on tool calls returns nothing usable.
        action = llm_step(messages, remaining_s=remaining,
                          force_final=(step == max_steps - 1))
        if action["type"] == "final":
            return action["content"]

        name, args = action["tool"], action["args"]
        key = idem_key(run_id, name, args)

        if key in ledger:                      # replay: reuse recorded outcome
            messages.append({"role": "tool", "content": ledger[key]})
            continue

        try:                                   # tool gets REMAINING time
            result = tools[name](args, timeout_s=max(0.1, deadline - time.monotonic()))
        except TimeoutError as exc:
            result = {"error": f"{name} timed out: {exc}"}   # feed back, don't raise
        ledger[key] = result                   # same tx as the side effect
        messages.append({"role": "tool", "content": json.dumps(result)})

    raise BudgetExceeded(f"step budget exhausted for run {run_id}")
Enter fullscreen mode Exit fullscreen mode

Wire ledger to Redis or a table with the key as primary key, and make the write atomic with the effect where you can.

Traces that answer the on-call question

The question at 2am is never "what was the prompt". It is "why did this run call refund twice" and "where did the $4 go". One span per step, with: step index, model, input/output tokens, latency, tool name, arg hash, result hash, outcome. Log the hashes, not the payloads, in the hot path; put payloads in object storage keyed by the same hash.

With that, duplicate side effects become a five-minute query: group spans by arg hash within a run and look for repeats.

Pitfalls

  • Retries multiplying across layers. SDK retry × your HTTP retry × agent retry = up to 27 attempts on the same write. Pick one layer to retry and disable the others.
  • Per-step timeouts instead of a run deadline. Twelve steps at 30s each is a six-minute request.
  • Catching Exception (or BaseException) around the loop. Cancellation and deadline exits get swallowed and the loop continues.
  • Idempotency keys built from a timestamp or a UUID generated at call time. Guarantees duplicates instead of preventing them.
  • Unbounded message history. Cost per step climbs, and at some point the tool outputs crowd out the instructions.
  • Cancel implemented only in the UI. The worker keeps going, and so does the bill.

Monday morning checklist

  • [ ] Each run has an explicit max_steps, wall-clock deadline, tool-call cap and token/dollar cap.
  • [ ] The final step is reserved for an answer, not a tool call.
  • [ ] Every blocking call — model, HTTP, subprocess, DB — has a timeout derived from the remaining run budget.
  • [ ] Cancellation is checked between steps and reaches in-flight tools.
  • [ ] Every mutating tool call carries a deterministic

Top comments (3)

Collapse
 
max_quimby profile image
Max Quimby •

This matches our scars almost exactly. The one that bit us hardest was the cancellation-is-a-lie point — our outer loop would "stop," but the actual work was a subprocess running under a 20-minute job, and killing the loop left the tool half-done with no reconciliation path. We ended up doing exactly what you describe: propagate the remaining deadline into each tool rather than a fresh per-step timeout, so a tool that starts late inherits less runway.

The token-budget-grows-superlinearly point deserves more attention than it usually gets. People set a step cap of 12 and reason about it as 12 equal calls, when step 12 is carrying the entire transcript and is the most expensive call in the run. Capping on cumulative tokens/dollars is the only budget that actually bounds a runaway.

Curious how you handle the idempotency key when the same logical write legitimately needs to happen twice in one run — do you fold an occurrence counter into the canonicalized args, or scope the key to a sub-step?

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

Budgets should count tokens per step, not calls. A retry that resends a long history costs more than the call that failed, and a call count never shows it.

Collapse
 
officialmailkr profile image
오피셜메일 •

run_id·tool·canonicalized_args를 키로 묶는 예시에서, 같은 실행 안에 같은 인자로 두 번 수행해야 하는 정상 작업은 어떻게 구분하시나요? 재시도에는 같은 키를 유지하되 업무 의도를 나타내는 operation_id를 별도로 두면, 중복 방지와 새 요청을 나누는 기준이 더 선명해질 것 같습니다.