The Agent Loop Is a Runtime: Budgets, Cancellation, and Idempotency

You have tool calling working. The demo terminates, the JSON parses, the model picks search_flights instead of book_flight. Then it goes into production and the incidents are not about model quality at all. They are about the loop: a run that quietly spun for forty minutes, a refund issued twice, a cancel button that stopped the UI but not the worker, a bill that doubled because every step resends the entire history.

Treat the loop as a runtime you own, with the same concerns as any scheduler: admission control, deadlines, cooperative cancellation, and at-most-once side effects. The model is a component inside that runtime, not the runtime itself.

Four budgets, not one

An agent run needs at least four independent limits:

• Steps — the outer for loop. Cheap to check, catches the obvious spin.
• Wall clock — a deadline, not a per-step timeout. Users care about the run, not the step.
• Tool calls — separate from steps, because one step can fan out.
• Money or tokens — the one that actually stops a runaway, since cost per step grows with history.

The last point matters and is easy to miss: because you resend the conversation each step, per-step input tokens grow roughly linearly with step count, so total run cost grows faster than linearly. A step budget of 12 is not "12 cheap calls"; step 12 is the most expensive call in the run. Cap on cumulative tokens or dollars if you want a real ceiling.

Which budget binds first is workload-dependent, and you should measure it rather than guess. Log steps-to-completion, tool-calls-to-completion and wall-clock-to-completion per task type, then set your caps near the p99 of successful runs. If a large fraction of runs hit the cap, the task is out of scope for the agent — tightening or loosening the number will not fix it.

The trade-off is honest: a hard cap converts a slow success into a fast failure. That is usually right for an interactive request and usually wrong for a batch job where you would rather finish in ten minutes than fail in ninety seconds. Make it a parameter, not a constant.

Cancellation has to reach the tool

asyncio.Task.cancel(), a threading.Event, a context flag — all of them cancel your loop. None of them cancel subprocess.run without a timeout, an HTTP client with no read timeout, or a Postgres query with no statement_timeout. Your cancel becomes a lie the moment the run is inside a tool.

Two things fix this. First, propagate the deadline, not the duration: each tool gets the remaining time and computes its own timeouts from it. Second, every blocking call gets an explicit timeout, including the model call, and a timeout is a returned error to the model rather than a crash.

And accept that after a cancel, the run is in an unknown state. Whatever the tool was doing may have half-completed. Do not pretend otherwise — put a reconciliation path behind anything that mutates.

Side effects need a key the model cannot choose

Agents retry constantly. The model re-emits a tool call after a truncated response; your SDK retries a 500; a worker restarts mid-step. Any of these can replay a write. The fix is the same one you would apply to a webhook handler: an idempotency key derived deterministically from the call, stored in the same transaction as the effect, checked before execution.

Derive it from (run_id, tool_name, canonicalized_args) and hash it. Not from the model's prose, not from a timestamp, not from anything the model picks. For third-party APIs that do not support idempotency keys, record intent before the call and reconcile after — the outbox pattern, unchanged from the last decade of distributed systems.

"""Bounded agent runner: deadline, step budget, idempotent side effects."""
import hashlib, json, time

class BudgetExceeded(Exception): pass
class Cancelled(Exception): pass

def idem_key(run_id: str, tool: str, args: dict) -> str:
    # Canonical JSON so key order can't produce two keys for the same call.
    # Never include a timestamp, a retry counter, or free-text from the model.
    payload = json.dumps({"run": run_id, "tool": tool, "args": args},
                         sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(payload.encode()).hexdigest()[:32]

def run_agent(task, llm_step, tools, ledger, *, run_id,
              max_steps=12, deadline_s=90.0, cancel=lambda: False):
    deadline = time.monotonic() + deadline_s
    messages = [{"role": "user", "content": task}]

    for step in range(max_steps):
        if cancel():
            raise Cancelled(f"run {run_id} cancelled at step {step}")
        remaining = deadline - time.monotonic()
        if remaining <= 0:
            raise BudgetExceeded(f"deadline hit at step {step}")

        # Reserve the final step for an answer. A loop that spends its whole
        # budget on tool calls returns nothing usable.
        action = llm_step(messages, remaining_s=remaining,
                          force_final=(step == max_steps - 1))
        if action["type"] == "final":
            return action["content"]

        name, args = action["tool"], action["args"]
        key = idem_key(run_id, name, args)

        if key in ledger:                      # replay: reuse recorded outcome
            messages.append({"role": "tool", "content": ledger[key]})
            continue

        try:                                   # tool gets REMAINING time
            result = tools[name](args, timeout_s=max(0.1, deadline - time.monotonic()))
        except TimeoutError as exc:
            result = {"error": f"{name} timed out: {exc}"}   # feed back, don't raise
        ledger[key] = result                   # same tx as the side effect
        messages.append({"role": "tool", "content": json.dumps(result)})

    raise BudgetExceeded(f"step budget exhausted for run {run_id}")

Wire ledger to Redis or a table with the key as primary key, and make the write atomic with the effect where you can.

Traces that answer the on-call question

The question at 2am is never "what was the prompt". It is "why did this run call refund twice" and "where did the $4 go". One span per step, with: step index, model, input/output tokens, latency, tool name, arg hash, result hash, outcome. Log the hashes, not the payloads, in the hot path; put payloads in object storage keyed by the same hash.

With that, duplicate side effects become a five-minute query: group spans by arg hash within a run and look for repeats.

Pitfalls

• Retries multiplying across layers. SDK retry × your HTTP retry × agent retry = up to 27 attempts on the same write. Pick one layer to retry and disable the others.
• Per-step timeouts instead of a run deadline. Twelve steps at 30s each is a six-minute request.
• Catching Exception (or BaseException) around the loop. Cancellation and deadline exits get swallowed and the loop continues.
• Idempotency keys built from a timestamp or a UUID generated at call time. Guarantees duplicates instead of preventing them.
• Unbounded message history. Cost per step climbs, and at some point the tool outputs crowd out the instructions.
• Cancel implemented only in the UI. The worker keeps going, and so does the bill.

Monday morning checklist

• [ ] Each run has an explicit max_steps, wall-clock deadline, tool-call cap and token/dollar cap.
• [ ] The final step is reserved for an answer, not a tool call.
• [ ] Every blocking call — model, HTTP, subprocess, DB — has a timeout derived from the remaining run budget.
• [ ] Cancellation is checked between steps and reaches in-flight tools.
• [ ] Every mutating tool call carries a deterministic

Comments

[오피셜메일]

run_id·tool·canonicalized_args를 키로 묶는 예시에서, 같은 실행 안에 같은 인자로 두 번 수행해야 하는 정상 작업은 어떻게 구분하시나요? 재시도에는 같은 키를 유지하되 업무 의도를 나타내는 operation_id를 별도로 두면, 중복 방지와 새 요청을 나누는 기준이 더 선명해질 것 같습니다.