Migration Diary: Freeze Tool Dispatch Until Every Paid-Runtime Lease Drains

You should refuse a runtime cutover until every in-flight tool call has an owner, a deadline, and a drain path. Paid agent APIs hide unfinished work behind retries, long streams, and vendor queues you never named. A new free-model lane will not inherit those hidden leases, so a bare key swap can strand callbacks and duplicate side effects. This diary gives you a staging workflow, a lease registry, and a cutover sequence you can rehearse without fake production metrics.

What actually breaks when you leave the old runtime

You usually inventory prompts, keys, and model aliases, then still miss the work that is already in motion. A tool call that started on the paid runtime may still be writing a ticket, charging a card, or holding an external lock. Your new runtime will accept a retry of the same user turn and fire the same tool again without shame. That failure is not a quality regression; it is a leftover contract sitting between your orchestrator and another system.

Streaming makes the leftover worse because the browser still holds a socket the old vendor considers live. Cancellation in your process does not always cancel the vendor-side tool, especially when that tool is a webhook into billing or mail. Timeouts that felt safe against a paid endpoint often under-estimate queue delay and cold starts on a free server. You need a named map of those timings before you point any write traffic at a new runtime.

Hidden retry middleware is the third leftover that turns one user click into two side effects. Vendor SDKs retry on 429 and 503 with jitter you never configured in your own code. During cutover those retries land on a runtime that already accepted the same idempotency key, or they land on a runtime that never saw it. Either way, your dispatcher must own the retry budget before the DNS record or the API key changes.

Build a lease for every tool call, not a log line

A log line is not an owner, and a metrics counter cannot compensate a doubled invoice. You want a lease record that survives process restart and tells the cutover job which calls are still dangerous. Store the turn id, the tool name, an idempotency key, the deadline, the payload hash, and the runtime that accepted the call. Heartbeat the lease while the tool runs, and mark it drained only when the side effect is confirmed or safely compensated.

The following example is a proposal you can adapt; it is not a production benchmark and it has not been executed against a live vendor. Keep secrets out of the table by hashing arguments, and mint the idempotency key before the first HTTP call. If you mint the key after a timeout, you will create two leases for one human action and the drain gate will lie.

1. Define the lease document

# proposal: in-flight lease schema, unexecuted example
from dataclasses import dataclass, field
from datetime import datetime, timedelta, timezone
from typing import Literal
import hashlib, json, uuid

RuntimeName = Literal["paid_vendor", "staging_free", "prod_free"]
LeaseState = Literal["open", "running", "draining", "committed", "compensated", "expired"]

@dataclass
class ToolLease:
    lease_id: str
    turn_id: str
    tool_name: str
    idempotency_key: str
    runtime: RuntimeName
    state: LeaseState
    deadline_at: datetime
    heartbeat_at: datetime
    payload_hash: str
    created_at: datetime = field(
        default_factory=lambda: datetime.now(timezone.utc)
    )

def hash_payload(args: dict) -> str:
    blob = json.dumps(args, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(blob.encode("utf-8")).hexdigest()

You keep payload_hash instead of raw arguments so support can detect duplicate turns without copying customer fields into the drain table. deadline_at should be shorter than the vendor timeout you are leaving, so your orchestrator expires first and still has room to compensate. The runtime field is a routing fact, not a model name, which keeps the table stable when you add a staging lane later.

2. Put a registry in front of every tool dispatcher

# proposal: registry with explicit drain, unexecuted example
class LeaseRegistry:
    def __init__(self, clock=lambda: datetime.now(timezone.utc)):
        self._clock = clock
        self._leases: dict[str, ToolLease] = {}

    def open(self, turn_id, tool_name, runtime, payload_hash, ttl_seconds=45):
        now = self._clock()
        key = f"{turn_id}:{tool_name}:{payload_hash}"
        existing = self._leases.get(key)
        if existing and existing.state in {"open", "running", "draining"}:
            return existing  # do not start a second side effect
        lease = ToolLease(
            lease_id=str(uuid.uuid4()),
            turn_id=turn_id,
            tool_name=tool_name,
            idempotency_key=key,
            runtime=runtime,
            state="open",
            deadline_at=now + timedelta(seconds=ttl_seconds),
            heartbeat_at=now,
            payload_hash=payload_hash,
        )
        self._leases[key] = lease
        return lease

    def heartbeat(self, idempotency_key: str) -> None:
        lease = self._leases[idempotency_key]
        now = self._clock()
        if now > lease.deadline_at:
            lease.state = "expired"
            raise TimeoutError(f"lease expired: {idempotency_key}")
        lease.heartbeat_at = now
        lease.state = "running"

    def mark_draining(self, runtime: RuntimeName) -> list[ToolLease]:
        draining = []
        for lease in self._leases.values():
            if lease.runtime == runtime and lease.state in {"open", "running"}:
                lease.state = "draining"
                draining.append(lease)
        return draining

    def inflight_for(self, runtime: RuntimeName) -> list[ToolLease]:
        live = {"open", "running", "draining"}
        return [
            lease for lease in self._leases.values()
            if lease.runtime == runtime and lease.state in live
        ]

You should wrap the dispatcher so a tool cannot run unless open() returns a lease this process owns. If open() returns an existing running lease, you wait or attach to that result instead of calling the vendor again. That single branch is the difference between a clean cutover and a doubled refund, a doubled email, or a doubled deploy. Workers that bypass the wrapper will not appear in inflight_for(), and the gate will report a false zero.

3. Rehearse drain with a clock you control

# proposal: deterministic drain test, unexecuted example
from datetime import datetime, timezone

def test_drain_blocks_cutover_while_tool_is_running():
    frozen = {"now": datetime(2026, 9, 6, 12, 0, tzinfo=timezone.utc)}
    registry = LeaseRegistry(clock=lambda: frozen["now"])
    lease = registry.open(
        "turn-9", "create_invoice", "paid_vendor", "hash-a", ttl_seconds=45
    )
    registry.heartbeat(lease.idempotency_key)
    draining = registry.mark_draining("paid_vendor")
    assert len(draining) == 1
    assert registry.inflight_for("paid_vendor")[0].state == "draining"
    # cutover gate: refuse to flip the router while this list is non-empty

You run that style of test in CI so the gate is a failing assertion, not a reminder in chat. Label the test unexecuted until you wire a real store such as Redis or Postgres with the same fields. The contract you want is simple: cutover stays blocked while inflight_for(old_runtime) returns rows. A green test against a fake clock is not evidence that your vendor cancels webhooks; it is evidence that your router cannot lie about ownership.

4. Give the router a freeze flag and a timeout map

# proposal: cutover router with a drain gate, unexecuted example
class RuntimeRouter:
    def __init__(self, registry: LeaseRegistry, flags: dict):
        self.registry = registry
        self.flags = flags

    def choose(self, tool_name: str) -> RuntimeName:
        if self.flags.get("freeze_old_runtime"):
            inflight = self.registry.inflight_for("paid_vendor")
            if inflight:
                raise RuntimeError(
                    f"cutover blocked: {len(inflight)} leases still live"
                )
            return "staging_free"
        return "paid_vendor"

# proposal: timeouts you intend to verify, not measured vendor SLAs
TIMEOUT_MAP = {
    "paid_vendor.create_invoice": {
        "client_s": 30, "lease_s": 45, "compensate_s": 90
    },
    "staging_free.create_invoice": {
        "client_s": 60, "lease_s": 75, "compensate_s": 120
    },
}

You should copy TIMEOUT_MAP into the cutover ticket and replace the numbers only after you observe your own client, not after reading a marketing page. The staging row is allowed to be more patient than the paid row because a free server may queue. What you must not do is keep the paid client timeout while pointing writes at a colder runtime, because the lease will expire while the tool is still working. Compensation then races the late success, which is how you get both a refund and a charge.

A cutover sequence you can actually follow

Follow these numbered steps in order, and do not skip the pause even if the new lane looks idle from your laptop.

1. Freeze new writes on the old runtime by setting a feature flag that opens leases as draining instead of running.
2. Snapshot inflight_for(old_runtime) and export JSON your incident channel can read without opening a dashboard.
3. Wait until every draining lease is committed, compensated, or expired, and refuse to proceed while the list is non-empty.
4. Point a canary of read-only turns at the staging runtime, still writing leases, and compare tool names plus payload hashes only.
5. Enable write tools on the staging runtime for a single tenant whose side effects you can reverse by hand.
6. Flip the default router only after that tenant completes a full business action without a duplicate lease.

# proposal: operator commands during drain, not a live runbook with SLAs
python -m agent_cutover snapshot --runtime paid_vendor --out /tmp/leases.json
python -m agent_cutover freeze --flag freeze_old_runtime
python -m agent_cutover wait --runtime paid_vendor --poll-seconds 5
python -m agent_cutover canary --runtime staging_free --tenant acme-dev
python -m agent_cutover diff-hashes --old /tmp/leases.json --new /tmp/canary.json

You should keep the old key alive in a break-glass path until the longest tool deadline has passed twice. That second window covers heartbeats delayed by a worker restart during the drain, which is common when you bounce pods after a flag flip. If a compensated lease later reports success from the vendor, you treat that as an incident, not as a reason to skip the registry next time. Dual-running read-only turns is cheaper than explaining a doubled side effect to the tenant you used as a canary.

Where a free model lane and a free server actually help

You need a staging fabric that is not your paid production quota, or you will rehearse the drain against the same bill you are trying to leave. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option you can use as that staging fabric while the lease registry remains the source of truth in your own store. Put the free server behind the same dispatcher wrapper, so the runtime name in the lease changes without a second code path for tool calls.

Do not treat the free lane as proof that latency, context behavior, or tool-calling shape will match the vendor you are leaving. You are borrowing a place to exercise drain, dual-run, and compensation, not outsourcing those contracts to a new brand name. Keep streaming frames and vendor envelopes behind an adapter so the registry never stores a payload you cannot replay. If the adapter cannot translate a tool result into your idempotency key, you are not ready to cut over that tool, and the freeze flag should stay on.

Decision table for the leftovers

Leftover	Drain action	Cut over only when
In-flight tool with side effects	mark_draining, wait, compensate on expire	inflight count is 0
Open SSE or stream	abort the client, confirm vendor cancel or TTL	no sockets remain in CLOSE-WAIT for that key
Vendor auto-retry	disable client retries, keep idempotency_key	retry budget is owned by your dispatcher
Webhook into SaaS	pause the subscription or filter by runtime header	duplicate deliveries are ignorable
Scheduled agent turns	freeze the cron, drain the queue	next fire time is after the flip
Cached tool results	namespace the cache by runtime name	the old namespace is read-only

You should paste this table into the cutover ticket so leftovers are named before someone deletes the old project. Each row needs an owner and a command, not a hope that the vendor drains in-flight work for you. If a row cannot be tested with the fake clock or with one reversible tenant, that tool stays on the old runtime. A missing row is how a forgotten cron keeps posting to a paid key you already rotated in the dashboard.

Limitations and who should not use this

This workflow does not measure answer quality, safety, or cost, and it will not make a non-idempotent tool safe overnight. If your tools cannot be retried or compensated, you should not cut over with a staging lane that might replay a turn. Teams that stream token-by-token UI with vendor-specific frame shapes need an adapter first, or the drain will look idle while the browser still holds a socket. Regulated workloads that cannot send prompts to a third-party staging host should rehearse drain against a local fake runtime instead.

The lease registry is only as true as the dispatcher wrapping, which means a rogue worker is an undeclared runtime. You also cannot infer capacity, duration, or permanence of any free offering from this diary; treat availability as a snapshot you re-check on the day you stage. If you need hard multi-region failover, this cutover plan is too small and you want an outbox plus a durable queue. Skip the approach entirely when your product cannot pause writes for the length of the longest tool deadline.

What you should leave behind after the flip

After the flip, keep the drain snapshot, the idempotency keys, and the old break-glass key for one extra deadline cycle. Delete vendor SDKs, retry middleware, and stream parsers that encode the old envelope only after the canary tenant survives a full day of real tools. The leftover that bites later is usually a hidden scheduler that still posts to the paid runtime with a key you thought was retired. Name that scheduler in the same ticket as the lease table, or you will migrate the chat path and leave the overnight path on the invoice.

You now have a conclusion you can defend in a review: no inflight leases, no silent retries, and no second dispatcher for the staging server. Run the unexecuted tests against a fake clock first, then against one reversible tool whose compensation you can see. The runtime you choose after that is a routing detail; the contract you refuse to abandon is the lease that makes a cutover boring.