Migration Diary: Rebuild Timeouts and Cancellation Before You Leave a Paid Agent SDK

You should rebuild the timeout ladder and cancellation path before you leave a paid agent SDK. Paid runtimes hide connect, first-token, tool, and total budgets inside client defaults you never copied. If you cut the API key first, in-flight streams keep retrying the old endpoint while the new runtime looks idle. That leftover is not a prompt file; it is work that still bills and still mutates tools.

The leftover nobody inventories

Most cutover checklists copy prompts, tools, and conversation history, then swap the base URL. That sequence misses the hidden contract the vendor SDK enforced for you on every call. Connect timeouts, read timeouts, retry-on-429, and stream abort were product features rather than code you owned. After the old key dies, those retries become zombie traffic against a disabled project or an unmetered host.

You also lose cancellation semantics that felt automatic while the paid gateway was in the path. A vendor proxy often closes upstream work when the browser tab drops or the socket resets. A raw HTTP client against a new host will keep a tool running unless you thread an abort signal. Duplicate side effects then appear as two refunds, two tickets, or two deploys from one turn.

Treat this writeup as a migration diary rather than a glamorous model rewrite. You freeze generation, drain in-flight work, and install your own ladder before any DNS change. Only after abort is proven should you point any live traffic at the new runtime. The model name sitting in the diff is the least important line in the cutover.

What you extract before any DNS change

Walk the old client with a notebook and write down numbers instead of slogans from the vendor dashboard. You need four budgets and one cancel rule, because most SDKs collapse them into a single timeout= argument. If the console never showed those values, you measure them from real turns rather than copying a blog default. Guessed seconds will flap the moment the first slow tool appears.

1. Connect budget: how long the TCP handshake and TLS setup may take before you fail closed.
2. First-token budget: how long you wait for the first streamed chunk before you abort the turn.
3. Tool budget: how long a single tool may run, including HTTP child calls it makes itself.
4. Total budget: wall-clock limit for one user turn, covering both the model and every tool.
5. Cancel rule: who may abort, and whether abort is cooperative for tools or a hard kill.

Wrap one representative turn with timestamps and record p50 plus p95 for each rung of that ladder. Do not copy a thirty-second folklore value and call the result a policy you can defend. Your refund tool and your search tool have different tails, and the paid SDK was hiding that mix. Write the measured numbers into ladder.yaml so reviewers argue about evidence instead of vibes.

A timeout ladder you can actually test

Label the following adapter as a proposal, not production gospel you paste into a payment path. It fails closed, records why it failed, and refuses to start a tool after the parent turn is cancelled. You should run it against a mock model first, then against a non-paid endpoint used only for dry-run traffic. Keep the circuit beside the ladder, because retries without a breaker recreate the paid SDK's worst habit.

# proposed_timeout_ladder.py — proposed adapter, not a production SDK
from __future__ import annotations

import time
from dataclasses import dataclass
from enum import Enum
from typing import Optional


class CutoverFault(str, Enum):
    CONNECT = "connect_timeout"
    FIRST_TOKEN = "first_token_timeout"
    TOOL = "tool_timeout"
    TOTAL = "total_timeout"
    CANCELLED = "cancelled"
    CIRCUIT_OPEN = "circuit_open"


@dataclass(frozen=True)
class Ladder:
    connect_s: float = 3.0
    first_token_s: float = 8.0
    tool_s: float = 12.0
    total_s: float = 25.0


@dataclass
class TurnClock:
    ladder: Ladder
    started: float
    cancelled: bool = False

    def remaining(self) -> float:
        return self.ladder.total_s - (time.monotonic() - self.started)

    def raise_if_dead(self) -> None:
        if self.cancelled:
            raise TimeoutError(CutoverFault.CANCELLED.value)
        if self.remaining() <= 0:
            raise TimeoutError(CutoverFault.TOTAL.value)


class Circuit:
    def __init__(self, fail_limit: int = 3, cool_s: float = 15.0) -> None:
        self.fail_limit = fail_limit
        self.cool_s = cool_s
        self.fails = 0
        self.opened_at: Optional[float] = None

    def allow(self) -> None:
        if self.opened_at is None:
            return
        if time.monotonic() - self.opened_at >= self.cool_s:
            self.opened_at = None
            self.fails = 0
            return
        raise TimeoutError(CutoverFault.CIRCUIT_OPEN.value)

    def record(self, ok: bool) -> None:
        if ok:
            self.fails = 0
            self.opened_at = None
            return
        self.fails += 1
        if self.fails >= self.fail_limit:
            self.opened_at = time.monotonic()

A 429 storm against a free or cheap host is still a storm you caused during cutover week. Fail closed after a short burst, cool down, then send one probe instead of looping forever. The paid SDK made that loop look like resilience, and your bill made it look like progress. Your new runtime will not invoice you politely for the same mistake.

Numbered cutover plan

Follow these steps in order, because skipping drain is how you double-write tools after the key looks disabled. Each step produces an artifact a reviewer can open without sitting in your terminal session. If a step has no artifact, you are still improvising the cutover in production. Freeze writes early, even if that feels slower than swapping a base URL.

1. Snapshot old SDK timeouts from logs, environment variables, and client constructors, then store them in ladder.yaml.
2. Freeze new agent sessions on the paid runtime, and let current turns finish or abort through the vendor cancel API.
3. Install the ladder in a sidecar that still points at the paid base URL, and compare abort reasons for one quiet day.
4. Add an explicit cancel flag around tool dispatch so a cancelled turn skips every remaining tool, not only the model stream.
5. Point a canary at the new runtime behind the same ladder, and keep the paid key read-only for replay rather than live tools.
6. Drain until in-flight counters hit zero, and only then disable the old key in the vendor console.
7. Keep the leftover list: retry headers, stream heartbeats, and any auto-repeat of failed tools you did not reimplement on purpose.

# ladder.yaml — proposed starting point; replace with measured values
connect_s: 3.0
first_token_s: 8.0
tool_s: 12.0
total_s: 25.0
circuit_fail_limit: 3
circuit_cool_s: 15.0
fail_closed_on_unknown_abort: true
# cancel_sources: browser_drop, operator_abort, total_budget, circuit_open

Print ladder.yaml in the cutover ticket so the review is about numbers rather than a model beauty contest. If someone wants a longer first-token budget, they change a file instead of a folklore comment in chat. If someone wants retries, they have to explain how tools stay idempotent under that retry. That conversation is the migration.

Prove abort actually stops tools

A timeout that logs and then continues is decoration, not a drain plan you can ship. You want a test that starts a slow tool, cancels the turn, and asserts the tool never committed its side effect. Label the suite unexecuted until you wire a real tool behind the same raise_if_dead check. If committed is true after cancel, your cancel path is still a comment.

# test_cancel_drains_tools.py — proposed; wire a real tool before cutover
import threading
import time
import unittest

from proposed_timeout_ladder import Ladder, TurnClock


class FakeTool:
    def __init__(self) -> None:
        self.committed = False
        self.started = threading.Event()

    def refund(self, clock: TurnClock) -> None:
        self.started.set()
        deadline = time.monotonic() + clock.ladder.tool_s
        while time.monotonic() < deadline:
            clock.raise_if_dead()
            time.sleep(0.05)
        self.committed = True


class CancelDrainTest(unittest.TestCase):
    def test_cancel_before_commit(self) -> None:
        clock = TurnClock(Ladder(tool_s=2.0, total_s=5.0), time.monotonic())
        tool = FakeTool()

        def run() -> None:
            try:
                tool.refund(clock)
            except TimeoutError:
                return

        worker = threading.Thread(target=run)
        worker.start()
        self.assertTrue(tool.started.wait(1.0))
        clock.cancelled = True
        worker.join(2.0)
        self.assertFalse(worker.is_alive())
        self.assertFalse(tool.committed)


if __name__ == "__main__":
    unittest.main()

Run it with python -m unittest test_cancel_drains_tools.py before you touch DNS or keys. A slower destination will hit this path more often than a paid low-latency gateway ever did. That is useful pain during canary, not a reason to rip the ladder out. Fix the abort, then move the canary, and only then argue about prompt quality.

Decision table for leftovers

Reviewers argue about models while sockets keep doing the expensive work in the background. Put the leftovers in a table inside the ticket so they cannot hide behind a successful hello-world chat completion. You keep policy you can test, and you drop vendor magic you cannot see. If a row has no owner, that leftover will bill you after the celebration thread.

Leftover from the paid SDK	Symptom after cutover	What you keep	What you drop
Single timeout=30	Hang until a proxy kills the socket	Four-rung ladder	One magic number
Retry on every 429	Duplicate tool side effects	Circuit plus idempotency key	Blind retries
Browser disconnect ignored	Tool finishes long after the tab closed	Abort flag on drop	Fire-and-forget threads
Vendor stream heartbeats	False first-token timeouts	Idle ping versus first token	Treating pings as tokens
Auto-repeat failed tools	Double POSTs into your own API	Explicit replay queue	Hidden SDK replay

Heartbeats deserve a special note because they masquerade as liveness while starving first-token detection. If the vendor sent comment chunks or empty deltas to keep a load balancer happy, your new client may treat silence as death. Teach the first-token rung to ignore pings, and teach the total rung to ignore nothing. Those two rules disagree on purpose, and that disagreement is the policy.

Where a free runtime fits, and where it does not

After the ladder lives in your repo, you still need a destination that will not bill you for abort drills. Dry-run traffic is noisy: you cancel mid-tool, you trip the circuit, and you replay traces until the test is boring. That pattern is a poor use of remaining paid quota, and it is a good use of a sink you can afford to bruise. Keep production tools stubbed while the sink absorbs those bruises.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access and free server option can host that dry-run if you already want a non-paid place to exercise the adapter. Use it as a sink for canary turns and cancel tests, not as a claim that production latency will match the vendor you are leaving. You still own the ladder, the circuit, and the drain checklist. If the free path is slow, that slowness is a gift, because it forces first-token and total budgets to be honest.

Do not treat a free server as an infinite queue for orphaned tools you failed to cancel. Meter concurrent turns, because a missing timeout on your side will fill the box even when the invoice stays at zero. Unmetered free capacity is how leftovers become an outage instead of a line item. The cutover is finished when in-flight work hits zero, not when the new chat UI looks fine.

Limitations, and who should skip this

This adapter does not replace distributed tracing, exactly-once tool semantics, or a durable queue you can inspect after a crash. Threads plus monotonic clocks are a teaching artifact for one process, not a cluster plan. If you run many workers, use a process-wide cancel token and an idempotency store both tools can see. If your tools are not idempotent, no timeout policy will save you from double commits.

You should not use this cutover while a contract still requires production traffic through the paid gateway. You should not use it if you cannot freeze writes for the drain window. You should not point live refunds or payments at a free server while you debug abort behavior. Stub those tools, prove drain, then reattach the dangerous ones behind the same clock.

The approach also assumes you can measure the old SDK instead of guessing from memory of a dashboard screenshot. If you have no logs, you are inventing budgets, and invented budgets will flap under the first slow retrieval call. Spend a day on timestamps before you spend a week retuning prompts for a runtime that is still drowning in zombies. Prompts cannot cancel a refund that already started.

Close the diary

Rebuild the timeout ladder, prove cancel stops tools, and then drain the paid key with counters you can screenshot. The model swap is the easy line in the diff, and it is the line people will want to merge first. The leftover is the work that continued after you thought the session was dead. Copy numbers into ladder.yaml, run the unit test, and only then move the canary off the vendor SDK.

If you need a non-paid sink for those abort drills, you can try MonkeyCode's free model access and free server option as a canary host, then keep the ladder in your repo either way.