Migration Diary: Extract Tool-Choice Policy Before You Leave a Paid Gateway

The paid gateway is still choosing your tools even after you think last week's cutover finished. You should pull tool-choice policy into your own process before production traffic leaves that vendor SDK. A free runtime then looks cheaper while it skips required tools, invents parallel calls, or ignores your deny list. This diary covers a local policy plane, a shadow comparison harness, and leftovers that remain after you move DNS.

The real failure is a missing policy plane

Paid agent SDKs usually hide three important decisions behind a single enum that looks deceptively portable. Those decisions are whether a tool must run, whether several tools may run together, and whether the model may answer with no tool. You copy tool_choice="auto" into the new client and assume those semantics survived the move unchanged. They usually did not, because free endpoints expose a thinner surface: a tool list, perhaps one required name, and no parallel flag.

Your traces then show successful completions that never touched inventory, billing, or the ticket queue you still bill against. You also inherit retry behavior that re-issues a tool after a timeout, which is not a second model decision. If the vendor retried inside its own loop, your new stack will not retry unless you rebuild that loop explicitly. Duplicate charges and missed side effects show up during the first week, not during the demo.

Step 1 — Inventory the decisions the SDK still owns

Export a week of traces and count outcomes before you touch routing, models, or DNS. You want frequencies for forced tools, denied tools, parallel batches, and empty tool lists on each named route. Keep the vendor request id beside your own idempotency key so later leftovers can be drained without guesswork. If a route cannot produce those four counts, freeze that route instead of guessing from a chat window.

# Proposal: rename fields to match your trace export.
jq -r '.route // "unknown"' traces.jsonl | sort | uniq -c | sort -nr
jq -r '.tool_choice // "missing"' traces.jsonl | sort | uniq -c | sort -nr
jq -r '[.tool_calls[]?.name] | join(",")' traces.jsonl | sort | uniq -c | sort -nr
jq -r 'select((.tool_calls | length) == 0) | .route' traces.jsonl | sort | uniq -c

Write four numbers on a card before you change code: required-tool rate, empty-call rate, parallel-call rate, and denied-name rate. If any number is unknown, you do not have a cutover plan yet. You have a hope that the next SDK will guess the same policy, and that hope is already a leftover. Treat missing spans as policy you cannot prove, not as traffic that looked quiet.

Step 2 — Encode policy as data, not as SDK flags

Move the decisions into a table your process owns, then teach every model client to consult it. The table should name the route, the allowed tools, the required tool if any, the deny list, and whether parallel calls are permitted. Keep model selection in a different column so a free runtime cannot widen the tool set by accident. Commit the table in git beside the tests, not in a vendor dashboard that disappears after billing stops.

route	allowed tools	required	deny	parallel	notes
retry_payment	charge_card, fetch_invoice	charge_card	refund_card	no	never parallel with refund
search_docs	search, cite	none	send_email	yes	model may skip tools
close_ticket	close_ticket	close_ticket	delete_ticket	no	forced single call

The required column is the one paid SDKs most often hide behind a friendly default. If your route must call charge_card, a free model that answers in prose is a failed request, not a cheap success. Encode that failure as a local error you can page on, not as a content filter you read by eye. The deny column is equally non-negotiable, because a refund tool that merely "was not mentioned" will still be offered if you pass the full catalog.

Step 3 — Intercept before the model client runs

Place a policy gate in front of every complete() call so the vendor SDK cannot be the last decider. The gate should reject unknown routes, strip denied tools from the payload, and force a required tool name when the table says so. After the model returns, the same gate should reject empty tool lists on required routes and reject parallel batches when the table forbids them. Log the route, the prepared names, and the validated names, because that triple is the evidence you will need during drain.

# Worked example you can adapt. Treat it as unexecuted until you wire your client.
from dataclasses import dataclass
from typing import Optional

@dataclass(frozen=True)
class ToolPolicy:
    route: str
    allowed: frozenset[str]
    required: Optional[str]
    deny: frozenset[str]
    parallel: bool

class PolicyViolation(Exception):
    pass

def prepare_tools(policy: ToolPolicy, requested: list[str]) -> list[str]:
    names = [n for n in requested if n in policy.allowed and n not in policy.deny]
    if policy.required and policy.required not in names:
        names = [policy.required] + [n for n in names if n != policy.required]
    if not names and policy.required:
        raise PolicyViolation(f"{policy.route} requires {policy.required}")
    return names

def validate_calls(policy: ToolPolicy, calls: list[str]) -> None:
    if policy.required and policy.required not in calls:
        raise PolicyViolation(f"{policy.route} missing required tool")
    if not policy.parallel and len(calls) > 1:
        raise PolicyViolation(f"{policy.route} forbids parallel tools")
    blocked = [n for n in calls if n not in policy.allowed or n in policy.deny]
    if blocked:
        raise PolicyViolation(f"{policy.route} blocked {blocked}")

Notice the gate never asks the model whether a refund is a good idea on a payment retry. You already decided refunds are denied on that route, and the model does not get a vote. That is the point of extracting policy: the runtime becomes a completion engine, not an authorization engine. If a helper still builds tool lists inside the old SDK, the gate is theater, and you should delete that helper before any shadow traffic starts.

Step 4 — Shadow a free path without widening tools

Once the gate is in place, send copies of live requests to a second runtime while the paid gateway still serves users. Compare tool-call names, not prose, because summaries will drift even when the policy is correct. Fail the shadow if the free path omits a required tool or emits a denied name, even when the text looks polished. Keep write tools on the paid path until name match and policy errors are boring for a recorded sample.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you need a second completion path for that shadow week, MonkeyCode's free model access and free server option can host the candidate runtime while your gate stays local. Keep the policy table in your repo either way, because a free server should not become the next place that silently owns tool-choice.

# Proposal: score tool names only; do not score refund speeches.
def shadow_report(policy: ToolPolicy, paid_calls: list[str], free_calls: list[str]) -> dict:
    validate_calls(policy, paid_calls)
    free_error = None
    try:
        validate_calls(policy, free_calls)
    except PolicyViolation as exc:
        free_error = str(exc)
    return {
        "route": policy.route,
        "paid": paid_calls,
        "free": free_calls,
        "name_match": paid_calls == free_calls,
        "free_error": free_error,
    }

export SHADOW_MODE=tool_names_only
export SHADOW_DENY_WRITES=1
python -m pytest tests/test_tool_policy.py -q
python -m tools.shadow_compare --in traces.jsonl --out shadow.jsonl
jq -r 'select(.free_error != null) | .route' shadow.jsonl | sort | uniq -c

Do not flip DNS because the free path sounds right in a chat window during lunch. Flip it when required-tool misses stay at zero for a sample you actually recorded, and when denied names never appear. Write that sample size next to the four inventory rates; an uncounted vibe check is how the last SDK captured your policy again. If the free path cannot satisfy the required column, keep that route on the paid gateway and fail closed.

Step 5 — Drain the leftovers after cutover

The leftovers are not the model names you swapped in a config file. They are the vendor flags still sitting in job templates, cron wrappers, generated clients, and copied notebooks. Search the repo for tool_choice, parallel_tool_calls, required_action, and any helper that built tool lists inside the old SDK. Then search dashboards for span names you will stop emitting, because queries that still filter on those names will look like traffic died.

rg -n "tool_choice|parallel_tool_calls|required_action|function_call" -t py -t ts
rg -n "bind_tools|with_structured_output|AssistantAgent|function_map" -t py
rg -n "openai.tools|anthropic.tools|vendor.tool_choice" -t py -t json

Walk this drain list in order and tick each item in the pull request, not in chat.

1. Delete SDK default objects that still inject auto when your table says a tool is required.
2. Remove generated clients that reconstruct tool lists from the vendor dashboard, not from git.
3. Rotate any webhook that posted tool results to a vendor-specific span name you no longer query.
4. Keep the old trace archive long enough to explain a charge, then stop writing new vendor span ids.
5. Re-run the inventory commands from Step 1 against the new runtime until the four rates match the card.

If a leftover still compiles, it still owns a slice of production, even when nobody imported it this week. Treat an unused tool_choice="auto" helper as live policy, because the next refactor will import it under time pressure. The cutover is not the merge that changed the base URL; it is the merge that made the old enum unreferenced.

A small test plan you can run this week

Label these tests as the contract you are migrating; skip them and you are demoing a new endpoint, not leaving a gateway. Run the same fixtures against the paid client and the candidate client, and compare exceptions before you compare paragraphs. You are not scoring fluency on the refund speech; you are scoring whether charge_card happened once.

# tests/test_tool_policy.py — proposal fixtures, not captured production traffic.
import pytest

PAYMENT = ToolPolicy(
    route="retry_payment",
    allowed=frozenset({"charge_card", "fetch_invoice"}),
    required="charge_card",
    deny=frozenset({"refund_card"}),
    parallel=False,
)

SEARCH = ToolPolicy(
    route="search_docs",
    allowed=frozenset({"search", "cite"}),
    required=None,
    deny=frozenset({"send_email"}),
    parallel=True,
)

def test_strips_denied_and_forces_required():
    names = prepare_tools(PAYMENT, ["refund_card", "fetch_invoice"])
    assert names[0] == "charge_card"
    assert "refund_card" not in names

def test_rejects_parallel_on_payment():
    with pytest.raises(PolicyViolation):
        validate_calls(PAYMENT, ["charge_card", "fetch_invoice"])

def test_rejects_prose_only_on_required_route():
    with pytest.raises(PolicyViolation):
        validate_calls(PAYMENT, [])

def test_search_may_skip_tools_but_never_email():
    validate_calls(SEARCH, [])
    with pytest.raises(PolicyViolation):
        validate_calls(SEARCH, ["send_email"])

If the candidate server cannot satisfy the required column, it is not a candidate for that route. Keep the route on the paid gateway until the gate can fail closed without human rereading of transcripts. Store failing shadow_report rows beside the table change that supposedly fixed them, or you will rediscover the same denied name after the next prompt edit.

Limitations and who should skip this

This approach assumes you can name routes and tools up front, which chat-with-anything prototypes cannot honestly do. If a regulator requires the vendor's hosted classifier on every tool argument, a local table is not a substitute, and you should stay on that contract. Teams without trace export will also fail here, because you cannot inventory rates you never stored, and a policy plane without rates is another hidden enum.

Do not use this diary to justify dropping human approval on money movement or account deletion. A policy plane can deny refund_card on a retry route and still be wrong about who may charge a card. Keep approvals, idempotency keys, and ledgers outside the model client, exactly as they should have been when the paid SDK still looked convenient. Skip the free-server shadow entirely when your tools are not idempotent and you cannot deny writes in configuration.

The cutover is done when the policy table, not the vendor enum, is the only decider in the path. After that, a free model and a free server are just another pair of workers behind the same gate. Keep the tests above in git first so the next leftover is a failing assertion, not a silent skipped charge.