Inventory the Agent Assumptions Before You Leave a Paid Runtime

You should not cut over an agent workflow until you have listed every assumption the old runtime made for you. Paid coding agents hide timeouts, tool schemas, retry rules, and file access behind a smooth editor experience. Those defaults become production bugs the moment you move jobs onto a free model path and a free server. This diary walks you through an inventory, a cutover plan, and a leftovers register you run before deleting the old key.

The paid runtime was never a thin HTTP client

You probably treat the old coding agent as a model endpoint with a nicer window around it. The editor injects workspace roots, default tools, secret redaction, and retry wrappers you never wrote. When you leave that product, those wrappers do not travel with your saved prompt files at all. Your new path only sees what you explicitly send, which is usually less than you think.

What you are extracting, not rewriting

This cutover is an extraction problem, not a prompt rewrite contest against the old vendor. You keep the same job, the same fixture files, and the same pass or fail assertions. You change only the transport, the tool host, and the process where retries now live. If a job passes on the old agent and fails on the new path, you treat that as a missing assumption.

Step 1 — Capture the silent contract as YAML

Start by dumping the hidden contract into YAML so the cutover is not a hallway conversation. You want fields the old product filled in silently, not fields you already store in git. Treat the next block as a proposed inventory template, not as a dump from a live vendor account.

# proposed inventory: agent_assumptions.yml
old_runtime:
  name: paid-coding-agent
  observed:
    request_timeout_ms: 120000
    idle_stream_timeout_ms: 30000
    max_tool_calls_per_turn: 8
    parallel_tools: true
    workspace_root: ${REPO_ROOT}
    default_cwd: ${REPO_ROOT}
    env_passthrough:
      - PATH
      - HOME
      - GIT_DIR
    secrets_redacted_from_prompts: true
    retries:
      on_status: [429, 502, 503]
      max_attempts: 3
      backoff_ms: [500, 2000, 8000]
    sampling:
      temperature_if_unset: 0.2
      max_output_tokens_if_unset: 4096
    tools:
      - name: read_file
      - name: apply_patch
      - name: run_command
new_runtime:
  name: self-hosted-job-runner
  must_set_explicitly:
    - request_timeout_ms
    - max_tool_calls_per_turn
    - default_cwd
    - retry_policy
    - sampling_defaults

Numbered capture steps

1. Open one real job you already trust, not a demo prompt from a vendor gallery.
2. Log timeout, tool names, cwd, and retry behavior while that trusted job actually runs.
3. Write each observed value into agent_assumptions.yml with a comment about the log line.
4. Mark every field you did not observe as unknown, because unknown is safer than a guessed default.

You should refuse to invent values for fields you cannot see in logs or settings. Guessed timeouts are how cutovers pass a local demo and then hang in CI overnight. Keep unknown in the file until a log line replaces it with a measured number. A YAML file full of guesses is not an inventory, and you should not cut over on it.

Step 2 — Freeze a golden job fixture

You need one job that does not depend on the old editor UI in any way. The fixture should read a file, propose a patch, and run a command whose output you can hash. Store inputs and expected hashes in git so both runtimes must answer the same question. If the job needs a mouse click inside the old IDE, it is not a fixture yet.

# proposed layout
fixtures/agent_job_01/
  prompt.md
  workspace/
    src/tax.py
    tests/test_tax.py
  expected/
    tool_trace.json
    test_hash.txt

# fixtures/agent_job_01/prompt.md
You are fixing a tax rounding helper in src/tax.py.
Read the file, apply the smallest patch that makes tests/test_tax.py pass, then run pytest.
Do not touch files outside this workspace. Stop after the tests pass.

# fixtures/agent_job_01/workspace/src/tax.py
# proposed fixture code, not production tax logic

def cents_to_dollars(cents: int) -> str:
    # intentional bug: integer division loses the remainder display
    dollars = cents // 100
    return f"{dollars}.00"

# fixtures/agent_job_01/workspace/tests/test_tax.py
from src.tax import cents_to_dollars

def test_change():
    assert cents_to_dollars(199) == "1.99"

Run the fixture once on the old runtime and save the tool trace you actually observed. Use a stripped environment so you can see which variables the vendor was injecting for you.

# proposed commands; replace the old-agent CLI with yours
mkdir -p traces/old traces/new
/usr/bin/time -f "elapsed_sec=%e max_rss_kb=%M" \
  env -i PATH="$PATH" HOME="$HOME" \
  old-agent run --prompt fixtures/agent_job_01/prompt.md \
  --cwd fixtures/agent_job_01/workspace \
  > traces/old/stdout.txt 2> traces/old/meta.txt
sha256sum fixtures/agent_job_01/workspace/src/tax.py \
  > traces/old/src_hash.txt

If your old agent has no CLI, record the trace from its log panel and paste it into traces/old/. The point is a frozen artifact you can diff later, not a perfect vendor integration. Do not edit the fixture after the first green run, or the two runtimes will answer different questions.

Step 3 — Replay the fixture on the new path

Now you rebuild the missing wrappers as your code, instead of hoping the new host copies them. If you need a place to run the harness outside your paid editor, MonkeyCode offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. You still must copy timeouts, tool caps, and cwd from the YAML, because the free path will not guess them.

# proposed file: replay_job.py
"""Replay one golden agent job with explicit assumptions. Unexecuted example."""
from __future__ import annotations

import json
import os
import subprocess
import time
from pathlib import Path
from typing import Any

import yaml


class AssumptionError(RuntimeError):
    pass


def load_assumptions(path: Path) -> dict[str, Any]:
    data = yaml.safe_load(path.read_text())
    required = [
        "request_timeout_ms",
        "max_tool_calls_per_turn",
        "default_cwd",
    ]
    observed = data["old_runtime"]["observed"]
    for key in required:
        if observed.get(key) in (None, "unknown"):
            raise AssumptionError(f"refuse to replay until {key} is measured")
    return data


def run_command_tool(cwd: str, argv: list[str], timeout_s: float) -> dict[str, Any]:
    started = time.monotonic()
    try:
        proc = subprocess.run(
            argv,
            cwd=cwd,
            capture_output=True,
            text=True,
            timeout=timeout_s,
            env={"PATH": os.environ.get("PATH", "")},
        )
        return {
            "tool": "run_command",
            "cwd": cwd,
            "argv": argv,
            "returncode": proc.returncode,
            "stdout": proc.stdout[-4000:],
            "stderr": proc.stderr[-4000:],
            "elapsed_ms": int((time.monotonic() - started) * 1000),
        }
    except subprocess.TimeoutExpired as exc:
        return {
            "tool": "run_command",
            "cwd": cwd,
            "argv": argv,
            "returncode": "timeout",
            "stdout": (exc.stdout or "")[-4000:],
            "stderr": "timeout",
            "elapsed_ms": int(timeout_s * 1000),
        }


def cap_tool_calls(trace: list[dict[str, Any]], max_calls: int) -> None:
    if len(trace) > max_calls:
        raise AssumptionError(
            f"tool trace hit {len(trace)} calls; inventory cap is {max_calls}"
        )


def main() -> None:
    root = Path(__file__).parent
    assumptions = load_assumptions(root / "agent_assumptions.yml")
    observed = assumptions["old_runtime"]["observed"]
    cwd = str((root / "fixtures/agent_job_01/workspace").resolve())
    if cwd != str(Path(observed["default_cwd"]).resolve()) and observed["default_cwd"] != "${REPO_ROOT}":
        raise AssumptionError("fixture cwd does not match inventory default_cwd")
    timeout_s = observed["request_timeout_ms"] / 1000.0
    # Proposed: inject your model client here. Do not hard-code a vendor name.
    # The runner only enforces the old contract around whatever client you pass in.
    trace: list[dict[str, Any]] = []
    result = run_command_tool(cwd, ["pytest", "-q"], timeout_s)
    trace.append(result)
    cap_tool_calls(trace, observed["max_tool_calls_per_turn"])
    out = root / "traces/new/tool_trace.json"
    out.parent.mkdir(parents=True, exist_ok=True)
    out.write_text(json.dumps(trace, indent=2))
    print(f"wrote {out}")


if __name__ == "__main__":
    main()

python3 -m pip install pyyaml pytest
python3 replay_job.py

You should fail the replay when a field is still unknown, not when the model writes a weak patch. Contract failures and model quality failures must stay in separate buckets or you will tune the wrong layer. A timeout in run_command_tool belongs in the contract bucket even if the model later produces a correct patch. Keep those failures in different CI jobs so a flaky sample does not hide a missing cwd.

Step 4 — Diff traces, then set cutover gates

A cutover gate is a test you can run twice, once on each runtime, with the same fixture hash. You are not looking for identical prose from the model, because that match will not happen. You are looking for the same tools, the same cwd, and the same timeout envelope. If the new path invents an extra run_command, you have not ported the tool cap.

# proposed file: diff_traces.py
"""Compare old and new tool traces. Unexecuted example."""
from __future__ import annotations

import json
from pathlib import Path


def load(path: str) -> list[dict]:
    return json.loads(Path(path).read_text())


def tool_names(trace: list[dict]) -> list[str]:
    return [event.get("tool") or event.get("name") for event in trace]


def main() -> None:
    old = load("traces/old/tool_trace.json")
    new = load("traces/new/tool_trace.json")
    failures = []
    if tool_names(old) != tool_names(new):
        failures.append(f"tool order {tool_names(old)!r} != {tool_names(new)!r}")
    old_cwd = {event.get("cwd") for event in old if "cwd" in event}
    new_cwd = {event.get("cwd") for event in new if "cwd" in event}
    if old_cwd and old_cwd != new_cwd:
        failures.append(f"cwd {old_cwd!r} != {new_cwd!r}")
    if failures:
        raise SystemExit("\n".join(failures))
    print("trace gates passed")


if __name__ == "__main__":
    main()

python3 diff_traces.py

Cutover decision table

Gate	Pass condition	Fail meaning	Action
Old timeout observed	YAML field is a number	You are guessing	Do not cut over
Tool order matches	Names and sequence equal	Missing tool wrapper	Port the tool, not the prompt
Cwd matches	Both traces use workspace root	Host defaulted to $HOME	Set cwd in the runner
Retry budget matches	Same max_attempts on 429	New path retries forever	Copy the backoff list
Pytest hash matches	Matches expected/test_hash.txt	Model or patch quality	Keep out of the contract bucket
Unknown fields = 0	No unknown left in YAML	Silent default still exists	Inventory another log

Numbered cutover sequence

1. Keep the paid runtime as the source of truth until every gate is green twice.
2. Run the fixture on the new path in shadow mode for a fixed batch of jobs.
3. Compare traces with diff_traces.py and file one leftover per mismatch you cannot yet delete.
4. Cut read-only jobs first, then jobs that apply patches, then jobs that run commands.
5. Revoke the old key only after the leftovers register has an owner for every open row.

Shadow mode means the new path writes traces, but the paid runtime still applies the patch. You should not flip that switch because a demo looked fluent in the editor. Fluency is not a gate, and it will not tell you that GIT_DIR disappeared on the new host.

Step 5 — Record leftovers instead of deleting them

Leftovers are editor-injected behaviors the old product still owns after your HTTP cutover already looks clean. They are not bugs in the model, and they are not items you should hide in a Slack thread. Put them in a register that survives the cutover, because someone will debug them after the vendor key is gone. Each row should name the assumption, not the symptom, so you do not keep patching prompts.

Leftover	Why it stayed	Owner	Burn-down
Editor secret redaction	New runner prints env on failure	platform	wrap logs
Parallel tool calls	New path is serial only	agent runtime	queue tools
Idle stream timeout	Batch job has no stream	ci	set wall clock
Default temperature 0.2	Client sends nothing	prompts	set explicitly
GIT_DIR passthrough	New host has no .git	eval jobs	skip git tools

You should keep the paid product available as a rollback until those rows are closed or accepted. Accepted leftovers need a comment in the runner so the next person does not "fix" them back. A leftover you accept is still part of the contract, and it belongs in agent_assumptions.yml beside the measured fields. If you cannot name an owner, you are not ready to revoke the old key.

Limitations

This method only works when you can observe the old runtime through logs, a CLI, or a trace panel. If the vendor hides tool calls, you cannot inventory them, and the YAML will be fiction. The fixture proves contract parity, not that free models match paid quality on every future task. Latency, queueing, and capacity on a free server will differ, and this diary does not benchmark them.

Do not treat a green diff_traces.py as permission to drop human review on patch-applying jobs. The tax helper is a toy, and a real workspace will expose path traversal and secret files. You still need your own sandboxing, which is a separate contract from this assumption inventory. Nothing here measures token spend, and you should not infer quotas from a single pytest hash.

Who should not use this approach

Skip this if you cannot freeze a fixture because every job needs a live customer workspace. Skip this if policy forbids sending code to a free model or to a shared free server. Skip this if you need a vendor SLA, because a free path is a lab, not a contract. Skip this if you were about to cut over without measuring timeouts, because the YAML will be guesswork.

Teams with regulated data should keep eval jobs on infrastructure they already control and can audit. Teams without a rollback key should not start this diary, because the first missing assumption will land in production. If your agent cannot run without the old editor's secret scanner, inventory that scanner as a leftover before you move any job.

After the leftovers have owners

Once the register has owners, the paid editor is optional for that job class, not for your whole company. Move one job class at a time, and keep the inventory file in the same pull request as the runner. If that split matches your cutover, try MonkeyCode's free models and free server after you finish the inventory, not before.