Migration Diary: Freeze a Golden Task Fixture Before You Leave a Paid Coding Agent

You should freeze a small golden task fixture before you cancel any paid coding agent subscription. Paid coding agents teach your team what done looks like through chat transcripts rather than through executable checks. When that chat window finally disappears, you lose the only acceptance surface your team actually used every day. A frozen fixture turns those unspoken expectations into files you can rerun on cheaper local or remote infrastructure.

The problem you actually have on cutover day

You probably already exported chats, workspace maps, and tool logs from the outgoing product. Those artifacts explain history, but they do not tell a new stack whether a refactor still counts as complete. The paid agent also absorbed style rules, review nits, and file-ban instincts that never landed in git. Your cutover plan therefore needs a tiny, boring suite that fails loudly when the destination agent drifts.

A golden task fixture is not a public benchmark leaderboard and not a model beauty contest. It is a checked-in set of prompts, inputs, and assertions that represent work you already shipped. You rerun the same records after you change vendors, models, servers, or wrapper scripts. If the fixture stays green, you can cancel the old seat without guessing about quality.

What belongs in the fixture, and what does not

Keep each record small enough that a human can review the diff in one sitting. Prefer tasks that already exist in your repository: a flaky test, a nil-check refactor, or a README correction. Avoid puzzles that reward cleverness, because cleverness is not what you paid the old agent to produce during ordinary weeks. Three to five honest records expose more destination drift than a sprawling suite you will not maintain.

Label every example below as a proposed layout, not as a run executed in your environment. You should copy the files into a throwaway branch and replace the sample paths with paths you actually own. Do not paste production secrets into prompts, and do not freeze tasks that require customer data you cannot reset. The fixture is an acceptance surface, not a second source repository.

Proposed fixture schema

# fixtures/golden_tasks.yaml
version: 1
tasks:
  - id: repair-nil-guard
    title: Add a nil guard without widening the diff
    prompt: |
      In internal/cache/client.go, return a typed error when the client is nil.
      Do not reformat the rest of the file. Do not touch tests unless they fail.
    cwd: .
    timeout_seconds: 180
    expect:
      files_allowed:
        - internal/cache/client.go
      files_forbidden_prefix:
        - vendor/
        - .git/
      stdout_must_not_match:
        - "TODO"
        - "as an AI language model"
      commands:
        - go test ./internal/cache -count=1
  - id: docs-only-changelog
    title: Update CHANGELOG without code churn
    prompt: |
      Add a one-paragraph Unreleased note for the cache nil-guard.
      Do not modify Go files.
    cwd: .
    timeout_seconds: 90
    expect:
      files_allowed:
        - CHANGELOG.md
      commands:
        - git diff --check

That schema encodes three things the chat window never encoded for you. It names the prompt, it names the allowed blast radius, and it names a command that already exists in your project. You can add more fields later, but you should not start with a dozen tasks. Extra fields without a resettable input tree will only give you flaky red builds.

Numbered cutover workflow

Follow these steps in order and write down the leftovers as you go. Skipping the freeze step is how teams rediscover missing rules two weeks after the invoice stops. The destination stack is not the first problem; the missing definition of done is the first problem.

1. Mine last week’s closed work for candidate tasks

Open your merge history and pick three changes an agent actually drafted or repaired. Write one sentence for each change that a newcomer could follow without the old chat. Discard anything that required a private dashboard, a vendor plugin, or a secret the destination stack cannot see. If you cannot restate the task in one sentence, it is not golden yet.

# proposed commands; run them in a clone you control
git log --since='7 days ago' --merges --oneline
git log --since='7 days ago' --name-only --pretty=format:'%h %s'

2. Freeze inputs next to the prompt

Copy the pre-change files into fixtures/inputs/<task-id>/ so the runner can reset state. You want the destination agent to start from the same broken tree every time. If you only keep the prompt, you will test memory rather than behavior, and memory is the first leftover the paid product will not give back. Resettable inputs are the difference between a diary and a test.

mkdir -p fixtures/inputs/repair-nil-guard
git show HEAD~1:internal/cache/client.go > fixtures/inputs/repair-nil-guard/client.go

3. Write a runner that never calls the old vendor

The runner should apply the frozen input, invoke whatever local agent command you now use, then assert files and tests. Keep vendor names out of the runner so you can swap destinations without editing assertions. Treat AGENT_CMD as a socket, not as a brand, and keep logs next to the fixture so failures stay reviewable.

# fixtures/run_golden.py
# Proposed runner. Adapt AGENT_CMD to your local wrapper.
from __future__ import annotations

import os
import subprocess
import sys
from pathlib import Path

import yaml

ROOT = Path(__file__).resolve().parents[1]
FIXTURE = ROOT / "fixtures" / "golden_tasks.yaml"


def run(cmd: list[str], cwd: Path, timeout: int) -> subprocess.CompletedProcess[str]:
    return subprocess.run(
        cmd,
        cwd=cwd,
        text=True,
        capture_output=True,
        timeout=timeout,
        check=False,
    )


def reset_inputs(task: dict) -> None:
    src = ROOT / "fixtures" / "inputs" / task["id"]
    if not src.exists():
        return
    for path in src.rglob("*"):
        if path.is_file():
            rel = path.relative_to(src)
            dest = ROOT / rel
            dest.parent.mkdir(parents=True, exist_ok=True)
            dest.write_bytes(path.read_bytes())


def assert_task(task: dict) -> None:
    expect = task["expect"]
    diff = run(["git", "diff", "--name-only"], ROOT, 30)
    changed = [line for line in diff.stdout.splitlines() if line]
    allowed = set(expect.get("files_allowed", []))
    extra = [path for path in changed if path not in allowed]
    if extra:
        raise SystemExit(f"{task['id']}: unexpected files {extra}")
    for command in expect.get("commands", []):
        result = run(command.split(), ROOT, task.get("timeout_seconds", 120))
        if result.returncode != 0:
            raise SystemExit(
                f"{task['id']}: command failed: {command}\n{result.stdout}\n{result.stderr}"
            )


def main() -> None:
    doc = yaml.safe_load(FIXTURE.read_text())
    agent_cmd = os.environ.get("AGENT_CMD")
    if not agent_cmd:
        raise SystemExit("Set AGENT_CMD to your destination wrapper")
    for task in doc["tasks"]:
        reset_inputs(task)
        invoke = run(
            agent_cmd.split() + [task["prompt"]],
            ROOT,
            task.get("timeout_seconds", 180),
        )
        log_path = ROOT / "fixtures" / "logs" / f"{task['id']}.txt"
        log_path.parent.mkdir(parents=True, exist_ok=True)
        log_path.write_text(invoke.stdout + "\n" + invoke.stderr)
        assert_task(task)
        print(f"PASS {task['id']}")


if __name__ == "__main__":
    main()

4. Score leftovers the paid agent still holds

Your fixture will fail for reasons that are not model quality. Those failures are leftovers, and you should inventory them before you blame the new stack. Use a table so the team argues about evidence instead of vibes. Every red cell is a migration task, not a reason to keep paying by default.

Leftover	How it shows up in the fixture	Cutover action
Hidden custom instructions	Destination rewrites style or adds comments you never asked for	Copy rules into a repo file the new agent reads
Unexported memory	Destination forgets file bans the old chat treated as obvious	Add files_forbidden_prefix and a short project note
Vendor-only tools	Destination cannot search docs or open the ticket	Drop that task from the golden set or replace the tool
Secret-backed calls	Destination hangs on an API the old agent injected	Move credentials to your own env; never store them in chat
Review nits as folklore	Tests pass but humans reject the diff	Add a git diff size check or a lint command

If a leftover cannot be expressed as a file, a command, or an env var, it was never a durable part of your workflow. Write that down in the cutover notes so nobody tries to recreate folklore inside a new vendor’s settings panel. Folklore is what you are leaving, not what you should port.

5. Cut over only when the fixture stays green twice

Run the suite against the destination stack on two clean checkouts after a full file reset. One green run can be luck, especially when the agent samples different continuations. Two greens on reset inputs are enough to cancel a seat for a small team. Keep the old product read-only for a week if your vendor allows it, and use that week only to chase leftovers, not to start new work.

python3 -m venv .venv
. .venv/bin/activate
pip install pyyaml
export AGENT_CMD='your-local-agent --non-interactive --apply'
python fixtures/run_golden.py
git checkout -- .
python fixtures/run_golden.py

Where a free destination stack fits

Once the fixture lives in git, you can point AGENT_CMD at any wrapper that edits the tree and prints a log. That is the moment a free coding-agent option becomes useful instead of merely cheaper. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that, per the operator, offers free model access and a free server option you can aim a runner at while you drain the paid seat.

You should not treat that free access as a permanent capacity plan or as a published quota. Wire it as one destination behind the same assertions you would use on a laptop. If the golden tasks pass there, you learned something about your workflow rather than about a brochure. If they fail, you still have the leftover table, which is more valuable than an unsupported comparison.

If you already keep the fixture in git, try the same runner against MonkeyCode on a throwaway branch and record only the leftover rows that change.

Limitations you should accept in writing

This approach does not measure latency, token cost, or code elegance in a way you could publish. Sampling agents can pass on Tuesday and fail on Wednesday with the same prompt, so two greens are a cutover bar, not a scientific result. The proposed runner also assumes you can reset files and run project tests without network credentials the old vendor used to inject. If your tests need those credentials, fix that leak before you change agents.

Do not put production secrets into fixture prompts, even when the destination calls itself free. Do not freeze tasks that require customer data you cannot synthesize. Do not expand the suite until the first three records stay boring for a full week. A large, flaky golden set will trap you on the paid product longer than having no suite at all.

Who should skip this diary

Skip the freeze if you only used the paid agent as a chat toy and never merged its diffs. Skip it if your repository cannot run tests offline, because the assertions will lie about destination quality. Skip it if policy forbids sending any snippet to a free or remote model, in which case you need an air-gapped wrapper before you need a vendor change. Everyone else should freeze the fixture first and argue about models second.

The leftover that hurts most is not a missing transcript from the old product. It is the missing definition of done, and you can check that definition into git this afternoon. After that, cancelling the seat is a billing event, not a guess.