Replay the Patch, Not the Chat: A Take-Home for Agent Sessions

The deliverable is a patch pipeline, not a transcript

AI coding agents produce persuasive sessions and unreproducible results. A candidate who submits a chat log has demonstrated nothing that a reviewer can re-run. The exercise below asks for something narrower and far more useful: a recorded tool trace plus a replay harness that rebuilds the final file states on a clean checkout.

Two constraints keep the task honest and comparable across candidates. The first is that no paid inference and no paid compute may appear in the submitted workflow, because a take-home should not depend on a candidate's credit card. The second is that every claimed fix must survive three independent replays from the pinned fixture, or it counts as unverified.

The interview brief

Scenario

A reviewer hands the candidate a small repository with one failing regression test and a written description of the desired behavior. The candidate may use an AI coding agent freely to reach a fix, but must hand back evidence rather than narration.

Constraints

1. The agent session must run against free model access, so the environment is reproducible for reviewers without a budget.
2. Any server-side step must run on a free server option rather than a candidate-provisioned machine.
3. The fixture is pinned by commit SHA, and the trace must record that SHA alongside every tool call.
4. No secrets, tokens, or proprietary paths may appear in the submitted trace.

Deliverables

• trace/toolcalls.jsonl — one JSON object per tool call, in execution order.
• harness/ — recorder, replay runner, and determinism scorer.
• tests/ — at minimum one test that fails before the fix and passes after it.
• README.md — what was fixed, what was assumed, and what the harness cannot prove.

MonkeyCode is relevant here mainly as the cheapest way to satisfy the first two constraints. Its free model access gives the exploration loop a home, and its free server option removes the need for a candidate-owned box. The allowance and the available models change over time, so verify the current terms before you write them into a rubric.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What the trace must contain

A trace is only replayable if each entry describes an effect on the filesystem or a process invocation. Conversational text belongs in a separate log and never in the trace.

Each line should carry these fields:

• kind — one of read, write, exec, test.
• payload — the concrete arguments, such as a target path and full content, or an argv array.
• files — a map of touched paths to short content digests at that moment.
• fixture_sha — the pinned commit the call was made against.

# harness/record.py
import hashlib, json, pathlib, time

TRACE = pathlib.Path("trace/toolcalls.jsonl")

def digest(p):
    return hashlib.sha256(pathlib.Path(p).read_bytes()).hexdigest()[:16]

def record(kind, payload, fixture_sha, files=()):
    entry = {
        "t": round(time.time(), 3),
        "kind": kind,
        "fixture_sha": fixture_sha,
        "payload": payload,
        "files": {str(f): digest(f) for f in files},
    }
    with TRACE.open("a") as fh:
        fh.write(json.dumps(entry, sort_keys=True) + "\n")
    return entry

The rubric

Weight	Criterion	What the reviewer inspects
20%	Trace completeness	Every write and exec has a corresponding entry
20%	Reproducibility	Three replays yield identical file digests
15%	Failure surfacing	Non-zero exit and a diff when replay diverges
15%	Constraint discipline	No paid inference or paid compute in the path
15%	Test quality	A failing test exists before the patch is applied
15%	Written limits	README states what the harness does not prove

A submission that scores well on reproducibility but omits the failing test is still a weak submission. The rubric deliberately rewards the smallest defensible fix, not the largest plausible refactor.

A reference solution

The layout below is deliberately small enough to review in one sitting. It separates recording, replay, and scoring so each stage can fail independently.

agent-handover/
  fixture/                 # pinned checkout
  trace/toolcalls.jsonl
  trace/fixture.sha
  harness/record.py
  harness/replay.py
  harness/score.py
  tests/test_regression.py
  README.md

# harness/replay.py
import json, pathlib, shutil, subprocess, sys

def load_trace(path):
    lines = pathlib.Path(path).read_text().splitlines()
    return [json.loads(line) for line in lines if line.strip()]

def apply_entry(entry, workdir):
    if entry["kind"] != "write":
        return
    target = pathlib.Path(workdir) / entry["payload"]["path"]
    target.parent.mkdir(parents=True, exist_ok=True)
    target.write_text(entry["payload"]["content"])

def run_commands(trace, workdir):
    for entry in trace:
        if entry["kind"] == "exec":
            subprocess.run(entry["payload"]["argv"], cwd=workdir, check=True)

def main(trace_path, out_dir):
    trace = load_trace(trace_path)
    shutil.rmtree(out_dir, ignore_errors=True)
    shutil.copytree("fixture/", out_dir)
    for entry in trace:
        apply_entry(entry, out_dir)
    run_commands(trace, out_dir)

if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2])

# harness/score.py
import hashlib, pathlib, sys

def digest(path):
    return hashlib.sha256(path.read_bytes()).hexdigest()

def collect(root):
    root = pathlib.Path(root)
    return {str(p.relative_to(root)): digest(p)
            for p in root.rglob("*") if p.is_file() and ".git" not in p.parts}

def score(runs):
    snaps = [collect(r) for r in runs]
    keys = set.intersection(*(set(s) for s in snaps))
    stable = [k for k in keys if len({s[k] for s in snaps}) == 1]
    return len(stable) / max(len(keys), 1)

if __name__ == "__main__":
    value = score(sys.argv[1:])
    print(f"determinism score: {value:.2f}")
    sys.exit(0 if value == 1.0 else 1)

The run sequence is three replays plus a score, and the score doubles as the CI gate.

git -C fixture rev-parse HEAD > trace/fixture.sha
python harness/replay.py trace/toolcalls.jsonl replay/1
python harness/replay.py trace/toolcalls.jsonl replay/2
python harness/replay.py trace/toolcalls.jsonl replay/3
python harness/score.py replay/1 replay/2 replay/3

Normalize before you compare

Raw traces are noisy for reasons that have nothing to do with the fix. Normalize these sources of drift first, or the determinism score will be meaningless.

• Absolute paths and timestamps, which differ per machine and per run.
• Model sampling, which changes wording but should not change final file content.
• Tool ordering, when two writes touch disjoint files and have no dependency.
• Network responses, which should be captured once and replayed from disk.

The rule of thumb is simple: compare semantic effects on files, not the narrative that produced them.

Common failure modes

1. Recording prose instead of tool calls, which makes replay impossible and the trace unreadable.
2. Comparing chat transcripts across runs and calling that verification, which proves only that the model was verbose.
3. Pinning the fixture commit while leaving environment variables unpinned, so a fix passes locally and fails in review.
4. Reporting a partial score as success; a determinism score of 0.7 means at least one artifact is not reproducible.
5. Applying the patch by hand before replay, which hides the exact step the harness was meant to verify.
6. Writing no failing test first, which leaves the reviewer unable to distinguish a fix from a coincidence.
7. Copying credentials into the trace, which turns a tidy take-home into an incident report.

Limitations and who should skip this

This harness verifies that a patch is reproducible, not that it is correct. Flaky tests, clock-dependent behavior, and hardware-specific bugs will still escape it, and the scorer reports stability rather than quality. Teams with strict regulatory requirements should not route proprietary source through any hosted model tier, free or otherwise.

It is also a poor fit for agents whose value lies in long-horizon planning across many repositories, because a single pinned fixture cannot express that work. If the deliverable is a design document rather than a patch, this rubric will reward the wrong behavior.

Closing

The point of the exercise is that reviewers should be able to run one command and see the same repository state the candidate saw. Everything else, including the model and the machine, is replaceable. If you want to try the brief yourself, the free model access and free server option on MonkeyCode are enough to run the exploration loop, and python harness/score.py will tell you honestly whether you are done.