I want to know whether a change to my coding-agent setup actually helps. But in everyday development, each issue is different. Completion times and successful runs alone tell me little about whether a new model, harness, or set of instructions is better.
I tried turning one past development task into an evaluation case: give a fresh agent session the original requirements and starting source, then check the implementation it produces.
The useful discovery was how much of that starting point I already had. My saved job contained the initial requirements and starting commit SHA. I could extract the task and retrieve the source from Git without building an agent to reconstruct missing files.
This article is adapted from my Japanese article on Zenn.
What I borrowed from Trace2Eval
The experiment was inspired by Trace2Eval, a paper about constructing coding-agent evaluations from real user interactions.
Its reconstruction process recovers three inputs: the user's task, the workspace before the interaction, and the initial execution environment. A candidate agent receives that context and produces an artifact for evaluation.
For files, the process uses content observed in the trace and infers or retrieves what is missing. It excludes artifacts created during the session and distinguishes dependencies that existed initially from those installed during execution.
I borrowed the idea of preparing those three inputs, but adapted the method to my own records:
| Input | What I used |
|---|---|
| Requirements | The initial job saved by my development system |
| Starting source | The Git tree at the job's recorded starting SHA |
| Execution environment | A runnable approximation based on observed versions and the lockfile |
This is an independent implementation inspired by the paper, rather than a reproduction of its full pipeline. I did not refer to the authors' implementation code.
I also simplified two substantial parts. I omitted the loop that replays original tool calls, compares their results, and repairs the recovered environment. For scoring, I implemented task-specific behavioral checks instead of adopting the paper's rubric and LLM-judge approach.
The code is in spike-trace2eval.
The task I chose
My development system, coding-worker, retains requirements, starting commits, tool events, patches, and verification output.
I selected a run that implemented yuurei issue #195. yuurei is my tool for running and recording coding-agent experiments.
The task was to support a task body supplied as a string:
- Execute it without creating a task file inside the project.
- Produce the same task digest for identical content, whether supplied through a file or as a string.
The requirements came from the issue body, acceptance criteria, and constraints saved at the start of the original run.
I deliberately chose a tractable case. It had retained initial requirements and a starting SHA, limited external-service dependencies, and behavior I could check by executing the CLI.
“Solving it again” meant implementing those requirements in a fresh session from the original source. The new patch did not have to match the old one.
The input was more than a chat log
I used the saved records for one execution:
| Record | Purpose |
|---|---|
| Initial job | Recover the issue body, acceptance criteria, constraints, check commands, budget, and starting SHA |
| Event log | Inspect the initial prompt, tool operations, and environment observations |
| Manifest | Cross-check the run identity and starting SHA |
| Checksum list | Check consistency of the saved evidence files |
| Original patch and verification output | Evaluator-side references, excluded from the new agent's input |
The event log contained 6,588 events and 85 distinct tool operations from one session. I also used the target repository's Git data.
That distinction matters: the log did not contain a complete source snapshot. The starting SHA let me retrieve that snapshot from Git.
The raw trace, full initial prompt, and patches are private. The public repository contains the recovery and evaluation code, synthetic tests, and reports with limited evidence detail.
Building the evaluation inputs
The following describes the current implementation, including safeguards added after the first real run. I separate those later changes from the historical result below.
Extract the initial task
The pipeline cross-checks the run identity and starting SHA across the job, manifest, and event log. It also checks event sequences, tool-call/result pairing, and evidence checksums.
Requirements are extracted mechanically from the initial job, with a source reference for each field. The issue body, for example, comes from job.issue.body.
I also regenerated the initial prompt from the saved job and the historical template, then checked that it matched the prompt recorded in the event log.
I chose not to summarize the full trace into a new task description. The trace also contains investigation, failed attempts, edits, and the final answer. Rewriting the task from all of that could introduce hints unavailable at the start.
Here, the original requirements were already present. Extracting them directly made their provenance easier to inspect.
Retrieve the starting source
The pipeline retrieves only the recorded starting commit into a new directory. It checks that HEAD matches the pinned SHA, history contains one commit, there are no remotes, and the working tree is clean.
The current implementation also removes and checks Git metadata that could retain the operator's local paths.
This recovers the Git-tracked source. It cannot recover unrecorded local files.
Prepare an approximate environment
The original log showed Node 26.8.2 and pnpm 11.6.0, while the starting repository configuration declared Node 24.
I selected the observed versions: Node 26.8.2 and pnpm 11.6.0. The environment record keeps observations, selected values, unknowns, and deviations separate.
Dependencies are installed from the starting commit's lockfile. Generated dependencies such as node_modules are kept distinct from the starting source. Setup and existing checks run in a separate working copy to confirm that the selected environment is usable.
The exact original OS version, full PATH and environment variables, internal harness instructions, and automatically injected context were unavailable. I treated the result as a documented approximation.
The environment documentation records the selection.
Limit the agent's input package
The package has four entries:
| Input | Contents |
|---|---|
task.md |
Requirements and execution conditions derived from the initial job |
source/ |
Source at the starting commit |
environment.json |
Environment observations and deviations |
setup.sh |
Dependency setup procedure |
The original trace, patch, final answer, and evaluator stay on the evaluation side. Checks inspect file inventories, hashes, provenance, escaping symlinks, and Git history.
The pipeline uses Python's standard library and separates extraction, source preparation, environment preparation, input inspection, controls, execution, scoring, and reporting. Each stage leaves a record, and the next stage requires valid preceding records.
The evaluator was specific to this task
Preparing the inputs was only part of constructing an evaluation case. I still needed a way to decide whether the new implementation satisfied the task.
The starting commit already passed all five existing repository checks. Passing those checks again would not demonstrate that the new feature worked.
I therefore implemented behavioral checks derived from the requirements. The evaluator runs the modified yuurei CLI and checks:
| Check | What it establishes |
|---|---|
| File input | Existing file-based task execution still works |
| String input | A supported inline form, such as stdin, executes |
| Body integrity | The supplied body reaches the runtime unchanged |
| Digest correctness | Both input forms produce the body's SHA-256 |
| Digest equality | Identical bodies have identical digests across input forms |
| Different content | Changing the body produces the expected different digest |
| Input provenance | Files retain their actual path; inline input has an identifier such as stdin
|
| File creation | No new project files appear outside evaluator-managed locations |
A dummy runtime records the body it receives. This checks that the content reached the runtime, beyond whether the command exited successfully. These behavioral checks require no model inference.
The evaluator also runs the existing tests, build, lint, format, and knip checks. A result is satisfied only if all required behavioral and repository checks pass. Missing verification does not count as success.
I tested the evaluator with two controls:
- The unchanged starting source must fail.
- The source with the original implementation patch applied must pass.
The original patch is used only by the evaluator for that control. It is excluded from the new agent's inputs.
These controls establish that the evaluator can distinguish those two states. They do not prove that it rejects every possible incorrect implementation.
The task inputs came from saved records, but the task-specific evaluator was implemented as part of this proof of concept. It was not generated automatically.
One fresh attempt
I gave the prepared inputs to OpenCode with opencode-go/deepseek-v4.1-flash and ran one new session.
| Item | Result |
|---|---|
| Termination | completed |
| Elapsed time | 521.5 seconds, about 8 minutes 42 seconds |
| Generated patch | 8 files, 13,084 bytes |
| Behavioral checks | All passed |
| Five existing repository checks | All passed |
| Verdict | satisfied |
See the run report.
The result means that a fresh implementation, starting from the same source and requirements in an approximate environment, passed the checks I prepared.
The original execution used the same model through an OpenCode-based setup. This was a fresh attempt, not a comparison between models. The original run went through coding-worker's adapter; the new run invoked the harness directly.
Later safeguards do not change the original result
Review after the run found input-separation problems.
Git's FETCH_HEAD and reflog still contained the operator's local path and Git identity. Environment-variable inheritance also used a blocklist, so most operator environment-variable names were passed through.
I subsequently added Git metadata cleanup and inspection, and changed environment-variable inheritance to an allowlist. I did not rerun the original experiment under those revised conditions.
The inspected tool trace showed no reads of that metadata or the original host-side source. But an absence of recorded reads does not establish that access was technically impossible.
Network and access outside the working directory were not fully isolated at the OS level. The original solution was also public, so I cannot establish that it was absent from the model's training data.
The successful outcome belongs to that one run, under those historical conditions. Because the original evidence and generated patch are private, public materials alone cannot support a complete independent recheck of the run.
What I want to preserve from future runs
The most useful lesson was about the records to retain.
The initial job supplied the requirements. Git supplied the starting source. The event log supported prompt consistency checks and environment observations.
A long execution log and a recoverable starting point are different things.
Saving initial requirements and a starting SHA is a practice I can carry into other tasks. The task-specific evaluator remains additional work when adding cases.
Without changing coding-worker, I connected existing records to case construction, fresh implementation, and scoring. This experiment did not cover requirements added during a conversation or reconstruction from incomplete traces.
Next, I want to freeze the inputs and scoring conditions, run a new experiment with the current safeguards, and try another case with a reconstruction difficulty absent from this one.
If I can preserve past work as tasks that remain runnable after model and harness changes, I can gradually build an evaluation set closer to my own development work.
References
- Trace2Eval paper — reconstruction procedure in §3.2 and Appendix A.2
- PoC plan — selected case and simplifications
- Evaluation design — input boundaries, behavioral checks, and controls
- Assessment of the first run — assessment at a specific revision of PR #5, excluding subsequent fixes
Top comments (0)