Comparing AI coding assistants without a shared harness produces noise, not data. If one tool gets a cleaner branch, better prompts, or a longer session than another, the comparison is measuring the setup rather than the tools. This is a practical, low-overhead way to build a harness that stays fair across every assistant you test, using nothing more exotic than version control and a checklist.
Start from a shared branch, not a live checkout
Pick a fixed commit that predates the real-world fix for each ticket in your test set, and cut a dedicated branch from it using Git. Every assistant works from that exact branch, with no changes made in between runs. This sounds obvious, but it is the single most common source of unfair comparisons: testing one tool on Monday's code and another on Thursday's code after three unrelated merges introduces variables that have nothing to do with the assistant itself.
git checkout -b eval/ticket-482-timeout-fix a1b2c3d
Tag or branch per ticket so you can reset cleanly between assistant runs without dragging leftover changes from a previous attempt into the next one.
Standardize the prompt template
Write one prompt template per task type, bug fix, feature addition, or test generation, and reuse it verbatim across every assistant. Include the ticket description, any relevant file paths, and nothing else that varies between runs. If you find yourself tailoring the prompt to play to one tool's strengths, that tool's score stops meaning anything relative to the others.
Fix the time budget in advance
Decide how long each assistant gets per task before you start, and hold every candidate to the same limit. A generous, unequal time budget measures patience, not capability. Write the budget into your harness documentation so anyone repeating the comparison later, including a future version of your own team, applies the same rule.
Capture the full session, not just the diff
Log the entire interaction, including intermediate steps, clarifying questions, and any self-corrections. A continuous integration-style log of the whole run, saved as a plain text transcript alongside the final diff, lets you go back later and understand why one assistant's result looked the way it did, which a diff alone cannot tell you.
Photo by K on Pexels
Score against your real test suite
Wherever your codebase already has automated tests, run each assistant's change against them directly rather than trusting a self-reported summary of what changed. Where coverage is thin, apply the same code review bar a human reviewer would use on a colleague's pull request, checking not just whether the change compiles but whether it preserves behavior the ticket did not ask you to change.
Run each task more than once
If the assistant's output has any randomness, a single run is a snapshot, not a measurement. Running the same task twice per assistant and comparing both results keeps one unusually strong or unusually weak session from dominating your conclusion about the tool overall.
Assign scoring to someone without a stake in the outcome
A harness only stays fair if the person scoring the results isn't also the person who picked their favorite tool going in. Whoever set up the comparison, wrote the prompts, or has a preference for a specific assistant should not also be the sole scorer. Split the work: one person runs the tasks and captures transcripts, a different person reviews the diffs against the answer key and the real test suite. This single change removes most of the unconscious bias that creeps into informal tool comparisons.
If your team is small and splitting roles isn't practical, at minimum have a second engineer spot-check a sample of the scored results before you treat the comparison as final. A quick sanity pass catches the cases where enthusiasm for a particular tool quietly softened the scoring criteria partway through.
Document the harness so it's reusable
Write down the branch strategy, the prompt templates, the time budget, and the scoring rubric in a short document that lives alongside your test ticket set. The goal is that six months from now, when a new assistant needs evaluating or you want to check whether your current tool is still performing the way it did at adoption, nobody has to reconstruct the process from memory. A harness that only exists in one engineer's head disappears the moment that engineer moves to a different project.
This documentation also makes the comparison defensible if someone later asks why the team chose one tool over another. A written record of the tasks, the scores, and the reasoning is a much stronger answer than "it felt faster in the demo."
Expect the first run to reveal gaps in your own process
The first time you run this harness end to end, you'll likely find gaps: a prompt template that was ambiguous, a task that turned out to be harder to grade objectively than expected, a time budget that was too tight for one task type and too generous for another. That's normal and expected. Treat the first pass as a dry run of the process itself, refine the rubric and templates based on what you learn, and then run the real comparison with the improved version. The investment compounds, because the same refined harness gets reused for every future tool evaluation.
Keep the harness boring on purpose
It is tempting to build a custom evaluation framework with automated scoring, dashboards, and a database of results. Resist that until you have run the comparison manually a few times and know what you actually need to track. A shared branch, a fixed prompt template, a stopwatch, and a shared spreadsheet is enough to run a fair comparison across three or four assistants, and it is a setup your whole team can maintain without dedicating an engineer to it full time.
A longer walkthrough of scoring correctness and tracking review burden is covered in 137Foundry's guide to benchmarking AI coding assistants, which goes through the full process from picking test tickets to catching regressions when the underlying model changes. This engineering team's approach to evaluating tools before adopting them follows the same harness described above.
The setup above takes an afternoon to build once and can be reused for every future comparison, whether that is a new assistant on the market or a check on whether your current tool still performs the way it did when you adopted it.
A minimal example you can copy
If you want a starting point rather than building the whole thing from a blank page, here's a minimal version that covers the essentials without much overhead:
harness/
README.md # branch strategy, time budget, scoring rubric
tickets/
ticket-001.md # ticket description + answer key
ticket-002.md
...
results/
assistant-a/
ticket-001-transcript.txt
ticket-001-diff.patch
assistant-b/
...
A README describing the rules, a folder of frozen ticket descriptions with answer keys, and a results folder per assistant is enough structure to keep a comparison honest without building anything resembling a full evaluation platform. You can version control the whole thing and reuse it the next time a new assistant needs testing.
Common pitfalls that quietly break the comparison
A few mistakes show up often enough to call out directly. Testing different assistants on different days, after unrelated commits have landed on the main branch, introduces variance that has nothing to do with the tools. Letting the person who prefers a specific assistant also write that assistant's prompts, even with good intentions, tends to produce prompts that play to that tool's strengths. And treating a single run as final, when the assistant's output has any randomness, risks drawing a conclusion from what was actually an unusually good or unusually bad session.
None of these mistakes are hard to avoid once you know to watch for them, which is exactly why writing the harness rules down in advance, before anyone starts testing, matters more than any particular tooling choice.
Top comments (0)