I Built a Hands-Free AI Harness for Migrating Legacy Tests

Migrating a large legacy test suite is not a code-generation problem. It is a confidence problem.

An agent can translate an old test into modern syntax very quickly. The difficult question is whether it preserved the behavior-and how you can prove that across hundreds or thousands of tests without manually supervising every generated change.

I built an agent harness to solve that problem. You point it at a specification, and it works through every test hands-free: it plans the work, writes the migration, explores the live application when needed, runs verification, records evidence, and moves on only when the evidence is complete.

The model does the work inside the loop. The harness owns everything between the steps: scope, retries, verification, evidence, and the final decision.

A model may propose a migration. It never gets to declare that migration complete.

This was built for a real migration; the repo is a safe public version

The original harness was built around a Ruby Watir-to-Playwright migration in a private codebase. For the public reference repository, I recreated the pattern using a bundled Protractor-to-Playwright example. The framework names differ; the design problem does not.

In both cases, the legacy test is a useful behavioral specification. It tells us what a user does and what the application must prove afterwards.

# The original shape: a legacy browser test describes behavior.
browser.goto("/login")
browser.text_field(id: "username").set("demo")
browser.button(id: "login-submit").click
expect(browser.url).to include("/tasks")

The goal is not a prettier translation. The goal is a Playwright test that demonstrably preserves that behavior.

Ground truth changes the agent problem

For many AI tasks, there is no single right answer. You need an evaluation framework to judge whether an answer is good enough.

Test migration is different. We already have ground truth:

• the old test defines the behavior to preserve;
• the running application can confirm whether the migrated test works;
• the assertions can be checked for coverage;
• the test runner produces an authoritative pass/fail result.

So I did not build the system around LLM “evals” that ask another model whether the output looks good. I built a verify step that runs the migrated test with Playwright, ensures the required assertions pass, and records the result as evidence for the change request.

That distinction matters. “The test passes” in an agent’s response is text. A Playwright result, the assertion mapping, and a hash of the real report are evidence.

It is not a chat workflow-it is a hands-free coding workflow

I would not use a chat prompt to migrate a large suite today. We now have terminal-based coding agents such as Claude Code and Codex that can inspect repositories, edit files, run commands, and use tools. The opportunity is bigger than asking one agent to convert one snippet.

The harness gives those agents an operating model:

1. Read the migration specification and enumerate the legacy tests.
2. Select one independently verifiable test.
3. Assemble the instructions and task context for that phase.
4. Let the agent perform the focused work.
5. Verify the result mechanically.
6. Capture evidence, commit the completed unit, and continue to the next test.

This is what makes it hands-free at scale. A human points to the suite and defines the destination; the harness drives the migration test by test, while retaining a reviewable record of every decision.

The important unit is one test, one verdict. My early design delegated whole files. That made failures ambiguous, retries wasteful, and progress fragile. When each test gets its own cursor, evidence, and commit, a stuck test does not block the rest of the file and a restart does not lose already-green work.

The harness prompts itself from the task

One of the ideas that stuck with me came from an Anthropic engineer: do not make the human continually prompt the agent. Define the task and build a system capable of prompting itself as the work unfolds.

That is the model I used here. The harness does not depend on someone sitting beside it and sending follow-up messages such as “now verify that,” “now fix the selector,” or “now commit it.” It constructs fresh, bounded context for the current phase from task files and the current unit.

The original harness used phase-specific Markdown job cards-think plan.md, implement.md, verify.md, review.md, and close.md. Each card gave the agent one responsibility, the allowed tools, the expected output, and the rules relevant to that phase.

For example:

Phase	Agent or machine responsibility
Plan / preflight	Understand the unit and confirm prerequisites.
Implement	Create one migration, not a speculative refactor.
Explore	Use the browser to confirm the live UI and locators.
Verify	Run the migrated test and capture Playwright’s verdict.
Review / close	Check evidence, conventions, and commit the unit.

Fresh contexts are deliberate. The verifier should inspect the implementer’s files and evidence, not inherit its reasoning and assumptions. That keeps verification independent.

I also avoided trying to build a massive, permanent skill.md that explains every future migration. Those documents become stale, and they can constrain an agent that is otherwise capable of reasoning about the current codebase. Instead, the durable guidance is a short gotchas.md: only the proven workarounds for places agents repeatedly get stuck or waste time. Retrospectives can propose additions; a human decides what earns a place there.

Browser exploration made the tests stronger

Translation alone is not enough. The old test may contain stale locators, timing hacks, or assumptions that no longer reflect the current UI.

In the original harness, the agent could use Playwright’s MCP browser capability during the implementation flow to inspect the running application. That made it possible to confirm the page structure and locator strategy before writing the assertion, instead of faithfully porting a selector simply because it once existed in Watir.

That is a meaningful upgrade over syntax conversion:

Legacy test says: click “Submit”
Agent explores live app: button has stable data-testid="save-profile"
Migrated test uses: page.getByTestId('save-profile').click()

The result is a migration that is not only equivalent, but usually more resilient. It also exposed a lesson that applies to every agent tool: a capability you cannot prove is being used is not a capability. If browser access, authentication, or a tool flag is silently misconfigured, a passing-looking result can be a false positive. Verify the mechanism, not just a lucky artifact.

Note: the public Protractor reference repo keeps this live exploration seam documented but intentionally simplified; the production-inspired design is where the browser exploration loop was exercised.

Evidence, not a claim

The close phase has a strict contract. A migration is not complete because an agent says it is. It is complete only when the harness can collect and attach evidence such as:

• a successful Playwright run for the migrated unit;
• a machine-readable test report and its hash;
• proof that each source assertion is covered in the target test, or an explicit reason it cannot be ported;
• the stable source-to-target migration tag;
• the commit and a concise evidence summary for the PR/MR.

That last point is crucial for large-scale migrations. Reviewers should not have to reconstruct whether an AI-generated test was actually run. The change itself carries the proof: what was migrated, which assertions were preserved, and the report that passed.

The public repo includes a break-it demonstration that deletes a completed unit’s evidence report. When the close phase runs again, it refuses to mark the unit done. If there is no report, there is no hash; if there is no hash, there is no success claim.

Retries should converge, not spin

A fixed retry count cannot distinguish progress from repetition. The harness stores a normalized failure signature for each attempt.

• A new signature usually means a change fixed one problem and exposed the next one: continue.
• The same signature means the agent is likely thrashing: stop, record the reason, and do not silently burn the budget.

This is especially useful in migration work because failures are not always agent mistakes. A legacy test may depend on a shared login session or an invisible setup step. The migration can faithfully expose that hidden precondition. That deserves a visible blocked or dropped state-not an endless retry loop and not a fake green result.

The transferable pattern

The public repository demonstrates Protractor-to-Playwright. The original work was Watir-to-Playwright. The framework pair is not the point.

This approach works whenever an existing artifact gives you a behavior to preserve and a real system can verify the new implementation: test framework migrations, language ports, framework upgrades, and deprecated-library replacements.

The durable ideas are:

• Use agents as hands-free workers, not chat assistants waiting for the next prompt.
• Delegate at the smallest unit with an independent verdict.
• Let agents explore the real system when that improves the implementation.
• Prefer executable verification over model-based evaluation when ground truth exists.
• Store the proof in the PR/MR, alongside the change.
• Keep learned guidance short, specific, and human-curated.

Try the reference implementation

The repository is a small, runnable teaching implementation. It has a bundled demo app, legacy Protractor tests, a Playwright target, phase job cards, mechanical checks, finished case-file examples, and a dry run that needs no model key.

npm install
npm run dry-run

Then run npm run break-it to see why evidence is part of the definition of done.

Repository: [https://github.com/harikrishna8121999/agentic-migration-harness]

The useful question is not whether an agent can write a test. It clearly can. The useful question is whether your workflow can let it migrate an entire suite while preserving behavior-and prove it did.

Comments

[Raknaos]

The distinction between "the test passes" as text and a Playwright run plus a report hash as evidence is the part most agent harnesses skip, and the break-it demo where deleting the report makes the close phase refuse to finish is a much stronger guarantee than any prompt instruction to be thorough.

Two things I'd be curious about from running it at scale. The normalized failure signature for retries is what I'd want most, but how do you normalize it - stack trace shape, assertion message, or the locator that failed? Because a legacy suite with shared login state will produce the same signature twice for two different reasons, and stopping there looks correct until you realise it was flake. And on gotchas.md as the only durable guidance: how large did it get before it started contradicting itself, and did you ever have to delete an entry because the model outgrew the workaround?

[Harikrishna V Shetty]

Great questions.

For retries, the harness retries a test a limited number of times. It normally fixes the missing piece itself. But if it hits a real external blocker- for example, a required login credential is missing from the vault, it cannot resolve that safely. After repeated failure at the same step, it skips the test and reports it clearly in the final summary.

We then inspect the exact failure, fix the missing capability in the new test framework, and resume the migration.

gotchas.md can definitely bloat. I review it after each spec migration and remove entries once the underlying issue is addressed in the new framework. It stays useful because it contains active workarounds, not every problem we have ever seen.