Stop Healing Your Tests: Why Throwaway Automation Fits the AI Era

Luthfi Ferdian — Wed, 30 Sep 2026 02:11:48 +0000

Self-healing tests are solving the wrong problem.

When a "healer" swaps a broken locator for a "close enough" element, the run stays green, but the test might now be asserting something nobody intended. You haven't fixed the test. You've traded a loud failure for a quiet false positive.

Here's the alternative I keep coming back to: don't maintain a script at all. If a model can read a test case and execute it in a browser by itself, the script stops being the thing you need. The test case is. You can't break a script that doesn't exist.

Self-healing fixes the symptom

A self-healing locator answers "the button moved, how do I still click it?" It never asks "should this script still exist?"

Worse, a healer can swap a broken selector for a nearby element that looks plausible. The test passes, but it now checks something the author never meant. You've traded a loud failure for a quiet false positive, and those are the most expensive kind of test debt. A locator that keeps breaking is also telling you something (unstable hooks, a churning screen), and auto-repair mutes that message.

The throwaway loop: test cases in, execution out

Here's the alternative I keep coming back to. Your source of truth is the test case in plain language, the same one you'd hand to a manual tester. Instead of turning it into a Playwright script that someone has to own, you let an AI agent read it and run it.

With a browser-control tool such as the Playwright MCP server, the agent can open a headless browser, follow the steps, look at the page, and report what happened. Because it works from the page as it is right now, there is nothing to heal. If the button moved, it finds the button, the way a human would.

A test case in this world is just this:

## TC-CART-014: Promo banner shows for a new promo SKU
Precondition: logged in as a standard test user, cart is empty
1. Add the item with SKU PROMO-001 to the cart
2. Open the cart page
Expected: a promo banner is visible above the item list and mentions the discount
Also check: no error toast, page has no layout overlap on mobile width

And the instruction to the agent is equally short:

Run TC-CART-014 against the staging URL in a headless browser.
For each step, note what you did and what you saw.
Take a screenshot at the end. Report PASS/FAIL per expected result, with evidence.

The output is a short report with screenshots. You verify visually. A human skims the evidence in seconds, which is often faster than debugging a red CI job. No script exists, so no script can rot.

Why this beats healing for the long tail

Most suites have a long tail of checks that matter for a moment: a release, a migration, a risky refactor, a single bug report you want to reproduce. Writing and maintaining scripts for those is a bad trade.

No maintenance. The "automation" is regenerated on every run from the test case. UI redesigns don't break anything.
Runs in the background. Kick it off in headless mode and come back to a report. It doesn't need your attention while it runs.
Visual verification. Screenshots catch things scripted assertions never asked about: overlapping elements, a broken layout, a wrong image.
Test cases stay the durable asset. Everyone can read and review them, including product and manual testers.

Promote only what earns it

Here's the part I like most: this isn't a one-way door. Those background runs double as a discovery pipeline for your maintained suite.

When a scenario keeps showing up, catches real bugs, or guards something critical, that's a signal it deserves a proper, deterministic script. At that point you promote it: take what the agent did (the steps, the selectors it found, the state it needed) and turn it into a reviewed test with real assertions.

import { test, expect } from '@playwright/test';

test('promo banner shows for a new promo SKU', async ({ page, request }) => {
  const res = await request.post('/api/cart', { data: { sku: 'PROMO-001', qty: 1 } });
  expect(res.ok()).toBeTruthy();
  const { cartId } = await res.json();

  await page.goto(`/cart/${cartId}`);
  await expect(page.getByTestId('promo-banner')).toBeVisible();
});

My rule: if you wouldn't urgently fix a script when it breaks, don't promote it. Everything else stays as a test case the agent runs on demand. Your maintained suite stays small and trusted, because every script in it was chosen, not accumulated.

Where this goes wrong

I don't want to oversell it. Some honest limits:

They aren't a merge gate. Agent runs are non-deterministic, slower than a compiled script, and cost tokens. Two runs may take different paths or judge "visible" differently. Fine for exploration and release checks, not for the thousand-test regression run.
Agents can be too forgiving. An agent that "works around" a broken flow may report a pass where a real user would be stuck. Write expected results that are strict, and read the evidence.
Core flows need code. Audit and compliance trails need deterministic, durable scripts.
Test data and environment knowledge still come from you. AI doesn't know your seeded accounts or your staging quirks unless you tell it.
Some failures are real bugs. Never re-run until it goes green.

The Bottom Line

Self-healing optimizes for keeping every script alive. The throwaway approach optimizes for keeping the right scripts alive: let AI read your test cases and run them headless, verify the evidence visually, and only turn a scenario into maintained code when it has proven its worth.

Your value as an SDET isn't how many scripts you maintain. It's the judgment about which checks deserve to become code at all.

So here's my question: which part of your current suite could be replaced by a well-written test case and an agent, and what would you need to see before you trusted it?

Originally published at luthfiferdian.com.

I Let AI Write My Entire Test Suite — Here's What It Missed

Luthfi Ferdian — Wed, 15 Apr 2026 14:07:00 +0000

Introduction

As an SDET, writing test cases is one of my core responsibilities. What we test and how we test it directly shapes the quality of what we ship. So when AI tools started promising to automate test case generation, I had to try it. I believe writing the test case 100% manually will be outdated and counterproductive, so I gave an AI my PRD and TCD, and asked it to generate the full suite. What came back in seconds would have taken me two to three days. Some of it was genuinely impressive — but some of it would have let real bugs slip into production.

What AI Got Right

The result stuns me. I wouldn't say we can trust it with 100% of the creation. Yes AI can also be wrong, yes AI can put its incorrect assumptions on the table, and yes absolutely it can create a very generic test case that could slow the test case creation rather than speed it up. As far as I know, AI is mostly about context. Less context means your result will be generic as per AI's assumptions. Too much context on the other hand may also reduce AI's accuracy and results as too many things it should consider. Not to mention not all the context is actually needed to perform the action. With proper context that we give to the AI, I would say at least 80% of the test case creation effort could be done by the AI.

What AI Missed

Though it helps a lot, I found that human intervention is still required (at least at this time 😂). Test cases generated by the AI still lack in some areas. First is the integration of the current PRD with the whole system. Many PRDs are essentially about enhancing current capabilities of our product. It means enhancing some part of the large system it is in. AI still faces difficulty in understanding how the current PRD could benefit the whole system and how the whole flow could be affected by it. Second is generating the proper test steps and test data. Since it is difficult for the AI to understand the whole system flow, sometimes it means very generic test steps. Steps that fail to capture the personalized flow of the system. Test data on the other hand is really important to the testing activities. Without it, it is simply impossible to do the testing. Some test cases need specific test data to be set. AI still lacks knowledge on how to prepare the test data.

The Bottom Line

So should you use AI to write your test cases? Absolutely — but treat it as a first draft, not a final product. Here's my rule of thumb: let AI handle the happy-path and standard validation cases (it's great at those), but always review what it generated. AI got me about 80% of what I need. That last 20% — the part that requires system knowledge, domain context, and real test data — is exactly where SDETs prove their value. The testers who learn to work with AI on the 80% and focus their energy on the 20% will be the ones who thrive.

Call to Action

Not leveraging AI in your daily work as an SDET would be a missed opportunity. Even though there's room for improvement, AI already saves significant time on the repetitive parts of test creation. Learning to work with it — trying, iterating, and figuring out where it fits in your workflow — is a process that I find joy in. How's your experience using AI in your daily tasks?

DEV Community: Luthfi Ferdian

Stop Healing Your Tests: Why Throwaway Automation Fits the AI Era

Self-healing fixes the symptom

The throwaway loop: test cases in, execution out

Why this beats healing for the long tail

Promote only what earns it

Where this goes wrong

The Bottom Line

I Let AI Write My Entire Test Suite — Here's What It Missed

Introduction

What AI Got Right

What AI Missed

The Bottom Line

Call to Action