Tools that turn a website into an API for an AI agent often generate Playwright code and repair it when it fails. On a recent Launch HN thread, commenters asked two questions about this: can a tool return success when the action never completed, and does anything notice when a field goes empty while the response still looks fine?
I measured it. I changed an orders page in 10 ways and ran 3 AI-generated tools against the baseline and each changed version, with a repair loop. In 16 of 66 runs (16 of the 60 on a changed page) the tool returned wrong data and reported success. The repair loop never saw those runs, because it only starts on an error. A schema check plus four checks against the page turned all 16 into errors, and the repair loop could then work on them.
The code, every generated tool and every run are in the PoC repo.
The setup
The site is a small orders page: 12 orders, 8 of them open, with a Cancel button on each row. The page loads its data from an API. There is a baseline version and 10 changed versions:
| Change | What the page or API does |
|---|---|
| css-rename | every class and container id renamed |
| column-swap | Customer and Total columns swapped |
| locale-format | totals shown as 1.234,50 € in place of €1,234.50
|
| api-rename | the API field total renamed to amount; the page shows —
|
| late-rows | the API answers after 2.5 s; placeholder rows until then |
| paginated | 5 orders per page, with a Next page button |
| soft-error | HTTP 200, an error banner, no rows |
| cancel-409 | the cancel API refuses with 409, but the page still says "Order cancelled" |
| cancel-confirm | Cancel opens a confirm dialog |
| card-layout | cards in place of the table |
Each tool has two tasks: list the open orders (id, customer, total), and cancel order A-1003.
-
Generation:
claude -p --model sonnetwrites the tool from the rendered HTML of the baseline page. I did this 3 times, so there are 3 different tools. The model gets the task prompt and the HTML: no tools, no project files, no CLAUDE.md. - Repair: when a run fails with an error, the model gets the error, the code and the HTML of the changed page, and writes a new version. Up to 2 repairs per run.
- Scoring: each run is checked against the server's data. Correct, loud (an error), or silent (no error, wrong result). For the error banner and the refused cancel, an error is the correct result.
The results
| Arm | Runs | Correct | Loud | Silent |
|---|---|---|---|---|
| Generated, with repair | 66 | 45 | 5 | 16 |
| + schema check | 66 | 46 | 7 | 13 |
| + checks against the page | 66 | 58 | 8 | 0 |
66 runs per arm: 3 tools, the baseline plus 10 changes, 2 tasks. The repair loop worked when it got an error: in the first arm, all 16 repaired runs ended correct or with an error, never silent. What it never got was the 16 silent runs.
What the silent runs returned
| Change | Silent (of 3) | What the tool returned |
|---|---|---|
| locale-format | 3 |
1.234,50 € read as 1.2345, 420,00 € as 42000 |
| cancel-409 | 3 |
cancelled: true, but the order is still open |
| api-rename | 3 | every total 0 |
| late-rows | 2 | an empty list |
| soft-error | 2 | an empty list |
| card-layout | 2 | an empty list |
| css-rename | 1 | an empty list |
Most come from a line that turns a failure into a value. These two are from the third generated tool:
return Number.isNaN(n) ? 0 : n; // a missing total becomes 0
await page.waitForSelector(ROW, { timeout: 10000 }).catch(() => {}); // a timeout becomes an empty list
The other two tools got the totals of 0 another way: they strip — to an empty string, and Number("") is 0.
All 3 money parsers removed the commas before they parsed the number, so the German format lost its decimal point. All 3 cancels counted as done when the row left the page. On cancel-409 the page removes the row even though the server refused, so each tool saw what it expected.
The common thread is code that prefers a value to an error, and a repair loop that waits for an error never sees it.
What caught them
A schema check on the output (id and customer not empty, total a positive number) caught the 3 runs with totals of 0, and nothing else. An empty list is a valid list. 1.2345 is a positive number.
Four checks against the page, run after the schema, caught the other 13:
- The page shows no error banner (
role="alert"). - The page's "N open orders" count equals the number of orders the tool returned.
- The digits of each total match the digits in that row on the page.
- After a cancel, a reload shows one open order less, and no row for that id.
With the schema and these checks, no run was silent. Of the 16 runs that had been silent:
- 8 were repaired to the correct result (locale-format 3, late-rows 2, card-layout 2, css-rename 1).
- 5 ended with an error, which is the correct result for the error banner and the refused cancel.
- 3 (api-rename) still failed after 2 repairs, with an error.
Across the arm, 27 runs needed a repair: 19 ended correct and 8 ended with an error (api-rename 3, cancel on the error page 3, cancel with late rows 2).
One check was wrong. On api-rename, two repaired tools read the correct totals from the API, but the page shows —, so check 3 failed them. I count those as errors, not as silent failures.
The limits
- I wrote the checks after I saw which runs failed silently in the first two arms. They check general things (an error on the page, a count, the digits, a re-read after an action), but they know this page's count label and row markup. Another site needs its own.
- Three tools from one model, two tasks, one small page. The rates are for this setup, not for any product.
- The changes are mine, and all of them at once is not how a real site changes.
- The generated code is saved, so a re-run gives the same numbers. A new generation can give different code.
The repo also has a hand-written tool with Playwright assertions. It had no silent failures in 22 runs, but I wrote it knowing the changes, so it is not a fair comparison.
What I'd do with it
- If a tool repairs itself on errors, check that it fails when it should. Count the silent results, not the repairs.
- Search the generated code for lines that turn an error into a value. In these tools:
.catch(() => {})on a wait,isNaN(n) ? 0 : n, andNumber()on text that can be empty. - Add at least one check that comes from the page and not from the tool: a count the page shows, or a re-read after an action.
- Treat a schema check as the floor. It catches missing values, not wrong ones.
Repo: qa-gary-parker/poc-ai-tool-silent-failures. The self-healing tests experiment and the mutation testing experiment look at the same problem from the test side. More are in the playbook.
Originally published at https://garyparker.dev/blog/ai-generated-playwright-tools-silent-failures
Top comments (1)
Some comments may only be visible to logged-in visitors. Sign in to view all comments.