An AI agent tested Toolshop from just the URL: 23 issues, 8 real

Toolshop's sign-up page says a password needs at least 8 characters. The form enforces 8. The API enforces 8. Type 7 and the error message says the minimum is 6.

That's the best of 8 real bugs our AI testing agent found on Toolshop, the demo hardware store a lot of testers learn Playwright and Cypress on. It got only the URL. No test plan, no credentials, no hints.

This is Weekly Teardown 2. Each week we point TestOptim, the agent we build, at a public app and publish what it finds, including what it got wrong. Teardown 1 was ParaBank, whose admin console had no login.

The run

In 30 minutes the agent explored 17 pages and mapped 16 user flows and 29 business rules. It wrote 289 tests, ran 71 of them before the time limit, and flagged 23 issues.

A list of 23 issues isn't worth much if you have to check all 23 yourself. So I did, on the live site.

What held up

1. The password rule changed, and its error message didn't. The sign-up page says a password must "Be at least 8 characters long." The form validator enforces 8:

// sprint5/UI/src/app/auth/register/register.component.ts
PasswordValidators.minLength(8)

So does the API:

// sprint5/API/app/Http/Requests/Customer/StoreCustomer.php
Password::min(8)

But the error text lives in a separate translation file, assets/i18n/en.json, under fields.password.errors.minlength, and it still says "Password must be minimal 6 characters long."

The rule says 8 in three places and 6 in one. Each piece looks fine on its own, which is why this kind of drift is easy to miss in a manual review and easy to catch for an agent that reads the page, submits the form and compares the two.

2. The wrong error for a missing symbol. Leave the special character out and the error says "Password can not include invalid characters." Nothing is invalid; something required is missing. A user who reads that deletes characters, which is the opposite of the fix.

3. A menu link to a category that doesn't exist. Categories → Special Tools opens "There are no products found." The public API has no Special Tools category at all.

4. An empty cart breaks checkout. Open /checkout with nothing in the cart and nothing renders under the step bar. The console says why:

TypeError: Cannot read properties of null (reading 'cart_items')

5. A product that doesn't exist gives a blank page. Mistype a product URL and you get an empty page with a "Related products" heading instead of a "not found" message.

6. "Forgot password" says nothing. Enter an email, click "Set New Password", and the page stays as it was: no confirmation, no error.

7. "Show differences only" only half works. It filters the specifications, but rows that are identical for every product, like Brand, Category and Availability, stay on screen.

8. "2years". The product page prints the warranty as "2years". The comparison page prints the same value as "2 years".

What I threw out, and why

9 findings didn't survive the re-check. They're the more useful half of this post if you're evaluating any AI testing tool.

• Fake payments "accepted". The agent rated a 1111-1111-1111-1111 card and a made-up gift card as critical. In a real store they would be. Toolshop has no payment processor, so on a demo it's expected. Same for an address lookup that invents a street for a made-up postcode.
• The eco discount "never applied". This one was our mistake. The agent guessed the rule. Toolshop's code applies the 5% eco discount only when more than half the cart is rated A or B for CO₂, and the agent's test cart didn't qualify.
• Design choices, not bugs: a contact form that disappears after sending, an order confirmation that shows only the invoice number, category pages without the home page's price filter, and a rentals card the agent expected to show a price.
• No longer reproduces: the agent said an empty sign-up form showed 10 of its 12 errors. Today all 12 show.

6 more need an account to check (a stale cart total, an HTTP 500 on registration, the payment step), so I've left them out rather than guess.

What I took from it

The agent was strongest where people are weakest: a rule written down in several places that drifts in one of them, and pages that break in states nobody visits on purpose, like an empty cart or a mistyped URL.

It was weakest where it had to guess a business rule the site never states, or tell a demo's shortcuts from real bugs. Closing that gap is what we're working on next.

If you want to try it on your own app, give TestOptim your staging URL. It explores the pages, maps the flows and business rules, writes and runs the tests, and sends each bug with screenshots, repro steps and a severity to Slack, Jira or GitHub.

Got a public app for next week's teardown? Drop the URL in the comments.