AI UI Testing: Why Local Demos Hit 9/10 and Production Hits 6/10
We built TestStar, an AI UI testing platform. Here's what we learned from 8 runs of one real case.
The gap nobody talks about
If you've used any AI browser-testing tool in 2025-2026, you've probably hit this:
Local demos succeed 9 out of 10 times. Production at scale? Maybe 6 out of 10. Sometimes worse.
We hit this head-on during our Tier 1 stability verification at TestStar. We took one real-world business case (login → SQL console → enter query → execute → assert, 19 steps) and ran it 8 times:
| Run | Result | Time | Tokens |
|---|---|---|---|
| 1 | failed | 367s | 480K |
| 2 | failed | 104s | 74K |
| 3 | passed | 178s | 191K |
| 5 | passed | 99s | 57K |
| 7 | passed | 114s | 54K |
| 8 | passed | 65s | — |
Each failure had a different cause. Run 1: bad data (AI did nothing wrong). Run 2: browser driver crashed. That's the part that breaks your sanity — there's no consistent failure mode.
But Run 6 was the one that taught us the real lesson. The subprocess finished successfully. The AI did everything right. But the result never made it back to the database. AI judgment and system judgment disagreed.
We spent two days debugging. Root cause: every AI step was triggering a sync memory write. By step 19, the worker thread was so backed up it timed out. Tools worked. Platform collapsed.
That's the production gap nobody warns you about.
What "execution engine ≠ platform" actually means
Browser-Use, Midscene, Skyvern — they're execution engines. They translate natural language into browser actions. Useful, but they're not testing platforms.
A testing platform needs at least:
- Failure handling: retry, give up, escalate?
- Self-healing: is this AI error or product change? Fixable?
- Observability: can humans replay, locate, attribute?
- Retry with idempotency: how do you retry network blips without double-clicking "Pay"?
- Memory: does today's failure experience carry forward?
- Scheduling: how do you run 1000 cases across priorities?
- Knowledge: where exactly is that button on this page?
None of these live in the execution engine. They all live in what we call the Harness layer.
A team we know spent 3 months evaluating 5 AI testing tools, picked the one with the best demo, then spent another 6 months building the Harness layer themselves. Total: 9 months. If they'd asked the right questions upfront, the decision would have been different.
Three real incidents that shaped our self-healing
Incident 1: AI confidently clicked the wrong button
Early on, AI clicked what it thought was "Delete user." Logs showed success. UI showed success. But the actual button — due to a CSS bug — was covering the "Archive" button. AI did exactly what it saw, but the user got the wrong operation.
We learned: self-healing can't trust "operation succeeded" alone. You have to verify the user's intended outcome, not just the system's reported outcome.
Incident 2: Self-healing broke a working test
We shipped the self-healing module. QA reported that previously-passing tests were now failing. We dug in.
The AI saw this line in logs:
"Continue on error: skip summary-xxx.json"
The word "error" triggered the diagnosis. AI generated a patch, applied it, broke a working test. The real failure was on a different line entirely.
Fix: add a "real error extraction" layer. Only scan lines starting with patterns like waitFor timeout / Assertion failed. Exclude configuration lines.
Lesson: the premise of self-healing is trustworthy failure signals. Get the signal wrong, and you make things worse.
Incident 3: "I can't find the login button" — but it's right there
AI reported it couldn't locate the login button. Developer went to the page — the button was right there. Took 30 minutes of debugging to figure out: at the moment AI took its screenshot, a popup ad was covering the button. AI wasn't lying. The signal was misleading.
The 60% number (and why we won't inflate it)
We deliberately created 5 failing test cases and measured self-healing recovery:
| Case | Failure type | Recovered? | Why |
|---|---|---|---|
| 1 | network | ❌ | DNS unreachable — physical failure |
| 2 | element | ✅ | Element relocalization |
| 3 | assertion | ✅ | Assertion condition updated |
| 4 | timeout | ❌ | 1s extreme timeout — test design error |
| 5 | rename | ✅ | Element renamed |
- Recoverable failures: 3/3 = 100%
- Overall: 3/5 = 60%
Case 4 (the 1-second timeout) was the most interesting. We debated whether to "rescue" it — let AI extend the timeout automatically. We decided not to. Why? The case was waiting 1 second for an element that genuinely needs 3 seconds to appear. If AI "fixed" this, it would hide a real test design problem the developer needs to address.
If we inflated the number to 80%, we'd be hiding the fact that physical failures and test design errors aren't supposed to be healed. Honest > inflated.
This principle matters more in AI testing than almost anywhere else. AI testing is inherently uncertain. Customer trust is fragile. Inflated data builds false trust — and false trust collapses harder than it was built.
Memory and the knowledge base
We have three kinds of memory:
- Wiki: "This button is at the top of the page" — persistent
- Skill distillation: "For similar scenarios, this pattern works" — from successful runs
- Failure memory: "Last time we did X, it failed" — accumulated
Storage is semantic-search-first, with local file fallback. Works in demo (no dependencies), scales to production.
But we learned the hard way: memory is not always good. One case: a transient network timeout got written into failure memory. From then on, every similar scenario triggered that "lesson," and AI started second-guessing normal operations.
Memory has noise. Auto-forgetting and conflict resolution are still open problems for us.
For the knowledge base, we inject page-element hints (semantic + location) into the AI prompt. Real effect: AI used to confuse "password" with "password login" — two similar elements. After injecting location hints, it locks onto the right element by position first. Error rate dropped noticeably.
When TestStar works (and when it doesn't)
Works for: form + list + detail UIs in admin/dashboard systems (OA, CRM, BI platforms, internal tools).
Doesn't work for:
- Canvas/WebGL (games, visualization, whiteboards) — AI sees pixels but doesn't know what they mean
- Heavy anti-bot pages (CAPTCHA, slider, behavior detection) — AI has no real human environment
- Very short cases (fewer than 5 steps) — cache hits don't offset startup overhead
- Cross-domain federated login (OAuth, SAML multi-step)
- Heavily dynamic SPAs that completely re-render on every refresh
We're honest about this. Acknowledging your limits is more respectable than pretending you don't have any.
Five questions before adopting any AI testing tool
- Are failure signals structured? (determines whether self-healing is even possible)
- Are logs externally subscribable? (determines observability)
- Can the execution be interrupted? (determines self-healing loops)
- Can the tool ingest external knowledge? (determines knowledge base value)
- Is there an API for scheduling? (determines scale)
These five answers matter more than how good the demo looks.
What we got wrong, honestly
First CI run: 36% pass rate. Stakeholder reaction: "How is this production-ready?" Our answer: "First-run 36% is normal. What matters is exit code 0 (CI didn't block), KPIs collected (healing rate, tokens, time), and data persisted. The real value is in the 2 weeks of production data that follow."
We did not inflate. We did not exclude "known failures." Some team members thought this was too aggressive. We held the line.
Real CI data: missing. Smoke scripts work. Two weeks of production data haven't accumulated yet. Real failure rate, monthly cost, human intervention frequency — all unknowns.
Multi-scenario validation: incomplete. We have deep validation on one case (data query console). Other scenarios (CRM, complex SPAs, low-code platforms) — unverified.
Closing
Three things, if I had to compress:
- The execution engine is the start, not the finish. The Harness layer is where the real battle is.
- Token cost is structural, not optimizable. Cache is not an optimization — it's a prerequisite for production.
- Engines will commoditize. Differentiation will land on memory and knowledge — whoever remembers more, whoever crystallizes better.
If you're evaluating AI testing tools, ask the five questions. If you're building one, build the Harness layer first — the engine can be substituted.
Test engineers' core value was never "knows which tool to use." It's "knows when to trust, when to doubt."
TestStar is an AI UI testing platform: 22K lines of Python + 10K lines of vanilla JS, with a built-in vision-driven AI browser engine and an orchestration/self-healing/memory layer on top.
Top comments (0)