I've spent more than 23 years in software engineering, quality engineering, automation, and delivery.
I've watched testing move from manual cases to automation, CI/CD, cloud, and now AI-powered testing.
Something has changed fundamentally.
For years the model was simple:
Human thinks
Human defines the test
Machine executes
Human evaluates the result
Now we're moving toward:
AI thinks
AI creates the test
AI executes
AI evaluates
AI reports: PASS
And that creates a problem we don't talk about enough:
If the AI performs the test and then tells us it passed, who verifies the AI?
That question is one of the reasons I started building TestNeo.
AI can already write tests
Ask an assistant: "Create a Playwright test for the login flow."
In seconds you get something reasonable — cases, code, API checks, selectors, retries, even error analysis.
The productivity gain is real.
But generating a test is not the same as proving the application works.
Then we started letting AI execute
Instead of code, we say:
"Login as an administrator, create a customer, open the dashboard, and verify the customer appears."
An agent can click through that path. It looks great — until you ask:
Did the customer actually get created?
The agent says yes. That's still a claim.
The problem with letting the agent grade itself
UI shows success → backend failed → agent still reports PASS
Checkout “completed” → payment failed → page transition counted as success
“Test passed” → agent reached a page but never verified the expected state
An agent can take the right-looking path and reach the wrong conclusion — or the right answer via the wrong path.
As AI moves from generating things to doing things, claimed success without independent proof becomes a release risk.
Claimed → Observed → Verified → Evidence → Learn
While building TestNeo we separated:
The important word is Verified. A PASS without evidence is just another statement.
Verification needs context — not only a green bar
A trustworthy result isn't only screenshots and assertions. It needs situation:
What code / PR changed?
What does the product context say (flows, APIs, domain map — the knowledge we keep about the system)?
Have we seen this failure before (bugs, RCAs, past runs)?
Which checks matter for this risk — functional, API, visual, performance, agent behavior?
What should we remember so we don't relearn the same incident?
That's why TestNeo is built around more than “run NLP in a browser”:
The goal isn't more results. It's results you can defend when someone asks: should we ship?
Volume isn't confidence
AI generates 1,000 tests. Impressive — until many are redundant, weak, flaky, or obsolete.
Better questions:
Given what changed, what should we verify?
What evidence do we have?
What risk remains?
That shifts the work from test generation to risk-based verification.
Agents raise the bar again
With classic automation, intent is mostly fixed.
With an autonomous agent, the agent may choose path, tools, retries, and whether it believes it succeeded.
So we need to evaluate not only the application — but the behavior of the AI operating on it.
We brought natural-language intent into Chrome for that reason. The interesting part isn't “AI can control a browser.” We already crossed that line.
The interesting question is: how do we know the agent achieved what we asked?
From a single run to a release decision
A modern release mixes code changes, web, APIs, agents, visuals, performance, security signals, and years of run history.
A giant PASS/FAIL list is not a decision.
What changed?
→ What matters? (context + risk)
→ What did we verify?
→ What does memory say we've seen before?
→ What evidence do we have?
→ What risk remains?
→ Should we ship? (PASS / WARN / BLOCK)
That's the direction of TestNeo Release Intelligence:
intent → execution → observation → verification → evidence → memory → PR/release gate.
We haven't “solved” this
We're still experimenting — agents surprise us; some checks need a human in the loop.
Hard questions remain:
How much independent evidence before we trust an agent?
When AI thinks, acts, and grades itself, where must the human stay?
I don't have a final answer. After two decades in this field, I believe this much:
AI making testing faster is inevitable.
Making results trustworthy enough to act on is the harder problem.
That's what we're trying to build with TestNeo.
One line I keep
AI can perform the work. Performance isn't proof.
Claimed → Observed → Verified → Evidence → Learn.
If you're putting AI agents into engineering workflows, I'd like to hear from you:
Where do you draw the line between an agent's claim and evidence you actually trust?
About TestNeo
We're building toward evidence-based verification and release confidence — NLP execution, product context, engineering memory, risk-aware regression, and PR-level PASS/WARN/BLOCK — not generation alone. Still evolving; learning in public.
Top comments (0)