DEV Community: Gururaj

Why AI-generated tests are only half the problem — verification and evidence matter more

Gururaj — Sun, 18 Oct 2026 12:32:35 +0000

I've spent more than 23 years in software engineering, quality engineering, automation, and delivery.

I've watched testing move from manual cases to automation, CI/CD, cloud, and now AI-powered testing.

Something has changed fundamentally.

For years the model was simple:

Human thinks
Human defines the test
Machine executes
Human evaluates the result
Now we're moving toward:

AI thinks
AI creates the test
AI executes
AI evaluates
AI reports: PASS
And that creates a problem we don't talk about enough:

If the AI performs the test and then tells us it passed, who verifies the AI?

That question is one of the reasons I started building TestNeo.

AI can already write tests
Ask an assistant: "Create a Playwright test for the login flow."
In seconds you get something reasonable — cases, code, API checks, selectors, retries, even error analysis.

The productivity gain is real.

But generating a test is not the same as proving the application works.

Then we started letting AI execute
Instead of code, we say:

"Login as an administrator, create a customer, open the dashboard, and verify the customer appears."

An agent can click through that path. It looks great — until you ask:

Did the customer actually get created?
The agent says yes. That's still a claim.

The problem with letting the agent grade itself

UI shows success → backend failed → agent still reports PASS
Checkout “completed” → payment failed → page transition counted as success

“Test passed” → agent reached a page but never verified the expected state

An agent can take the right-looking path and reach the wrong conclusion — or the right answer via the wrong path.

As AI moves from generating things to doing things, claimed success without independent proof becomes a release risk.

Claimed → Observed → Verified → Evidence → Learn

While building TestNeo we separated:

The important word is Verified. A PASS without evidence is just another statement.

Verification needs context — not only a green bar
A trustworthy result isn't only screenshots and assertions. It needs situation:
What code / PR changed? What does the product context say (flows, APIs, domain map — the knowledge we keep about the system)? Have we seen this failure before (bugs, RCAs, past runs)? Which checks matter for this risk — functional, API, visual, performance, agent behavior? What should we remember so we don't relearn the same incident?

That's why TestNeo is built around more than “run NLP in a browser”:

The goal isn't more results. It's results you can defend when someone asks: should we ship?

Volume isn't confidence
AI generates 1,000 tests. Impressive — until many are redundant, weak, flaky, or obsolete.

Better questions:

Given what changed, what should we verify?
What evidence do we have?
What risk remains?
That shifts the work from test generation to risk-based verification.

Agents raise the bar again
With classic automation, intent is mostly fixed.
With an autonomous agent, the agent may choose path, tools, retries, and whether it believes it succeeded.

So we need to evaluate not only the application — but the behavior of the AI operating on it.

We brought natural-language intent into Chrome for that reason. The interesting part isn't “AI can control a browser.” We already crossed that line.

The interesting question is: how do we know the agent achieved what we asked?

From a single run to a release decision
A modern release mixes code changes, web, APIs, agents, visuals, performance, security signals, and years of run history.

A giant PASS/FAIL list is not a decision.

What changed?
→ What matters? (context + risk)
→ What did we verify?
→ What does memory say we've seen before?
→ What evidence do we have?
→ What risk remains?
→ Should we ship? (PASS / WARN / BLOCK)
That's the direction of TestNeo Release Intelligence:
intent → execution → observation → verification → evidence → memory → PR/release gate.

We haven't “solved” this
We're still experimenting — agents surprise us; some checks need a human in the loop.

Hard questions remain:

How much independent evidence before we trust an agent?
When AI thinks, acts, and grades itself, where must the human stay?
I don't have a final answer. After two decades in this field, I believe this much:

AI making testing faster is inevitable.
Making results trustworthy enough to act on is the harder problem.

That's what we're trying to build with TestNeo.

One line I keep
AI can perform the work. Performance isn't proof.

Claimed → Observed → Verified → Evidence → Learn.

If you're putting AI agents into engineering workflows, I'd like to hear from you:
Where do you draw the line between an agent's claim and evidence you actually trust?

About TestNeo
We're building toward evidence-based verification and release confidence — NLP execution, product context, engineering memory, risk-aware regression, and PR-level PASS/WARN/BLOCK — not generation alone. Still evolving; learning in public.

Why Testing Still Feels Broken (Even with AI & MCP Tools)

Gururaj — Mon, 30 Mar 2026 14:35:48 +0000

Why Testing Still Feels Broken (Even with AI & MCP Tools)

We have:

Selenium
Playwright
Cypress
AI-powered test generators
MCP / autonomous testing tools

And yet…

Testing still feels painful.

🚨 The Real Problem

We’ve improved how tests are created.

But not how they are understood.

Today, even with AI:

Tools generate scripts
Tools execute tests
Tools give logs

But when a test fails…

👉 We’re back to the same loop:

Open logs
Check screenshots
Replay videos
Try to reproduce
Guess

😤 What AI Didn’t Fix

AI helped us write tests faster.

But it didn’t solve:

Why did the test fail?

That question still takes the most time.

⏱️ The Hidden Cost

A failed test is not just a failure.

It’s:

15–30 minutes of debugging
Multiple tools involved
Context switching between dev & QA

And sometimes…

👉 It’s not even a real issue (just flaky behavior)

💡 What’s Actually Missing

We don’t need more test generation.

We need:

Test Intelligence

Systems that can:

Explain failures in plain English
Detect flaky patterns across runs
Connect failures to code changes
Recommend what to test next

🔄 A Different Way to Think About Testing

Instead of:

Generate → Run → Debug manually

What if it became:

Generate → Run → Understand instantly

⚙️ Example

Instead of this:

“Element not found”

Imagine seeing:

“Login button moved due to layout shift in header component after recent CSS change”

That’s the difference between:

❌ Data

✅ Understanding

🛠️ What This Means Practically

If you're working with modern testing stacks today:

You're not struggling with writing tests anymore
You're struggling with understanding failures
Most debugging still happens outside your main workflow

This is where most teams lose time — not execution, but investigation.

🧠 Key Insight

Testing isn’t broken because of lack of automation

It’s broken because of lack of insight.

🚀 Where This Is Going

The next phase of testing won’t be:

More frameworks
More scripts
More AI-generated code

It will be:

Systems that explain
Systems that learn from failures
Systems that guide testing decisions

We’ve been exploring this direction while building TestNeo:

👉 https://testneo.ai

Still early — would genuinely love feedback.

💬 Curious to hear from you

What takes more time in your workflow — writing tests or debugging them?

Tags:

testing qa automation ai devtools softwareengineering