DEV Community: Help Me test

Approval Testing Explained: What It Is and How It Works

Help Me test — Sat, 25 Jul 2026 16:10:54 +0000

What do you do when you can't write an expected value for your assertion? When the output is a complex JSON object, a rendered report, a formatted table, or an entire HTML page—and writing expect(output).toBe(...) would require typing out hundreds of lines?

Approval testing solves this. Instead of specifying expected output upfront, you capture real output, review it, and approve it as correct. Future test runs compare against that approved output. If anything changes, the test fails and shows you exactly what changed.

The Core Concept

Traditional assertion-based testing:

// You write the expected value
expect(calculateTax(100, "CA")).toBe(9.25);

Approval testing:

1. Run the code → get output
2. Review the output → "yes, this looks right"
3. Approve it → save as the "golden master"
4. On future runs: compare output to approved version
5. If different → test fails, shows diff

The key insight: you're reviewing, not pre-specifying. The first approval is a human judgment call. After that, it's automated regression detection.

Why It Exists

Some output is too complex to specify manually. Consider:

A PDF report with 200 fields
A rendered React component tree
A serialized domain object graph
A formatted email template
A generated SQL migration

Writing the full expected value is impractical. But you can look at the output and say "yes, that's correct." Approval testing turns that human judgment into a test.

How It Works in Practice

Step 1: First run (no approved file)

The test runs and produces output. No approved file exists, so the test fails. The framework shows you the output.

FAILED: no approved file found
Received output: 
  { "user": { "id": 1, "name": "Alice", "plan": "pro", "createdAt": "2025-01-15" }, 
    "invoice": { "amount": 100, "currency": "USD", "dueDate": "2025-02-15" } }

Step 2: Review and approve

You look at the output. It's correct. You approve it—this saves the output to a .approved.txt file.

tests/
  __approved__/
    UserInvoiceTest.produces_invoice.approved.txt

The approved file now contains exactly what the test produced.

Step 3: Subsequent runs

The test runs again. Output is compared to the approved file.

Same output: test passes
Different output: test fails with a diff

  { "user": { "id": 1, "name": "Alice", "plan": "pro", 
-   "createdAt": "2025-01-15"
+   "createdAt": "2025-01-15T00:00:00Z"
  }, ...

You see exactly what changed. Now you decide: is this an intentional change (approve the new output) or a regression (fix the code)?

A Complete Example

Using the approval-tests library for JavaScript:

import { verify } from "approvals/lib/Providers/Jest/JestApprovals";

function generateUserSummary(user, orders) {
  return {
    user: { id: user.id, name: user.name, memberSince: user.createdAt },
    orderCount: orders.length,
    totalSpent: orders.reduce((sum, o) => sum + o.amount, 0),
    lastOrder: orders[orders.length - 1]?.createdAt ?? null
  };
}

it("generates user summary", () => {
  const user = { id: 1, name: "Alice", createdAt: "2024-01-01" };
  const orders = [
    { id: 101, amount: 49.99, createdAt: "2025-01-10" },
    { id: 102, amount: 99.99, createdAt: "2025-01-20" }
  ];

  const summary = generateUserSummary(user, orders);

  // First run: fails, shows output, you approve it
  // Subsequent runs: compares to approved output
  verify(JSON.stringify(summary, null, 2));
});

On the first run, the test fails and opens a diff tool showing you the output. You review it, close the diff tool accepting the changes, and the approved file is created. Every subsequent run compares automatically.

The Approval Workflow

          ┌──────────┐
          │ Run Test │
          └────┬─────┘
               │
               ▼
         Output exists?
         /           \
        No            Yes
        │              │
        ▼              ▼
   Show output    Compare to
   → Approve?     approved file
        │              │
       Yes         Match?
        │          /     \
        ▼         Yes     No
   Save to        │        │
   approved       ▼        ▼
   file        PASS     Show diff
                        → Fix code
                          or approve

What Gets Approved

Approval tests can capture any string output:

JSON objects: serialized domain objects, API responses
Plain text: formatted reports, emails, logs
HTML: rendered templates, component output
XML: configuration, API responses
SQL: generated queries, migrations
Binary (via base64 or hex representation): images, PDFs (though this is rare)

The output must be deterministic. Random IDs, timestamps, and volatile data must be normalized before approval.

Handling Non-Determinism

The biggest challenge with approval testing is making output deterministic. Random values and timestamps change on every run.

function generateOrder(userId) {
  return {
    id: Math.random().toString(36), // PROBLEM: changes every run
    userId,
    createdAt: new Date().toISOString(), // PROBLEM: changes every run
    status: "pending"
  };
}

Solutions:

// Option 1: Inject deterministic values
function generateOrder(userId, { id = uuid(), createdAt = new Date() } = {}) {
  return { id, userId, createdAt: createdAt.toISOString(), status: "pending" };
}

// In test: inject fixed values
const order = generateOrder(1, { id: "order-001", createdAt: new Date("2025-01-01") });
verify(JSON.stringify(order, null, 2));

// Option 2: Scrub non-deterministic fields before approving
const order = generateOrder(1);
const scrubbed = { ...order, id: "[id]", createdAt: "[timestamp]" };
verify(JSON.stringify(scrubbed, null, 2));

Benefits

You don't need to know the expected output upfront. This is ideal for legacy code, complex algorithms, or systems that are being reverse-engineered.

Regression detection is automatic. Any change to output triggers a failure with a precise diff.

Tests are readable. The approved file is the expected behavior. New developers can read it to understand what the system produces.

Refactoring is safe. If behavior doesn't change, tests stay green. If it does change, you get an immediate, detailed diff.

Limitations

Approved files must be committed to version control
Output must be deterministic (timestamps, IDs need handling)
Initial approval requires human review time
Large approved files can be hard to read in diffs

How HelpMeTest Complements Approval Testing

Approval tests are excellent for verifying internal outputs—generated text, serialized objects, formatted reports. But they don't test the end-to-end user experience.

HelpMeTest runs plain-English tests against your live application—real browser, real flows. When your approval tests confirm the order summary generates correctly, HelpMeTest confirms that the order summary displays correctly to users in the actual UI.

Summary

Approval testing captures real output and uses it as the expected result
You review and approve output on the first run; automation handles subsequent runs
Any change triggers a diff—you decide if it's intentional (approve) or a regression (fix)
Best suited for complex output that's impractical to specify manually
Requires deterministic output—scrub or inject volatile values
Approved files live in version control alongside tests

Test Result Reporting and Failing Fast in CI Pipelines

Help Me test — Sat, 25 Jul 2026 15:52:27 +0000

A test failure that takes 20 minutes to surface, buries the error in 3000 lines of log output, and gives no context about what changed is nearly useless. Good test reporting transforms raw pass/fail data into actionable signals. Failing fast — stopping the pipeline the moment you have enough information to make a decision — keeps feedback loops tight and respects developer time.

These two concerns are deeply connected: you can only fail fast confidently when your reporting is good enough that a fast failure still gives you everything you need to fix the problem.

What Good Test Reporting Looks Like

Before discussing implementation, it's worth being precise about what "good" means here:

Immediate visibility — failures are surfaced at the PR/commit level, not buried in logs
Failure context — what failed, with what input, producing what output, and in which file/line
Historical comparison — is this a new failure or a pre-existing one?
Trend data — is this test getting flakier? Is the suite getting slower?
Actionability — the report points to a fix, not just a symptom

Most teams get #1 and stop. The teams that nail all five have fundamentally different debugging velocity.

JUnit XML: The Universal Format

JUnit XML is the lingua franca of CI test reporting. Almost every test framework can emit it, and almost every CI platform can ingest it. Understanding the format helps you produce better reports.

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="My Test Suite" tests="42" failures="2" errors="0" time="8.432">
  <testsuite name="UserService" tests="15" failures="1" time="2.1">
    <testcase name="should create user with valid email" classname="UserService" time="0.234">
      <!-- Empty = passed -->
    </testcase>
    <testcase name="should reject duplicate email" classname="UserService" time="0.089">
      <failure message="Expected 409, got 200" type="AssertionError">
        Expected status code 409 but received 200

        Request: POST /api/users
        Body: {"email": "existing@example.com"}

        at UserService.test.js:47:5
        at Object.<anonymous> (UserService.test.js:44:3)
      </failure>
    </testcase>
  </testsuite>
</testsuites>

The failure element's content is what gets displayed in CI UIs. Invest in making it descriptive. Compare these two failure messages:

Bad:  "AssertionError: expected 404 to equal 200"
Good: "POST /api/users/999 returned 404 Not Found. Expected 200 OK. User ID 999 does not exist in test database — check test setup in beforeAll()"

The second message often eliminates the need to read logs at all.

Configuring Test Reporters

Jest

// jest.config.js
module.exports = {
  reporters: [
    'default',  // Console output during test run
    ['jest-junit', {
      outputDirectory: 'test-results',
      outputName: 'junit.xml',
      classNameTemplate: '{classname}',
      titleTemplate: '{title}',
      ancestorSeparator: ' › ',
      usePathForSuiteName: true,
    }],
    ['jest-html-reporters', {
      publicPath: './test-results',
      filename: 'report.html',
      openReport: false,
      includeFailureMsg: true,
      includeConsoleLog: true,
    }],
  ],
};

For richer failure context, customize your test assertions:

// Better assertion messages
expect(response.status).toBe(200); // Poor: "Expected 404, received 200"

expect(response.status).toBe(200); // Better with custom message:
expect(response.status, 
  `POST ${path} failed with ${response.status}: ${JSON.stringify(response.body)}`
).toBe(200);

// Or use a helper
function assertStatus(response, expected) {
  expect(response.status).toBe(expected,
    `Expected ${expected}, got ${response.status} ${response.statusText}\n` +
    `URL: ${response.url}\n` +
    `Body: ${JSON.stringify(response.body, null, 2)}`
  );
}

Pytest

# pytest.ini
[pytest]
addopts = 
    --junit-xml=test-results/junit.xml
    --junit-logging=all
    -v
    --tb=short

# conftest.py — add custom metadata to reports
import pytest

def pytest_runtest_makereport(item, call):
    """Add extra context to failure reports."""
    if call.when == "call" and call.excinfo:
        # Attach any fixtures' state to the failure
        if hasattr(item, 'funcargs'):
            for name, value in item.funcargs.items():
                if hasattr(value, '__dict__'):
                    item.add_report_section(
                        'call',
                        f'Fixture state: {name}',
                        str(value.__dict__)
                    )

# Use pytest-html for rich HTML reports
# pip install pytest-html

# pytest.ini
addopts = --html=test-results/report.html --self-contained-html

Go

# go test produces output but not JUnit by default
# Use gotestsum for JUnit output + better formatting
go install gotest.tools/gotestsum@latest

gotestsum --junitfile test-results/junit.xml --format testname ./...

// For richer failure messages in Go, use testify
import "github.com/stretchr/testify/assert"

func TestUserCreation(t *testing.T) {
    user, err := CreateUser("test@example.com")

    // Bad: "Not equal: expected 1, got 0"
    assert.Equal(t, 1, user.ID)

    // Better: includes context automatically
    assert.NoError(t, err, "CreateUser should not return an error for valid email")
    assert.NotZero(t, user.ID, "Created user should have a non-zero ID")
}

Platform-Specific Reporting Integration

GitHub Actions

- name: Run tests
  run: npm test -- --reporters=jest-junit

- name: Publish test results
  uses: dorny/test-reporter@v1
  if: always()  # Run even if tests failed
  with:
    name: Jest Test Results
    path: test-results/junit.xml
    reporter: jest-junit
    fail-on-error: true

The if: always() is critical — you want the report even (especially) when tests fail.

For annotating individual lines in PRs with failures:

- name: Annotate test failures
  if: failure()
  uses: mikepenz/action-junit-report@v4
  with:
    report_paths: 'test-results/**/*.xml'
    include_passed: false
    annotate_only: false

This creates inline PR annotations pointing to the exact file and line where each test failed.

GitLab CI

unit-tests:
  script:
    - npm test -- --reporters=jest-junit
  artifacts:
    when: always
    reports:
      junit: test-results/junit.xml
    paths:
      - test-results/
      - coverage/
    expire_in: 30 days
  coverage: '/All files[^|]*\|[^|]*\s+([\d\.]+)/'

GitLab's native JUnit support shows test results directly in the merge request UI, including a diff view showing which tests are new failures vs. pre-existing.

Jenkins

post {
    always {
        junit(
            testResults: 'test-results/**/*.xml',
            allowEmptyResults: true,
            keepLongStdio: true
        )

        publishHTML([
            reportDir: 'test-results',
            reportFiles: 'report.html',
            reportName: 'Test Report',
            keepAll: true,
            alwaysLinkToLastBuild: false,
        ])

        // Trend analysis across builds
        perfReport(
            sourceDataFiles: 'test-results/junit.xml',
            modePerformancePerTestCase: true
        )
    }
}

Fail-Fast Strategies

Failing fast means stopping as soon as you have sufficient signal that the build should not proceed. The key word is "sufficient" — stopping too early wastes information; stopping too late wastes time.

Stage-Level Fail-Fast

The most basic form: don't run slow stages if fast stages fail.

# GitHub Actions — jobs depend on each other
jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - run: npm run lint

  unit-tests:
    needs: lint  # Only runs if lint passes
    runs-on: ubuntu-latest
    steps:
      - run: npm test:unit

  integration-tests:
    needs: unit-tests  # Only runs if unit tests pass
    runs-on: ubuntu-latest
    steps:
      - run: npm test:integration

# GitLab CI — stages enforce ordering
stages:
  - lint
  - unit
  - integration
  - e2e

# Jobs in 'unit' stage only run if 'lint' stage passes
# Jobs in 'integration' stage only run if 'unit' stage passes

Test-Level Fail-Fast (Bail)

Stop running tests within a suite after the first N failures:

# Jest — stop after first failure
npx jest --bail=1

# Or after 3 failures (useful for flaky test detection)
npx jest --bail=3

# Pytest
pytest --maxfail=1 tests/

# Go
go test -failfast ./...

The tradeoff: --bail=1 gives fastest feedback on catastrophic failures (like a missing import or broken test setup) but gives you less information when multiple independent tests fail. --bail=5 is often a better default — it catches cascading failures while still collecting enough data to work from.

Conditional Fail-Fast Based on Changed Files

For large monorepos, run only tests related to changed files:

// scripts/get-affected-tests.js
const { execSync } = require('child_process');

const changedFiles = execSync('git diff --name-only origin/main...HEAD')
  .toString()
  .split('\n')
  .filter(Boolean);

const testPatterns = new Set();

for (const file of changedFiles) {
  if (file.startsWith('src/auth/')) {
    testPatterns.add('auth');
  }
  if (file.startsWith('src/payments/')) {
    testPatterns.add('payments');
  }
  if (file.startsWith('src/')) {
    // Any src change runs the full unit suite
    testPatterns.add('unit');
  }
}

// Output as Jest --testPathPattern
console.log([...testPatterns].join('|'));

- name: Get affected test patterns
  id: affected
  run: echo "pattern=$(node scripts/get-affected-tests.js)" >> $GITHUB_OUTPUT

- name: Run affected tests
  run: npx jest --testPathPattern="${{ steps.affected.outputs.pattern }}"

Timeout-Based Fail-Fast

A test that hangs indefinitely is worse than a test that fails — it blocks the pipeline and provides no information.

// jest.config.js
module.exports = {
  testTimeout: 10000,  // 10 seconds per test

  // For entire suite
  globals: {
    __TEST_TIMEOUT__: 30000,
  }
};

# GitHub Actions — job-level timeout
jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 15  # Kill the job if it takes more than 15 minutes

# pytest timeout per test
# pip install pytest-timeout
@pytest.mark.timeout(30)
def test_slow_operation():
    # Will fail if it takes more than 30 seconds
    result = slow_api_call()
    assert result is not None

Flaky Test Detection and Reporting

Flaky tests undermine trust in your entire test suite. Detecting them requires tracking failure rates over time, not just current state.

# Run failed tests multiple times to detect flakiness
- name: Run tests with retry
  run: npx jest --reporter=jest-junit --retries=2

# If a test passes on retry, mark it as flaky (not failed) in the report

A more systematic approach uses test history:

// scripts/detect-flaky.js
// Compare this run's results with the last 10 runs
// Tests that fail < 100% of the time are flaky candidates

const currentResults = parseJUnit('./test-results/junit.xml');
const historicalResults = loadHistory('./test-history/');

const flaky = currentResults.failures.filter(failure => {
  const history = historicalResults[failure.name] || [];
  const failureRate = history.filter(r => r === 'fail').length / history.length;
  return failureRate < 1.0 && failureRate > 0;
});

if (flaky.length > 0) {
  console.warn(`Potentially flaky tests detected:\n${flaky.map(t => `  - ${t.name}`).join('\n')}`);
  // Don't fail the build for flaky tests — but do report them
}

Building a Test Health Dashboard

Beyond individual run reports, tracking metrics over time reveals trends that single-run reports miss.

Key metrics to track:

// After each CI run, emit metrics to your monitoring system
const metrics = {
  total_tests: results.total,
  passed: results.passed,
  failed: results.failed,
  skipped: results.skipped,
  duration_seconds: results.duration,
  slowest_test: results.slowestTest,
  flaky_tests: results.flaky.length,
  branch: process.env.GITHUB_REF,
  commit: process.env.GITHUB_SHA,
  timestamp: new Date().toISOString(),
};

// Push to your analytics backend
await fetch(process.env.METRICS_ENDPOINT, {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify(metrics),
});

The most valuable trend to watch: test suite duration over time. A test suite that grows 10% slower every month will become a problem within a year. Catching this trend early — before it becomes a crisis — lets you address it incrementally rather than with a painful "fix the test suite" sprint.

Good test reporting isn't a CI configuration task you do once — it's an ongoing investment that pays off every time someone can diagnose and fix a test failure in 5 minutes instead of 50.

The AI Testing Chasm: Why 75% of QA Teams Plan to Use AI But Only 16% Actually Do

Help Me test — Sat, 14 Mar 2026 00:46:20 +0000

A Reddit thread in r/QualityAssurance asked a simple question: "Is anyone successfully using gen AI in their QA?" The responses told the real story — not the one you see in tech press. Most teams are experimenting. Few are shipping. The chasm between what media reports and what practitioners experience is one of the biggest disconnects in software engineering right now.

Key Takeaways

75% of organizations call AI-driven testing "pivotal" to their 2025 strategy. Only 16% have actually adopted it. That gap — 59 percentage points — is the chasm nobody in tech media is talking about.

Developer trust in AI is falling even as usage rises. 84% of developers now use AI tools, but only 33% trust the output. In 2024, 31% distrusted AI accuracy. In 2025, that number jumped to 46%.

Experience = skepticism. Early-career developers are the most enthusiastic AI adopters. Engineers with 10+ years — who understand what bugs actually cost — are the most cautious.

The chasm exists for structural reasons, not laziness. AI can't understand business logic, context, or the reason a test should exist. It can generate test code. It cannot generate test judgment.

What is the AI adoption chasm in QA? The AI adoption chasm in software testing is the gap between how broadly AI is discussed as a solution to QA problems versus how rarely it's actually deployed in production testing workflows. Survey data consistently shows 5–8x more teams plan to use AI than have successfully implemented it — creating a persistent, measurable disconnect between industry narrative and practitioner reality.

Head to any tech conference in 2025 and you'll see a wall of talks about AI-powered testing. Open LinkedIn and you'll find announcements about AI that "revolutionizes QA." Read the vendor press releases and you might conclude that every engineering team has already automated testing with AI agents.

Then go to r/QualityAssurance and ask: "Is anyone successfully using gen AI in their QA?"

The responses will bring you back to earth.

The Numbers Behind the Chasm
What the Media Gets Wrong
Why the Chasm Exists
The Trust Curve in Practice
What's Actually Working
The Human Review Requirement
The Leadership-Frontline Disconnect
What to Do With This
The Bottom Line

The Numbers Behind the Chasm

The disconnect between AI hype and adoption reality isn't anecdotal — it's measurable.

The strategy-to-adoption gap:

In a Perforce industry survey, 75% of respondents identified AI-driven testing as a pivotal component of their 2025 strategy. Only 16% had actually adopted it. That's a 59-percentage-point gap between what teams say they'll do and what they're doing.

An earlier data point shows how slowly this moves: 48% of respondents said they were interested in AI for testing but hadn't started any initiatives. Only 11% were implementing AI techniques in software testing.

The trust paradox:

The 2025 Stack Overflow Developer Survey found that 84% of developers now use AI tools — up substantially from prior years. But only 33% trust AI output. Just 3% describe themselves as highly trusting of AI-generated code or tests.

More troubling: trust is falling as usage rises. In 2024, 31% of developers said they distrusted AI accuracy. In 2025, that number jumped to 46%. The more people actually work with AI day-to-day, the less they believe its output.

The agent gap:

AI assistants (Copilot, Claude, ChatGPT for code) are one thing. Autonomous AI agents — systems that take actions with minimal human oversight — are another. Only 31% of developers are using AI agents to any degree. 38% have no plans to adopt them at all.

Deloitte's Tech Trends 2026 report found that only 11% of companies have AI agents fully operational in production, despite 25% running pilots. The gap between "experimenting" and "deployed" is vast, and it isn't closing quickly.

What the Media Gets Wrong

The technology press has structural incentives to cover AI adoption as further along than it is. "75% of teams using AI for testing" is a more compelling headline than "16% have adopted, 48% are still thinking about it." Vendor press releases announce product launches, not the percentage of teams that tried the product and stopped.

This creates a persistent narrative that practitioners have to push back against constantly.

The narrative: AI is replacing QA engineers and automating testing end-to-end. The reality: 45% of practitioners believe manual testing is irreplaceable, and actual AI implementations focus on augmenting specific tasks — not replacing testing as a discipline.

The narrative: AI-generated tests are saving teams massive time. The reality: A survey of more than 600 software developers across four countries found that teams using AI testing tools weren't yet saving time on tedious tasks. The tools required enough oversight and correction that efficiency gains largely evaporated.

The narrative: Everyone in QA is using AI. The reality: The 2025 research paper "Expectations vs Reality: A Secondary Study on AI Adoption in Software Testing" found only 17 peer-reviewed and grey literature studies that examined real industry adoption — confirming that empirical research on AI in testing remains sparse despite extensive academic and media interest.

Why the Chasm Exists

The gap isn't about teams being slow or resistant to change. It's structural. Testing has properties that make it genuinely harder to automate with current AI than most work.

AI can generate test code. It cannot generate test judgment.

A test doesn't just execute — it asserts something meaningful. It claims: "this behavior is correct." Generating assertions requires knowing what correct behavior is, which requires understanding the business logic the software is supposed to implement. AI doesn't have that context. You do.

This leads to a specific failure mode practitioners report repeatedly: AI generates tests that execute without errors but validate nothing meaningful. The tests pass. The bugs ship anyway.

The overfitting problem:

AI test generation tends to overfit to happy paths — the flows that are most documented, most obvious, most similar to examples in training data. Happy paths are the least valuable tests to generate. They're the flows most likely to already be working.

The bugs that matter live in edge cases, error states, race conditions, and interactions between systems. These are exactly the scenarios where AI has the least signal and produces the least useful output.

Domain expertise doesn't transfer:

"Does AI have enough domain knowledge?" is the question practitioners ask immediately after seeing AI-generated test cases. Usually, the answer is no. AI doesn't know your product's business rules, its legacy constraints, the workarounds that exist for historical reasons, or the specific failure modes that have burned your team before. An experienced QA engineer carries that knowledge. AI starts fresh every time.

AI is an amplifier, not a foundation:

One of the clearest insights from practitioners: AI won't save weak testing processes. If your test strategy is unclear, AI-generated tests will be unclear tests, faster. If your automation is fragile, AI will add fragile tests to your fragile suite. AI amplifies what's already there — which is exactly why teams with strong fundamentals get value from it and teams without fundamentals get more noise.

The Trust Curve in Practice

The pattern of AI adoption in QA follows a recognizable shape:

Phase 1 — Enthusiasm: Team hears about AI testing tools. Leadership gets excited. There's a push to adopt.

Phase 2 — Experimentation: Someone runs AI test generation on a feature. Output looks plausible. Quick wins seem possible.

Phase 3 — Reality check: Generated tests start failing. Maintenance burden appears. Domain-specific assertions are wrong. Human review takes as long as writing the tests manually would have.

Phase 4 — Selective use: Teams identify the specific tasks where AI actually helps — generating boilerplate, suggesting test structures, handling repetitive variations — and stop expecting it to do the work that requires judgment.

The developers who've gone through all four phases are the ones who report the lowest trust in AI output. They've seen the failure modes. Their skepticism is earned.

This explains a counterintuitive finding from the 2025 Stack Overflow survey: experienced developers (10+ years) are the most skeptical of AI, while early-career developers are the most enthusiastic. Experienced engineers have the longest history of debugging AI-generated code. They know what it costs when an AI-generated test passes on code that's actually broken.

What's Actually Working

The chasm doesn't mean AI has no value in QA. It means the value is narrower and more conditional than the narrative suggests.

Where practitioners report genuine wins:

Boilerplate generation: Scaffolding test files, generating repetitive parametrized variations, writing setup/teardown that follows patterns. AI is good at pattern repetition.
Test name and description writing: AI generates readable test descriptions faster than humans write them.
Root cause analysis assistance: Pasting failing test output and stack traces into AI for initial debugging suggestions.
Test data generation: Creating structurally valid but semantically unusual inputs that expose edge case bugs (there's a reason our AI hallucination test data post resonated with QA engineers).
Self-healing tests: AI that detects when UI selectors break and suggests updated locators, reducing maintenance burden on automated UI test suites.

Where AI consistently underdelivers:

Generating assertions that validate business rules (requires domain knowledge)
End-to-end test design (requires understanding user intent and failure scenarios)
Complex integration testing (requires understanding system dependencies)
Security and performance test design (requires adversarial thinking AI doesn't do well)

The teams extracting real value from AI in QA are using it on the first list, not the second.

The Human Review Requirement

Perhaps the most significant data point in the research: 67% of testers say they would trust AI-generated tests — but only with mandatory human review.

This isn't a ringing endorsement. It's an acknowledgment that AI can produce useful raw material, but the judgment about whether that material is correct still sits entirely with humans. The labor of review doesn't disappear — it shifts.

By early 2025, 76% of enterprises had implemented explicit human-in-the-loop (HITL) review processes for AI outputs. Knowledge workers spend an average of 4.3 hours per week reviewing and fact-checking AI outputs. This is new work that didn't exist before.

The question teams need to answer honestly: does AI generate tests fast enough to justify the time spent reviewing them? The answer varies by context, team, and tool. But the "AI saves time" narrative glosses over the review burden that comes with it.

The Leadership-Frontline Disconnect

One data point that doesn't get enough attention: there's a significant gap between how leadership perceives AI adoption and how frontline practitioners experience it.

Research shows 39% of research and engineering leaders say AI has revolutionized their processes. Only 19% of frontline engineers agree.

This disconnect matters because it shapes expectations. When leadership believes AI has already transformed QA, they may cut QA headcount, set unrealistic timelines, or stop investing in test infrastructure — based on an adoption state that doesn't actually exist on their teams.

QA engineers navigating this gap face a difficult position: they know the AI tools aren't delivering what leadership thinks they are, but pushing back risks being labeled as resistant to change.

The chasm isn't just between media narrative and practitioner reality. It's between the conference room and the pull request.

What to Do With This

If you're a QA engineer trying to figure out where AI fits in your workflow:

Start from specific problems, not capabilities. Don't ask "how do I use AI in my testing?" Ask "what's the most painful, repetitive part of my testing workflow?" Then check whether an AI tool addresses that specific pain. Usually the answer is more targeted than "AI-powered testing."

Trust the skeptics in your network. The engineers with the most experience and the most skepticism about AI in testing are usually the ones who've actually tried it seriously. Their caution is information, not stubbornness.

Separate hype from your context. A tool that works well for a large team with mature test infrastructure may not work for a startup with three engineers and no test suite. Adoption statistics mix these contexts. Your situation is specific.

Treat AI as an amplifier. If your testing fundamentals are strong — clear test strategy, good coverage of critical paths, reliable automation — AI can make those fundamentals more efficient. If your fundamentals are weak, AI will accelerate the chaos.

Be honest about the review burden. If reviewing AI-generated tests takes 80% of the time that writing them manually would, the efficiency gain is 20%. That may still be worth it. But it may not. Calculate honestly.

The Bottom Line

The chasm between AI testing hype and QA adoption reality is real, documented, and not closing as fast as the press suggests. The gap between "75% say AI is pivotal to their strategy" and "16% have actually adopted it" is 59 percentage points. That's not a small lag — it's a fundamental disconnect between aspiration and implementation.

The reasons for the gap aren't mysterious: AI doesn't have domain knowledge, can't generate meaningful assertions without context, and consistently overfits to happy paths. The most experienced engineers know this. Their skepticism is the result of trying, not avoiding.

The practical path forward isn't wholesale AI adoption or wholesale rejection. It's honest assessment of where AI helps in your specific context — which requires ignoring most of what tech media says about it and paying close attention to what practitioners in your situation actually report.

The Reddit thread asking whether anyone is successfully using gen AI in QA wasn't a sign of ignorance. It was a sign that practitioners know the difference between what's marketed and what works.

HelpMeTest uses AI for test generation, self-healing selectors, and visual flaw detection — the specific tasks where AI consistently delivers value in QA workflows. See how it works.