DEV Community: Anand Pawar

How Much of Your CI Pipeline Is Just Cucumber Scenarios You're Too Afraid to Delete

Anand Pawar — Sun, 26 Jul 2026 18:39:59 +0000

The CI job just hit 28 minutes. Again.

You pull up the duration report expecting to blame a bloated integration test or a slow environment spin‑up. Instead the longest stage stares back at you: a collection of Cucumber feature files that haven’t caught a real bug in months. Maybe years. They run on every commit, green circle after green circle, while your team mutters about slow pipelines and nobody dares touch them.

Most teams treat those scenarios like documentation. “They describe the system,” someone once said, as if a Gherkin file were a legal contract. Others cling to the sunk cost: a year ago a whole squad spent two sprints writing them, polishing the grammar, aligning step definitions. Deleting them would feel like admitting waste.

Experienced engineers see it differently. They treat a scenario that never fails as a liability you’re paying for on every push. Not neutral. Liable. Compute cycles, developer attention, flake‑debugging time, and the quiet toll it takes on trust in the pipeline.

The principle is blunt: if a test hasn’t failed in the last few sprints, you’re already paying its full cost and receiving nothing in return. That doesn’t mean you delete everything green. But it does mean you audit with the same seriousness you’d use for a memory leak.

What the green wall actually costs

The damage is not abstract. A pipeline bloated with stale scenarios hurts you in five concrete ways.

First, feedback slows. Every extra minute between push and result stretches the loop that tells a developer they’re safe to merge. Multiply across a team and you’re losing hours per week to waiting.

Second, flakiness increases. When you have many scenarios, a single unstable environment variable can produce a handful of failures that are not regressions at all. Engineers learn to retry, then to ignore.

Third, confidence erodes. If half the suite is ceremonial, a genuine failure might be dismissed as “just another flaky test” until it reaches production.

Fourth, maintenance compounds. When you rename a button or re‑orchestrate a flow, someone still has to update the corresponding step definitions. Stale scenarios become tax with every refactor.

Fifth, you lose the ability to reason about coverage. A slim, surgical set of tests tells a clear story about what’s protected. A wall of noise makes it impossible to know what would actually break if you removed something.

How to see the rot

You can’t fix what you don’t measure. The first step is to gather execution data from CI. Most test runners produce a JSON or JUnit report that includes scenario names and outcomes. If your current setup doesn’t, it’s worth a couple of lines of CI script to capture it.

The lowest‑hanging fruit are scenarios that never run at all. Feature files left behind after a feature was rewritten, scenarios that reference pages that no longer exist, step definitions that are unreachable. They’re invisible because the runner skips them silently.

Here’s a TypeScript utility that compares all scenario names in your feature files against what your CI report says actually executed recently. It’s deliberately minimal — you’d want a proper Gherkin parser and unique IDs for production use, but this gives you the shape.

import fs from 'fs';
import path from 'path';

const featureDir = './features';
const reportData = JSON.parse(fs.readFileSync('./ci-report.json', 'utf-8'));
const executedNames = new Set(reportData.scenarios.map((s: any) => s.name));

const stale: string[] = [];
const featureFiles = fs.readdirSync(featureDir).filter(f => f.endsWith('.feature'));
for (const file of featureFiles) {
  const content = fs.readFileSync(path.join(featureDir, file), 'utf-8');
  const scenarioPattern = /Scenario:\s*(.+)$/gm;
  let match;
  while ((match = scenarioPattern.exec(content)) !== null) {
    const name = match[1].trim();
    if (!executedNames.has(name)) {
      stale.push(`${file}: "${name}"`);
    }
  }
}

console.log(`Found ${stale.length} scenario(s) never seen in the last CI run:`);
console.log(stale.join('\n'));

Running that once gives you a list of candidates for immediate deletion. If a scenario isn’t being executed, it’s not guarding anything. Cut it.

The harder category is the scenario that runs but never fails. For that you need a window of history. A small Python script can parse a directory of archived reports and flag any scenario with zero failures in, say, the last 90 days. The logic is straightforward: accumulate pass/fail counts per unique scenario ID and mark those with failure_count == 0. I won’t paste the full script here, but you can write it in twenty lines.

Once you have your stale list, don’t delete everything at once and trigger alarm bells. Remove the bottom 5% and watch a few cycles. Often nothing changes except the pipeline runs a little faster.

The Cucumber tax on simple flows

You know what a login scenario looks like in Gherkin, and you know how much ceremony surrounds it. A feature file with background steps, a half‑dozen step definitions scattered across files, hooks for driver management, perhaps custom parameter types. All to say “fill two fields, click a button, see a dashboard.”

Now look at the same intent expressed as a Playwright test.

import { test, expect } from '@playwright/test';

test('successful login', async ({ page }) => {
  await page.goto('/login');
  await page.fill('[data-testid="username"]', 'user');
  await page.fill('[data-testid="password"]', 'pass');
  await page.click('[data-testid="login-button"]');
  await expect(page.locator('[data-testid="dashboard"]')).toBeVisible();
});

This isn’t a dig at BDD as a practice for collaboration. It’s a recognition that many teams used Cucumber to write UI tests — often years ago when the tooling was different — and those tests are now just verbose wrappers around interactions that the framework itself describes more clearly.

The problem compounds when step definitions grow organically. “Given I

3 Portfolio Mistakes Hiring Managers Spot Instantly

Anand Pawar — Sun, 26 Jul 2026 18:39:40 +0000

The manager opens your portfolio.

Your resume says you have five years of automation experience. The README lists Selenium, Playwright, Appium, Jenkins, Docker, Kubernetes. He scrolls. There is no code. The browser tab closes.

This is you. Not because you lack skill—you have it—but because your public proof reads like a shopping list. The tools you name say nothing about how you think when a flaky test fails at 2am, or how you convince a developer that a bug is real.

If you’re serious about landing a role that demands more than record-and-playback, you need to stop treating your portfolio like a keyword bingo card. Here are three mistakes that kill your chances instantly, and exactly how to fix them.

Mistake 1: Tool jockeying

Listing every automation framework you’ve heard of is a reflex. A hiring manager sees "Proficient in Cypress, Playwright, Selenium, WebDriverIO" and assumes you ran npm init once in each and called it done.

Most testers frontload tools because they’re scared of the empty space where code belongs.

Experienced testers show one test, deliberately written, with a comment that explains a trade-off they chose.

The difference is not volume. A single 30-line script that handles a login flow with a purposeful wait strategy teaches more about you than a six-tool résumé. I’ve deleted my own old projects after re-reading them and realizing they said nothing about why any assertion existed. That quiet cringe is the signal you’re ready to improve.

What you ship in your portfolio must answer one question: "What did this person decide, and why?" Move your tool list to a footnote. Let a real test carry the message.

Mistake 2: The perfect test trap

A portfolio full of green builds is a trap. Every team knows that real automation breaks: the CI node runs slow, the third-party API throttles you, the DOM renders a fraction of a second late. Showing only passing tests hides how you handle the ugly parts of the job.

Most testers polish every assertion until it’s spotless. Experienced testers include retry logic, explicit logging, and evidence that they anticipated failure.

The fix is not to add five more test cases. It’s to embed the thinking you already do when you debug. Instrument your showcase test so it survives a hiccup and leaves breadcrumbs for the next engineer. Write it as if you’re handing it to a teammate who will curse your name if it fails silently.

Here is a runnable Python snippet that illustrates the kind of decision-making a manager wants to see. It uses Playwright to log in and retries on transient failures while capturing screenshots.

from playwright.sync_api import sync_playwright, expect

def test_login_with_retry(page):
    for attempt in range(3):
        try:
            page.goto("/login")
            page.fill("#username", "user")
            page.fill("#password", "pass")
            page.click("button[type=submit]")
            expect(page.locator(".welcome")).to_be_visible(timeout=5000)
            break
        except Exception as e:
            # Log explicitly so failure context is preserved
            print(f"Login attempt {attempt+1} failed: {e}")
            if attempt == 2:
                raise
            page.screenshot(path=f"login_failure_attempt_{attempt}.png")

This is not complicated. But it tells a story: you know that UI tests are fragile, you choose to retry instead of immediately failing, and you leave artifacts that make debugging faster. The technical detail is the logging and the screenshot, not the locator strategy.

What this teaches a hiring manager: you understand that automation is a maintenance job, not a writing job. That distinction separates someone who can own a suite from someone who just fills a ticket.

Mistake 3: No evidence of impact

Saying "reduced manual testing time by 70%" without linking to a single regression you caught is noise. The number is meaningless unless a real feature shipped because of it.

Most testers describe their work in generic efficiency terms. Experienced testers connect a specific test to a specific risk that was mitigated.

Your portfolio doesn’t need a case study that reads like a press release. It needs one paragraph that states: "This suite protects the checkout flow. The retry logic caught a race condition in the payment gateway integration that would have charged customers twice. Here’s the commit that added the fix." A link to a GitHub issue or a closed bug report is worth more than any percentage you can invent.

A hiring manager who reads that will stop scanning and start evaluating your judgment. That moment is the whole point.

This week’s action

Open your GitHub profile or your pinned repo. Find the test you’re least proud of. The one that’s too clean, or too thin, or too quiet. Add the logging you know it needs. Write a sentence above it that says why it exists and what happens if it breaks. Push it.

Could you read your own portfolio cold and point to the decision that proves you’re not just another tester who can copy-paste from docs? If you can’t, fix that before you apply anywhere else.

The Manual Tester Who Can Write a SQL Join Will Always Beat the SDET Who Can't

Anand Pawar — Sun, 26 Jul 2026 18:39:16 +0000

Most people think the SDET title means you are automatically more valuable than a manual tester.

The SDET writes Playwright scripts. The SDET configures CI pipelines. The SDET talks about page objects and retry strategies. The manual tester clicks through screens and writes bug reports.

Here is the truth I have watched play out across teams: the manual tester who can write a SQL join will consistently outperform the SDET who cannot.

Not because SQL is magic. Because SQL is the shortest path to understanding what the system actually stores, not what the UI shows you.

The problem with automation-first thinking

I have seen SDETs spend three sprints building a test suite that validates every button, every dropdown, every error toast. The suite passes in CI. The suite passes in staging. The suite passes in production.

And the bug still ships.

Why? Because the test checked that the UI rendered correctly. It never checked that the database actually saved the right record. The SDET wrote assertions against DOM elements, not against data.

The manual tester, meanwhile, ran a simple query. Saw the order status was "pending" when it should have been "confirmed." Filed a bug with the exact SQL that proved the issue. The developer fixed it in ten minutes.

That is not a story about manual versus automated. That is a story about data literacy versus UI obsession.

What a SQL join gives you that a locator never will

A Playwright locator tells you something is on the screen. A SQL join tells you something is true.

When you write page.getByText('Order confirmed'), you are testing that the frontend displays those words. You are not testing that the backend actually confirmed the order. You are not testing that the payment gateway returned success. You are not testing that the inventory decremented.

A SQL join connects those dots.

SELECT o.id, o.status, p.status AS payment_status, i.quantity AS remaining_stock
FROM orders o
JOIN payments p ON o.id = p.order_id
JOIN inventory i ON o.product_id = i.product_id
WHERE o.id = 12345;

One query tells you: the order exists, the payment cleared, and the stock dropped. That is a complete end-to-end assertion. No UI needed.

The manual tester who writes that query knows more about the system's health after five seconds than the SDET who watched a green test suite for five minutes.

The real skill gap is not automation versus manual

It is data literacy versus tool fluency.

I have coached testers who could not write a single Playwright test but could explain exactly how a financial reconciliation should work. They knew which tables held the source of truth. They knew which joins would reveal mismatches. They could prove a bug existed before anyone wrote a single line of automation.

I have also worked with SDETs who could build a parallel test suite across three browsers but could not tell you what a foreign key constraint does. They wrote tests that passed because the UI looked right. The data underneath was wrong.

Which one would you rather have on a payment system? On a healthcare platform? On anything where correctness matters more than pixel alignment?

A concrete example: testing a refund flow

Here is a common scenario. A user requests a refund. The UI shows "Refund processed." The Playwright test clicks the refund button, waits for the toast, and passes.

But the refund never actually hit the payment gateway. The database shows refund_status = 'failed'. The UI just displayed a success message because the frontend assumed the API call succeeded.

The SDET who only tests the UI will ship that bug. The manual tester who runs this query will catch it:

SELECT r.id, r.status, r.amount, r.error_message
FROM refunds r
WHERE r.order_id = 12345 AND r.created_at > NOW() - INTERVAL '5 minutes';

The error_message column tells the story. The UI never showed it. The test never checked it. The data never lied.

How to build data literacy without becoming a DBA

You do not need to be a database administrator. You need three things.

First, learn to read a schema. Open your database client. Look at the tables. Understand which columns are foreign keys. That is the map of how your system connects.

Second, learn three SQL patterns. SELECT with WHERE. JOIN between two tables. GROUP BY with HAVING. That covers ninety percent of what you will need as a tester.

Third, write a test that queries the database after every UI action. Here is a minimal Playwright example in TypeScript that does exactly that:

import { test, expect } from '@playwright/test';
import { Pool } from 'pg';

const pool = new Pool({
  connectionString: process.env.DATABASE_URL
});

test('order confirmation persists in database', async ({ page }) => {
  await page.goto('/checkout');
  await page.fill('#email', 'test@example.com');
  await page.click('button[type="submit"]');
  await expect(page.getByText('Order confirmed')).toBeVisible();

  const result = await pool.query(
    `SELECT status FROM orders WHERE email = $1 ORDER BY created_at DESC LIMIT 1`,
    ['test@example.com']
  );

  expect(result.rows[0].status).toBe('confirmed');
  await pool.end();
});

That is not complicated. That is one query after one assertion. It catches the refund bug, the payment bug, the inventory bug. The UI test alone catches none of them.

What this teaches about career growth

The market rewards people who can prove things. Not people who can click things. Not people who can write the most elegant Playwright page object.

A manual tester who writes a SQL join proves a bug exists. An SDET who only writes UI assertions proves a button rendered. One of those is more valuable to a business that ships real products.

I have watched testers double their impact in a quarter by adding one database query to their existing test suite. They did not learn a new framework. They did not buy a course. They learned to ask the database what happened.

The SDET who cannot do that is building a career on a fragile foundation. Frameworks change. Locators break. Browsers update. Data outlasts all of it.

Your move

If you are a manual tester, open your database client today. Find the orders table. Write a join against the payments table. See what you learn.

If you are an SDET, add one database assertion to your next test. Not a UI check. A data check. See how many bugs you catch that your locators missed.

The title on your resume does not determine your value. The questions you can answer about the system do.

What is one bug you caught by looking at the database that your UI tests missed entirely? I would like to hear the story.

How to Test an AI Agent's Tool Selection Without Trusting Its Own Logs

Anand Pawar — Tue, 21 Jul 2026 09:51:28 +0000

You have built an AI agent harness. It calls tools, routes requests, and returns results. Your team trusts its telemetry to tell you which tool was selected and why.

That trust is a liability.

An agent's own logs are self-reported. They tell you what the agent thinks it did, not what actually happened. A hallucinated tool name, a misrouted parameter, a silent fallback to a different function — none of these surface in the agent's own trace. You need an external witness.

Here is how to build one.

The Problem: Self-Reported Truth Is Not Truth

Most teams validate agent behavior by reading the agent's own output. They check the tool_calls field in the response, match it against an expected schema, and call it done.

This works until it doesn't.

Consider a common failure mode: the agent decides to call search_knowledge_base but the LLM formats the tool name as searchKnowledgeBase. The routing layer silently normalizes it, the call succeeds, and the agent logs search_knowledge_base. Your test passes. The actual execution path was different from what you verified.

Another pattern: the agent selects the correct tool but passes a parameter that the tool silently coerces. A date string gets parsed into a different timezone. A user ID gets truncated. The tool returns a result, the agent logs success, and your test never catches the drift.

The root cause is the same. You are testing the agent's intent, not its execution. Intent is cheap to fake. Execution leaves fingerprints.

The Solution: An External Observer

You need a layer that sits between the agent and the tools it calls. This observer records every invocation — tool name, parameters, response, latency — without the agent knowing it is being watched.

The observer does not trust the agent's logs. It trusts what it sees on the wire.

Here is the architecture at a high level:

Intercept every outbound call from the agent to a tool.
Record the raw request before any normalization or routing.
Compare the recorded call against your expected invocation.
Assert on the recorded data, not the agent's summary.

This is not middleware. It is a test harness that wraps the tool-calling layer.

Technical Detail: Building the Observer in Python

I will show you a minimal implementation using Python and a mock tool server. The same pattern works in TypeScript with Playwright's route interception or a custom fetch wrapper.

Start with a simple tool registry. Each tool has a name, a handler, and a schema.

from dataclasses import dataclass, field
from typing import Any, Callable, Dict
import json
import time

@dataclass
class Tool:
    name: str
    handler: Callable
    schema: Dict[str, Any]

class ToolRegistry:
    def __init__(self):
        self._tools: Dict[str, Tool] = {}
        self._invocations: list = []

    def register(self, tool: Tool):
        self._tools[tool.name] = tool

    def call(self, name: str, params: Dict[str, Any]) -> Any:
        # Record the raw invocation before any processing
        invocation = {
            "tool_name": name,
            "params": params,
            "timestamp": time.time(),
            "raw_name": name  # This is what the agent actually sent
        }

        tool = self._tools.get(name)
        if tool is None:
            invocation["error"] = f"Tool '{name}' not found"
            self._invocations.append(invocation)
            raise ValueError(f"Tool '{name}' not found")

        start = time.time()
        try:
            result = tool.handler(**params)
            invocation["result"] = result
            invocation["latency"] = time.time() - start
        except Exception as e:
            invocation["error"] = str(e)
            invocation["latency"] = time.time() - start
            raise
        finally:
            self._invocations.append(invocation)

        return result

    def get_invocations(self) -> list:
        return self._invocations

    def clear(self):
        self._invocations.clear()

The key detail: invocation["raw_name"] captures exactly what the agent sent. No normalization. No aliasing. If the agent sends searchKnowledgeBase, you record searchKnowledgeBase. Your test can then assert that the agent sent the canonical name, not a variant.

Now register a tool and simulate an agent call.

def search_kb(query: str, max_results: int = 5) -> list:
    # Simulated search
    return [{"id": 1, "title": f"Result for {query}"}]

registry = ToolRegistry()
registry.register(Tool(
    name="search_knowledge_base",
    handler=search_kb,
    schema={"query": "string", "max_results": "integer"}
))

# Simulate an agent call with a non-canonical name
try:
    registry.call("searchKnowledgeBase", {"query": "AI testing", "max_results": 3})
except ValueError:
    pass

invocations = registry.get_invocations()
print(invocations[0]["raw_name"])  # "searchKnowledgeBase"

Your test can now assert on raw_name directly.

def test_agent_uses_canonical_tool_name():
    registry.clear()
    # Simulate the agent's decision loop
    agent_decides_to_call("search_knowledge_base", {"query": "testing"})
    invocations = registry.get_invocations()
    assert len(invocations) == 1
    assert invocations[0]["raw_name"] == "search_knowledge_base"

This catches the normalization failure. If the agent sends a variant, the test fails.

Extending the Observer

The same pattern catches parameter drift. Record the raw parameters before the tool handler processes them. If the agent sends a string where an integer is expected, your test sees the raw string.

def test_agent_passes_correct_param_types():
    registry.clear()
    agent_decides_to_call("search_knowledge_base", {"query": "testing", "max_results": "3"})
    invocations = registry.get_invocations()
    params = invocations[0]["params"]
    assert isinstance(params["max_results"], int), "max_results should be int"

The agent's log might show max_results: 3 as an integer because the routing layer coerced it. Your observer shows the raw string. That difference matters when the tool's behavior depends on type.

What This Teaches

The principle is simple: test the boundary, not the summary.

An agent's internal logs are a summary of what it intended. The actual execution happens at the boundary between the agent and the tool. That boundary is where failures live. Normalization, coercion, fallback routing, silent retries — none of these appear in the agent's own trace.

By placing an observer at that boundary, you shift your testing from intent to execution. You stop asking "did the agent think it called the right tool?" and start asking "did the agent actually call the right tool with the right parameters?"

This is not a new idea. It is the same principle that makes contract testing valuable in microservices. You test the API contract, not the service's internal state. The agent is just another service with a particularly unreliable internal narrator.

A Note on Cost

An external observer adds latency and storage. Every invocation gets recorded, serialized, and stored for the duration of the test. In production, you might sample or aggregate. In test, you record everything.

The trade-off is worth it. A single undetected tool misrouting can cascade into hours of debugging. The observer pays for itself the first time it catches a failure the agent's logs missed.

Closing

Your team is probably testing the agent's intent right now. The logs look clean, the traces are green, and the demos work. But the real failures live in the gap between what the agent says it did and what actually happened.

Build an observer. Record the raw invocation. Assert on what you see, not what you are told.

Which of your agent's tool calls have you never actually witnessed?

The Test That Passes in Staging But Fails When a Customer Runs It

Anand Pawar — Mon, 20 Jul 2026 18:34:12 +0000

You have been here. The test suite is green. The deployment pipeline reports all checks passed. Then a customer opens a ticket with a screenshot that shows something your test never caught.

The test passed in staging. It fails in production. And you cannot reproduce it locally.

This is not a flaky test problem. It is a fidelity problem. Your test environment and your production environment are not the same thing. The gap between them is where real bugs live.

Let me walk through one concrete example, the fix, and what it teaches about writing tests that survive the handoff to a real user.

The Problem: Environment Drift

A fintech team I worked with had a checkout flow. The test clicked "Pay Now", waited for a success message, and asserted the text "Payment successful" appeared on screen. It passed every time in staging.

Customers reported that after paying, they saw a blank white page for several seconds before the success message appeared. Some of them closed the tab during that blank period, thinking the payment failed. The transaction went through. The customer never saw the confirmation. Support tickets piled up.

The test never caught this because the staging environment served the success page in under 200 milliseconds. The blank period did not exist there. Production had a slower downstream service that introduced a three-second delay between the payment confirmation and the page render.

The test was correct in what it checked. It was wrong in what it assumed about timing and state.

The Fix: Test the Experience, Not Just the Outcome

The fix was not to add a longer wait. The fix was to test what the user actually experiences during that gap.

Here is a minimal Playwright test in TypeScript that catches this class of problem:

import { test, expect } from '@playwright/test';

test('checkout shows loading state before success', async ({ page }) => {
  await page.goto('/checkout');
  await page.fill('#card-number', '4111111111111111');
  await page.fill('#expiry', '12/28');
  await page.fill('#cvc', '123');
  await page.click('button[type="submit"]');

  // The user sees a loading indicator before the success message
  // This assertion catches the blank-page gap
  await expect(page.locator('[data-testid="loading-spinner"]')).toBeVisible({ timeout: 5000 });

  // Then the success message appears
  await expect(page.locator('text=Payment successful')).toBeVisible({ timeout: 10000 });
});

The key line is the loading spinner assertion. It forces the test to observe the intermediate state. If the spinner never appears because the page goes blank, the test fails. That failure tells you something about the real user experience that a simple text assertion never would.

In Python with Playwright, the same logic looks like this:

from playwright.sync_api import sync_playwright

def test_checkout_loading_state():
    with sync_playwright() as p:
        browser = p.chromium.launch()
        page = browser.new_page()
        page.goto("/checkout")
        page.fill("#card-number", "4111111111111111")
        page.fill("#expiry", "12/28")
        page.fill("#cvc", "123")
        page.click('button[type="submit"]')

        # Assert the loading state appears
        spinner = page.locator('[data-testid="loading-spinner"]')
        assert spinner.is_visible(timeout=5000)

        # Assert the final success state
        success = page.locator("text=Payment successful")
        assert success.is_visible(timeout=10000)

        browser.close()

This is not about adding more assertions. It is about asserting the right things at the right moments.

Technical Detail: Why This Works

The blank page problem happens because the browser receives the HTTP response for the success page before the JavaScript that renders the loading state has finished executing. In staging, the response arrives so fast that the JavaScript finishes before the browser paints. In production, the response arrives, the browser paints a blank page, and then the JavaScript catches up.

A standard waitForSelector or toBeVisible on the success text will pass in staging because the text appears within the default timeout. In production, the same assertion might pass after a longer wait, but it never observes the blank period. The test reports green. The user reports a broken experience.

By asserting the loading state, you force the test to observe the transition. If the loading state never appears, you know the page went through an unexpected state. That is the signal you need.

What This Teaches

Three things.

First, your test environment is a lie. It is a controlled simulation that hides timing differences, network latency, and service degradation. The only way to catch environment-specific bugs is to write tests that assume the environment is hostile. Assert intermediate states. Assert loading indicators. Assert error boundaries. Do not just assert the happy path endpoint.

Second, a passing test is not proof of quality. It is proof that your test and your environment agree on what should happen. That agreement can be wrong. The customer is the only judge that matters.

Third, the most valuable tests are the ones that fail in staging. A test that fails in staging tells you something about your system before it reaches a customer. A test that passes in staging and fails in production tells you that your test was not testing the right thing.

The Call to Action

Look at your test suite right now. Find one test that only asserts a final state. A success message. A redirect. A database row.

Add one assertion for an intermediate state. A loading spinner. A disabled button. A progress bar. Run it in staging. Then ask yourself: if this test ran in production, would it still tell the truth?

The gap between staging and production is not going away. But your tests can learn to see it.

What is one test in your suite that you suspect is lying to you?

Is Your BDD Framework Just a Fancy Way to Write Manual Test Cases in Gherkin?

Anand Pawar — Mon, 20 Jul 2026 18:33:58 +0000

Gherkin is not a test automation tool. It never was.

Yet here we are, five years into your SDET career, and you're staring at a feature file that reads like a step-by-step manual for a human tester. Given I log in with username "admin" and password "password123". When I click the "Submit" button. Then I see the text "Welcome" on the screen.

You've written two years of these. Your team calls it BDD. Your manager calls it "living documentation." And somewhere in the back of your mind, a quiet voice whispers: This is just a manual test case with extra steps.

That voice is right.

Let me say it plainly: if your Gherkin scenarios describe how the system works instead of what it should do, you are not doing BDD. You are writing manual test cases in a structured English format and calling it automation. The only thing you've automated is the illusion of progress.

The problem isn't Gherkin. The problem is how we use it.

Most teams adopt BDD because someone read a blog post about "collaboration" and "shared understanding." They install Cucumber or SpecFlow. They write feature files. They map steps to Selenium or Playwright code. And they call it a day.

But look closely at what happens next. The product owner never reads the feature files. The developer skims them once and goes back to writing code. The QA engineer — that's you — becomes the sole maintainer of a growing pile of Gherkin that nobody else touches.

You're not facilitating collaboration. You're translating manual test cases into a format that requires a compiler.

Here's the real test. Take any feature file from your project. Hand it to a developer who has never seen it. Ask them to implement the feature using only the Gherkin as a spec. If they can write production code from it, you have real BDD. If they ask you for clarification, you have documentation theater.

I've seen teams with hundreds of feature files. Beautifully formatted. Perfect indentation. Tags for every regression cycle. And not a single one of them could survive that test.

So what went wrong?

The mistake is subtle but fatal. We confused structure with abstraction. Gherkin gives us a structure: Given-When-Then. But structure alone doesn't create abstraction. Abstraction means hiding implementation details behind intent.

A good Gherkin scenario says: Given a user with an expired subscription. It does not say: Given I navigate to the login page, enter "test@example.com" in the email field, enter "password123" in the password field, click "Sign In", navigate to the billing page, and check that the subscription status is "expired".

The first version captures intent. The second version captures keystrokes.

When you write the second version, you've created a test that breaks every time the UI changes. You've also created a document that nobody can read without knowing the exact state of every page element. That's not living documentation. That's dead weight.

Here's the uncomfortable truth for you, the five-year SDET who suspects something is off.

You're probably the most technical person on your QA team. You can write Playwright selectors in your sleep. You understand async/await. You know when to use page.waitForSelector versus page.waitForLoadState. And yet, you're spending your days writing Gherkin that a junior manual tester could have written.

The reason is organizational inertia. Someone decided "we do BDD" three years ago. The framework is built. The CI pipeline expects feature files. Changing it would require admitting that the emperor has no clothes.

But here's what I want you to consider: you can fix this without throwing away the framework.

The solution is not to abandon Gherkin. The solution is to push the abstraction down one layer. Write your step definitions so thin they're almost empty. Move the real logic into page objects or service layers. Let your Gherkin describe what, and let your code describe how.

Here's what that looks like in practice. I'll use Playwright with TypeScript because that's what most teams are using today.

The wrong way:

// step-definitions/login.steps.ts
import { Given, When, Then } from '@cucumber/cucumber';
import { page } from '../world';

Given('I log in with username {string} and password {string}', async (username: string, password: string) => {
  await page.goto('https://example.com/login');
  await page.fill('#email', username);
  await page.fill('#password', password);
  await page.click('button[type="submit"]');
  await page.waitForURL('**/dashboard');
});

When('I click the "Submit" button', async () => {
  await page.click('button:has-text("Submit")');
});

Then('I see the text {string} on the screen', async (text: string) => {
  await page.waitForSelector(`text=${text}`);
});

This is manual testing with a compiler. Every step is a UI interaction. Every scenario is a script. If the login page changes its URL or the button text changes, every feature file breaks.

The right way:

// step-definitions/login.steps.ts
import { Given, When, Then } from '@cucumber/cucumber';
import { loginPage } from '../pages/login-page';
import { dashboardPage } from '../pages/dashboard-page';

Given('I am logged in as a standard user', async () => {
  await loginPage.loginAs('standard_user');
});

When('I submit the form', async () => {
  await loginPage.submit();
});

Then('I should see the dashboard', async () => {
  await dashboardPage.isVisible();
});

// pages/login-page.ts
import { Page } from '@playwright/test';

export class LoginPage {
  constructor(private page: Page) {}

  async loginAs(userType: string) {
    const credentials = this.getCredentialsFor(userType);
    await this.page.goto('/login');
    await this.page.getByLabel('Email').fill(credentials.email);
    await this.page.getByLabel('Password').fill(credentials.password);
  }

  async submit() {
    await this.page.getByRole('button', { name: 'Sign In' }).click();
    await this.page.waitForURL('/dashboard');
  }

  private getCredentialsFor(userType: string) {
    // Returns credentials from a fixture or config
    return { email: `${userType}@example.com`, password: 'test123' };
  }
}

Notice the difference. The Gherkin now says what happens, not how. The step definition is three lines. The real logic lives in the page object, where it belongs.

When the login page changes, you update one page object. Not twenty feature files. When a new team member reads the feature file, they understand the business flow without knowing the DOM structure. When the product owner reviews the scenarios, they can actually follow along.

This is what BDD was supposed to be.

But here's the part nobody tells you. Even with perfect abstraction, BDD has a cost. You're maintaining three layers of code: feature files, step definitions, and page objects. Each layer is a point of failure. Each layer requires synchronization.

For a team of five, that overhead might be worth it. For a team of two, it's probably not. For a solo QA engineer maintaining a legacy suite, it's a trap.

So what should you do?

Start by auditing your feature files. Pick five at random. Count how many steps describe UI interactions versus business intent. If more than half are UI interactions, you have a problem.

Then, refactor one scenario. Move the implementation details into a page object. Rewrite the Gherkin to describe intent. Run the tests. See if they still pass. See if they're easier to read.

If they are, you have your answer. The framework wasn't the problem. The abstraction was.

What this teaches us.

BDD is a communication tool, not a test automation tool. When we treat it like automation, we get the worst of both worlds: verbose tests that nobody reads and brittle scenarios that break constantly.

The teams that succeed with BDD are the ones who treat the feature file as a contract, not a script. They write scenarios that a non-technical stakeholder could review. They keep step definitions thin. They invest in page objects and service layers.

And they're honest about the tradeoff. BDD adds structure, but it also adds ceremony. For some projects, that ceremony is worth it. For others, a well-structured Playwright test with descriptive test names and good assertions is more valuable than a hundred feature files.

Here's my challenge to you.

Take one feature file from your current project. Rewrite it so that a product manager could read it and nod along. Then rewrite the step definitions so that a developer could change the UI without touching the Gherkin.

If you can do that, you're not writing manual test cases in Gherkin anymore. You're doing BDD.

If you can't, you know what you're really maintaining.

And that's okay. The first step is admitting you have a problem. The second step is deciding what to do about it.

What's one scenario in your current suite that you know is just a manual test in disguise?

Your First Week of AI-Assisted Automation Will Be a Debugging Nightmare

Anand Pawar — Mon, 20 Jul 2026 18:33:35 +0000

Most engineers expect AI-assisted automation to be the easy part. You describe a test, the model writes it, you move on.

The first week will prove you wrong.

Not because the code is bad. Because the code is almost right. And almost-right code is harder to debug than wrong code. Wrong code fails loudly. Almost-right code passes on Monday, fails on Tuesday, passes again on Wednesday, and by Thursday you are questioning whether you understand your own application.

I have watched teams adopt AI copilots into their Playwright suites and spend the first five days doing nothing but untangling false passes. If you are about to start this journey, here is what that week actually looks like.

The Problem: The Model Does Not Know What "Stable" Means

A language model has never waited for a network response. It has never watched a flaky selector survive three CI runs and then collapse on the fourth. It writes tests from a static understanding of your page, not from the dynamic reality of your application.

You will ask it to write a test that clicks a button and waits for a confirmation toast. The model will produce something like this:

await page.click('button:has-text("Submit")');
await page.waitForSelector('.toast-success');

Looks fine. Runs fine. Then your team deploys a new build where the toast takes 400ms longer to appear because of an analytics call. The test fails. Not because the feature broke. Because the model assumed a timing that was never guaranteed.

This is the core problem. The model writes tests that match the page as it was when the model saw it. It does not write tests that match the page as it will be.

The Solution: Treat AI-Generated Tests as Drafts, Not Deliverables

The shift is mental before it is technical. You cannot review AI-generated tests the way you review human-written tests. Human tests come with intent. AI tests come with patterns.

You need a different review lens.

First, look for every hardcoded wait. Replace it with a state-based assertion. Second, look for every selector that relies on text content that could change. Third, look for every assumption about element order on the page.

The model will write page.locator('button').first() because it saw one button. Your page has three buttons. The test will click the wrong one.

The solution is not to stop using the model. The solution is to treat every line it writes as a first draft that needs a human edit.

Technical Detail: The Three Failure Modes You Will Hit

I have seen three patterns repeat across teams. If you know them in advance, you can catch them in review instead of in CI.

1. The Optimistic Selector

The model assumes elements are uniquely identifiable by text or role. In practice, your page has multiple elements with the same label.

// AI-generated
await page.getByRole('button', { name: 'Save' }).click();

This works until a second "Save" button appears in a modal. Now the test clicks the wrong one. The fix is to scope the selector to a specific container:

// Human-edited
const dialog = page.getByRole('dialog', { name: 'Confirm changes' });
await dialog.getByRole('button', { name: 'Save' }).click();

2. The Missing State Check

The model writes actions but skips the verification that the action actually completed.

// AI-generated
await page.fill('#email', 'test@example.com');
await page.fill('#password', 'password123');
await page.click('button[type="submit"]');

No assertion that the form submitted. No check that the next page loaded. The test passes even if the submit button does nothing. You only discover the problem when a real regression slips through.

3. The Brittle Wait Strategy

The model defaults to fixed timeouts or waits for elements that may not exist in every state.

// AI-generated
await page.waitForTimeout(2000);

This is the most common pattern I remove. It passes locally, fails in CI, and wastes hours of debugging time. Replace every waitForTimeout with a waitForSelector, waitForURL, or waitForResponse.

What This Teaches: The Model Is a Junior Engineer

Here is the honest take. A good AI copilot writes tests at the level of a junior engineer who has read your documentation but never used your product. The code is syntactically correct. The logic is structurally sound. But the judgment is missing.

The junior engineer does not know that this particular button only appears after a network call. The junior engineer does not know that this toast sometimes takes three seconds. The junior engineer writes tests that work in the happy path and break everywhere else.

Your job is not to accept the code. Your job is to mentor it.

Every edit you make to an AI-generated test is teaching the model, indirectly, what matters in your application. Over time, the drafts get better. But the first week will be brutal because you are starting from zero shared context.

The CTA: Start With One Test, Not a Suite

If you are about to integrate an AI copilot into your Playwright suite, do not generate fifty tests on day one. Generate one. Debug it. Fix it. Run it ten times. Then generate the next one.

The temptation is speed. The reality is that speed without stability is just faster failure.

Pick one critical user flow. Ask the model to write it. Then spend the time to make it production-ready. Measure how long that took. Multiply by the number of flows you need. That is your real timeline.

The model will save you time on boilerplate. It will not save you time on debugging. That part is still yours.

What was the first AI-generated test you had to completely rewrite? I would like to hear which failure mode hit you first.

3 Tests That Pass in LangFlow But Fail in n8n Production

Anand Pawar — Mon, 22 Jun 2026 18:50:41 +0000

You built a LangFlow prototype. Every test passed. You exported the flow, dropped it into n8n, and the first production run broke.

This is not a bug report. It is a pattern.

The three-year SDET who has built LangFlow prototypes but hit mysterious failures when deploying the same logic in n8n production already knows the feeling. The prototype felt solid. The production pipeline felt like a different language.

It is not. The difference is execution context. LangFlow runs in a notebook-like environment where state is forgiving and retries are invisible. n8n runs in a workflow engine where every node is a transaction boundary and every failure is final unless you explicitly handle it.

Here are the three tests that pass in LangFlow but fail in n8n production, and what they teach about building reliable AI pipelines.

Test 1: The "LLM Returns Valid JSON" Test

What passes in LangFlow: You send a prompt asking the model to return JSON. The response comes back as a string. You parse it with json.loads(). It works. You move on.

What fails in n8n: The model returns a string that starts with a code block. Or a trailing comma. Or a markdown fence. Or a preamble sentence before the JSON. Or nothing at all because the context window was exceeded.

Why the difference: LangFlow's Python node silently tolerates malformed output. If json.loads() fails, you see the error in the output panel and fix the prompt. n8n's JSON node does not retry. It does not fall back. It throws a structured error that stops the entire workflow.

The fix is not a better prompt. The fix is a validation layer that normalizes LLM output before parsing.

import json
import re

def extract_json(raw: str) -> dict:
    # Strip markdown fences
    cleaned = re.sub(r'^```

(?:json)?\s*', '', raw.strip())
    cleaned = re.sub(r'\s*

```$', '', cleaned)

    # Find the first { and last }
    start = cleaned.find('{')
    end = cleaned.rfind('}')

    if start == -1 or end == -1:
        raise ValueError("No JSON object found in response")

    candidate = cleaned[start:end+1]

    # Attempt parse
    try:
        return json.loads(candidate)
    except json.JSONDecodeError:
        # Try fixing trailing commas
        candidate = re.sub(r',\s*}', '}', candidate)
        return json.loads(candidate)

This function lives in a shared utility node in n8n. Every LLM call routes through it. It catches the edge cases that LangFlow silently absorbed.

What this teaches: Prototype environments hide fragility. The test that passes in LangFlow is not testing the output format. It is testing whether the model usually returns something parseable. Production requires a contract, not a hope.

Test 2: The "Context Window Fits" Test

What passes in LangFlow: You feed a document into a prompt. The document is 8,000 tokens. The model handles it. You test again with 12,000 tokens. Still fine. You declare the pipeline ready.

What fails in n8n: The production document is 18,000 tokens. The model truncates silently. Or the context window fills with system prompts and conversation history, leaving no room for the actual input. The output becomes generic. The test that passed now produces garbage.

Why the difference: LangFlow runs each prompt in isolation. You control exactly what goes in. n8n workflows accumulate state. A node that prepends conversation history, a sub-workflow that adds metadata, a loop that concatenates previous outputs — each one eats context. The prototype never tested this because the prototype never ran the full chain.

The fix is not a larger model. The fix is a context budget that is measured and enforced.

// n8n node: Context Budget Checker
// Place this before any LLM call

interface ContextBudget {
  maxTokens: number;
  reservedForOutput: number;
  systemPromptTokens: number;
  historyTokens: number;
}

function checkBudget(input: string, budget: ContextBudget): boolean {
  const available = budget.maxTokens - budget.reservedForOutput;
  const used = budget.systemPromptTokens + budget.historyTokens + estimateTokens(input);

  if (used > available) {
    throw new Error(
      `Context budget exceeded: ${used} tokens needed, ${available} available. ` +
      `Truncate input or increase model capacity.`
    );
  }

  return true;
}

function estimateTokens(text: string): number {
  // Rough estimate: 4 characters per token
  return Math.ceil(text.length / 4);
}

This node fails fast. It does not let the LLM run with an overstuffed context. The error message tells the operator exactly what to fix.

What this teaches: A test that passes in isolation is not a test of the system. The system includes every node that touches the context. LangFlow tests the prompt. n8n tests the pipeline. They are different things.

Test 3: The "Retry Is Free" Test

What passes in LangFlow: The LLM call fails. You click "Run" again. It works. You assume the transient error was a network blip. You do not write a retry handler.

What fails in n8n: The LLM call fails at 2 AM. The workflow stops. No one notices until morning. The data that should have been processed is stuck in an error queue. The test that passed in LangFlow never ran at 2 AM.

Why the difference: LangFlow is a development environment. You are the retry handler. You see the error, you decide what to do, you click the button. n8n is a production environment. It runs unattended. If you did not tell it what to do on failure, it does nothing.

The fix is not a simple retry. The fix is a retry with exponential backoff and a dead-letter queue.

// n8n sub-workflow: LLM Call with Retry
// Wraps the LLM node

const MAX_RETRIES = 3;
const BASE_DELAY_MS = 1000;

async function callWithRetry(prompt: string, attempt: number = 1): Promise<string> {
  try {
    // This is the actual LLM call
    return await $node["LLM"].call(prompt);
  } catch (error) {
    if (attempt >= MAX_RETRIES) {
      // Send to dead-letter queue
      await $node["Dead Letter Queue"].send({
        prompt,
        error: error.message,
        attempt
      });
      throw new Error(`LLM call failed after ${MAX_RETRIES} attempts`);
    }

    const delay = BASE_DELAY_MS * Math.pow(2, attempt - 1);
    await new Promise(resolve => setTimeout(resolve, delay));
    return callWithRetry(prompt, attempt + 1);
  }
}

The dead-letter queue is critical. It preserves the failed input so you can replay it after fixing the issue. Without it, the error is a black hole.

What this teaches: Prototypes do not have failure modes. Production does. The test that passes in LangFlow is testing the happy path. The test that matters in n8n is testing the unhappy path. If you only test the happy path, you are not testing.

What These Three Tests Teach

Each of these failures shares a root cause: the prototype environment and the production environment have different assumptions about reliability.

LangFlow assumes you are watching. It assumes you will handle edge cases manually. It assumes the input is clean and the model is cooperative.

n8n assumes you are not watching. It assumes every edge case must be handled explicitly. It assumes the input is dirty and the model is unreliable.

The three-year SDET who built LangFlow prototypes and hit mysterious failures in n8n production is not making a mistake. They are learning the difference between a prototype and a production system. That difference is not the tool. It is the contract.

LangFlow says: "I will show you what the model returns."

n8n says: "I will do exactly what you told me, every time, without asking."

The second one is harder. It is also the one that runs at 2 AM.

What to Do Next

If you are moving a LangFlow prototype to n8n, do not export the flow and hope. Do this instead:

Add a validation layer for every LLM output. Normalize before parsing.
Measure context usage before every LLM call. Fail fast, not silently.
Implement retry with backoff and a dead-letter queue. Assume every call will fail at least once.

These three changes will catch the tests that pass in LangFlow but fail in n8n. They will also teach you something about your pipeline that the prototype never revealed.

The prototype is a sketch. Production is the building. Do not confuse the two.

Which of your LangFlow tests have you never run at 2 AM?

3 Things Nobody Tells You About BDD Before Your Cucumber Suite Becomes a Maintenance Nightmare

Anand Pawar — Sat, 20 Jun 2026 21:17:07 +0000

You have six years of automation experience and a Cucumber suite that takes forty minutes to run, and you still can't tell me what a single feature file actually tests without reading three layers of glue code.

I've been there. You're the senior automation engineer on a team that adopted BDD because "business stakeholders need to read the tests." Except those stakeholders stopped reading them after sprint two. Now you're maintaining a suite where every scenario is a small novel, every step definition calls three other step definitions, and the person who wrote the original framework left the company eighteen months ago.

Here's what nobody tells you about BDD before your Cucumber suite becomes a maintenance nightmare.

Thing One: Gherkin Is Not a Programming Language, But You're Treating It Like One

Your feature files look like this:

Scenario: User completes checkout with valid payment
  Given the user is logged in with email "test@example.com" and password "Password123!"
  And the user has 3 items in their cart
  And the user's shipping address is "123 Main St, Springfield, IL 62701"
  And the user's billing address matches their shipping address
  When the user selects "Credit Card" as payment method
  And the user enters card number "4111111111111111"
  And the user enters expiry date "12/28"
  And the user enters CVV "123"
  And the user clicks "Place Order"
  Then the order confirmation page is displayed
  And the order number is shown
  And the user receives an email confirmation

This is not BDD. This is a test script written in a markup language that happens to parse into Cucumber.

The original promise of BDD was that scenarios describe behavior, not implementation. But somewhere along the way, teams started treating Gherkin as a readable scripting language. Every parameter gets inlined. Every step becomes a concrete action. The feature file becomes a play-by-play of the UI interaction.

The result? When the checkout flow changes — and it will — you have to update every scenario that touches checkout. Not the step definitions. The feature files themselves. Because the business logic is embedded in the Gherkin, not abstracted behind it.

Here's what a better version looks like:

Scenario: User completes checkout with valid payment
  Given the user has items ready to purchase
  When the user pays with a valid credit card
  Then the order is confirmed

That's it. Three lines. The "how" lives in the step definition. The "what" lives in the feature file. When the payment provider changes, you update one step definition, not twenty scenarios.

Your step definition file should be the place where complexity lives. Your feature files should be so simple that a product manager could read them in a standup and nod along. If your Gherkin looks like a test script, you've already lost.

Thing Two: Your Step Definitions Are a Dependency Graph, Not a Library

Here's the pattern I see in every struggling Cucumber suite:

// step_definitions/checkout_steps.ts
import { Given, When, Then } from '@cucumber/cucumber';
import { loginAsUser } from './auth_steps';
import { addItemsToCart } from './cart_steps';
import { fillShippingAddress } from './shipping_steps';

Given('the user has items ready to purchase', async function () {
  await loginAsUser.call(this, 'test@example.com', 'Password123!');
  await addItemsToCart.call(this, 3);
  await fillShippingAddress.call(this, '123 Main St');
});

This is a dependency graph masquerading as a test framework. Every step definition imports other step definitions. The call order matters. The shared this context accumulates state across steps. If one step fails, the entire scenario is poisoned, and you can't tell whether the failure is in the step that failed or in the accumulated state from the three steps before it.

The counterargument: "But we need to reuse steps across scenarios. That's the whole point of BDD."

I agree with reuse. I disagree with how you're doing it.

The fix is to separate your page objects or service clients from your step definitions entirely. Your step definitions should be thin wrappers that call a shared domain layer. Not other step definitions.

// domain/checkout.ts
export class CheckoutFlow {
  constructor(private page: Page) {}

  async prepareItemsForPurchase(): Promise<void> {
    await this.page.goto('/login');
    await this.page.fill('#email', 'test@example.com');
    await this.page.fill('#password', 'Password123!');
    await this.page.click('#login-button');
    await this.page.waitForURL('/dashboard');
    // ... add items, set address
  }

  async payWithCreditCard(): Promise<void> {
    await this.page.click('#credit-card-option');
    await this.page.fill('#card-number', '4111111111111111');
    await this.page.click('#place-order');
  }
}

// step_definitions/checkout_steps.ts
import { CheckoutFlow } from '../domain/checkout';

Given('the user has items ready to purchase', async function () {
  this.checkout = new CheckoutFlow(this.page);
  await this.checkout.prepareItemsForPurchase();
});

When('the user pays with a valid credit card', async function () {
  await this.checkout.payWithCreditCard();
});

Now your step definitions are stateless. They don't import each other. They don't share mutable context. Each step creates or calls a domain object that encapsulates the behavior. If a step fails, the failure is isolated to that step's domain operation.

Your step definition file becomes a routing table, not a tangled web of imports. You can delete a step definition without worrying about breaking three other scenarios that depend on it.

Thing Three: Your Cucumber Suite Is a Documentation Project That You're Treating as a Test Project

This is the one nobody wants to say out loud.

BDD was never primarily about testing. It was about communication. The original vision was that scenarios would serve as living documentation — a shared language between developers, testers, and business stakeholders. The tests were a side effect.

But somewhere in the last decade, teams started measuring BDD success by test coverage. "We have 500 Cucumber scenarios." "Our BDD suite runs in CI." "We achieved 90% Gherkin coverage."

None of those metrics tell you whether your scenarios are readable. None of them tell you whether a new team member can open a feature file and understand what the system does without running the tests.

Here's the uncomfortable truth: if your Cucumber suite is a maintenance nightmare, it's because you optimized for the wrong thing. You optimized for test execution speed, for step reuse, for parameterization. You forgot to optimize for reading.

I learned this the hard way. I had a suite with beautiful step definitions. Clean abstractions. Zero duplication. But when a product manager asked me what the checkout flow looked like, I had to open three feature files and trace through five scenario outlines to explain it. The documentation was technically there, but it was useless.

The fix is brutal but simple: every time you add a scenario, ask yourself whether someone who has never seen your codebase can understand it in under thirty seconds. If the answer is no, rewrite the scenario. Not the step definition. The scenario.

Your feature files should be the first thing a new engineer reads when they join your team. Not the README. Not the wiki. The feature files. If they can't understand the system from those files alone, your BDD suite has failed its primary purpose.

What to Do Tomorrow

Pick one feature file. The worst one. The one with the most steps, the most parameters, the most inline data.

Rewrite it to three to five lines per scenario. Move every concrete value into a step definition or a test data factory. Delete every step definition that imports another step definition. Replace them with domain objects.

Run the suite. If it passes, you've just made your documentation better and your tests more maintainable in one move.

Then do it again for the next file.

A Question for You

Your Cucumber suite runs in CI. It passes. Your team is shipping. But if you had to explain the system's behavior to a new hire using only your feature files, could you do it in under five minutes?

If the answer is no, you know where to start.

The Tester Who Had 10 Certifications But Couldn't Write a Single Test That Caught a Bug

Anand Pawar — Sat, 20 Jun 2026 09:29:18 +0000

You have ISTQB Foundation. ISTQB Advanced. Certified ScrumMaster. A cloud cert. A security testing cert. Maybe a Python for Testers badge from a platform.

And you still cannot write a test that finds a real bug.

I interviewed someone like you last quarter. The resume was a wall of acronyms. The conversation was a wall of theory. "I follow the V-model." "I use equivalence partitioning." "I believe in shift-left."

Then I asked: "Show me one test you wrote that caught something the developer missed."

Silence.

Not because they were nervous. Because they had never written a test that found a bug. They had written tests that passed. They had written tests that covered requirements. They had never written a test that broke something.

That is the difference between a certification holder and a tester.

Certifications test your memory. Bugs test your thinking.

Let me show you what I mean.

The Certification Trap

Certifications are not useless. They give you vocabulary. They give you structure. They give you something to put on LinkedIn so recruiters stop asking if you know what a test case is.

But they do not teach you how to find bugs.

Here is why. Every certification exam tests known knowledge. You study a syllabus. You memorize definitions. You answer multiple-choice questions about boundary value analysis. You pass.

Then you sit in front of an application. The application does not have a syllabus. It does not have a boundary value analysis section in the documentation. It has a login form that sometimes lets you in with a password that is clearly wrong, but only on Tuesdays, and only if the server clock is behind by exactly four minutes.

No certification prepares you for that.

The tester with 10 certifications treats testing like a checklist. They write test cases from requirements. They execute them. They mark pass or fail. They report coverage metrics.

The tester who finds bugs treats testing like an investigation. They start with a hypothesis. They try to prove the application wrong. They follow the weirdness.

What a Real Bug-Finding Test Looks Like

Let me show you the difference in code.

Here is a typical test written by someone who studied for certifications but never learned to hunt bugs:

import { test, expect } from '@playwright/test';

test('user can complete checkout with valid credit card', async ({ page }) => {
  await page.goto('/products');
  await page.click('[data-test="add-to-cart"]');
  await page.click('[data-test="checkout"]');
  await page.fill('[data-test="card-number"]', '4111111111111111');
  await page.fill('[data-test="expiry"]', '12/28');
  await page.fill('[data-test="cvc"]', '123');
  await page.click('[data-test="submit-payment"]');
  await expect(page.locator('[data-test="order-confirmation"]')).toBeVisible();
});

This test passes. It covers the happy path. It proves the feature works when everything is perfect.

It will never find a bug.

Now here is a test written by someone who thinks like a bug hunter:

import { test, expect } from '@playwright/test';

test('checkout rejects card number with typo in first digit', async ({ page }) => {
  await page.goto('/products');
  await page.click('[data-test="add-to-cart"]');
  await page.click('[data-test="checkout"]');

  // Card number with first digit changed from 4 to 5
  await page.fill('[data-test="card-number"]', '5111111111111111');
  await page.fill('[data-test="expiry"]', '12/28');
  await page.fill('[data-test="cvc"]', '123');
  await page.click('[data-test="submit-payment"]');

  // The application should reject this card
  // But does it show an error, or does it silently fail?
  await expect(page.locator('[data-test="error-message"]')).toBeVisible();

  // Also check: did the order confirmation accidentally appear anyway?
  const confirmationCount = await page.locator('[data-test="order-confirmation"]').count();
  expect(confirmationCount).toBe(0);
});

The second test is not more complex. It is more suspicious. It assumes the application will fail. It checks for the error. It also checks that the success state did not accidentally appear.

This is the mindset. Not "does it work?" but "where does it break?"

The Real Gap: You Have Never Failed a Test

Here is the honest truth for the reader I am speaking to directly. You, the tester with three years of experience and a stack of certs. You have never written a test that failed for a reason you did not expect.

You have seen tests fail because the environment was down. You have seen tests fail because the data changed. You have seen tests fail because someone updated the locator.

But you have never written a test that failed because you found a real, previously unknown defect in the application logic.

That is your career gap. Not a gap in your resume. A gap in your experience.

And no certification can fill it.

How to Close the Gap

You close it by writing tests that are designed to fail.

Not flaky tests. Not tests that fail because of timing. Tests that fail because the application has a bug.

Here is the exercise I give every tester I mentor. Pick one feature in your current application. Spend one hour writing tests that try to break it. Not tests that verify it works. Tests that verify it breaks correctly.

For a search feature, do not test that "search returns results for a valid query." Test that "search returns zero results for gibberish" and "search handles SQL injection characters without crashing" and "search with an empty string shows a helpful message instead of a 500 error."

For a file upload feature, do not test that "user can upload a valid PDF." Test that "uploading a file named ../../../etc/passwd does not overwrite system files" and "uploading a 2GB file shows a size error before the upload starts" and "uploading a file with no extension does not crash the preview generator."

Write these tests. Run them. When they fail, you have found a bug. When they pass, you have proven the application handles edge cases correctly. Either way, you learn something.

The Code That Changed My Mind

I was wrong about something for years. I thought good test automation meant covering every requirement with a passing test. I measured success by green builds.

Then I worked on a payment system for a fintech team. We had 95% test coverage. Every test passed. The system went to production and lost money for three days before anyone noticed.

The bug was in a rounding calculation. When the total was something like $10.005, the system rounded down for the customer but rounded up for the merchant. The difference was pennies per transaction. Over thousands of transactions, it was real money.

Our tests covered the happy path. They covered the sad path. They did not cover the "what if the decimal math is wrong" path because nobody thought to write that test.

After that, I changed how I write tests. Now every test starts with a question: "What would have to be true for this feature to fail silently?"

Here is a Python example of that mindset:

import pytest
from decimal import Decimal

def test_rounding_does_not_favor_any_party():
    """Verify that rounding is consistent between customer and merchant."""
    amounts = [
        Decimal("10.005"),
        Decimal("19.995"),
        Decimal("100.005"),
        Decimal("0.015"),
    ]

    for amount in amounts:
        customer_charge = round_to_cents(amount, party="customer")
        merchant_payout = round_to_cents(amount, party="merchant")

        # The total should be the same regardless of who we round for
        assert customer_charge == merchant_payout, (
            f"Rounding mismatch for {amount}: "
            f"customer pays {customer_charge}, merchant gets {merchant_payout}"
        )

This test found nothing in the first version. The developers had written correct rounding. But when we added a new currency with different decimal places, this test caught the bug in the first hour. It was already written. It was already running. It failed immediately.

That is the value of tests that are designed to find bugs. They sit there, waiting for the code to change, and they catch the regression before it reaches production.

What to Do This Week

You have the certs. You have the years. You do not have the proof.

Here is one action. Do not build a portfolio site. Do not update your LinkedIn. Do not study for another certification.

Write one test that is designed to fail. Not a test that verifies something works. A test that tries to prove something is broken.

Run it. If it passes, you have documented a safety guarantee. If it fails, you have found a bug. Either way, you have something real to show.

Write it up. Three paragraphs. What you tested. How you tested it. What you found. That is your portfolio. That is proof that you can think.

The next time someone asks you to show them a test that caught a bug, you will have an answer.

Not a certification. An answer.

What is the one test you would write this week to prove you can find bugs?