DEV Community: Tudor Brad

Fuzz testing found bugs in our API that unit tests never would

Tudor Brad — Thu, 09 Apr 2026 08:34:50 +0000

I used to think our test suites were solid. We had unit tests, integration tests, contract tests for the API layer. Good coverage numbers. The kind of setup that makes you feel safe when you merge to main on a Friday afternoon.

Then we ran a fuzzer against the same API and watched it fall apart in under an hour.

Fourteen crashes. Server panics on malformed JSON. A file upload endpoint that accepted literally anything as long as you set the right Content-Type header. An input field on a form that crashed the entire backend process when it received a float instead of an integer.

None of these showed up in our existing tests. Not one.

That was the day I stopped treating fuzzing as a "nice to have" and started treating it as the part of security testing that actually finds the bugs hiding between your test cases.

What fuzzing actually does

Fuzzing is simple in concept. You throw garbage at your software and see what breaks.

More precisely: you take valid inputs, mutate them in thousands of ways (wrong types, oversized strings, null bytes, nested objects 500 levels deep, unicode edge cases, truncated payloads), and send them at your application as fast as you can. Then you watch for crashes, hangs, memory leaks, unexpected error codes, and data that leaks out in error messages.

The OWASP fuzzing page describes the technique well if you want the textbook version. But here is what it looks like in practice: you point a tool at an endpoint, go make coffee, and come back to a list of inputs that made your software do something it should not have done.

The reason this works so well is that developers test for what they expect. You write a test that sends valid JSON and checks the response. Maybe you write a test that sends empty JSON and checks for a 400 error. But you probably do not write a test that sends JSON with a key that is 50,000 characters long, or a nested array 200 levels deep, or a number where a string should be with a trailing null byte.

Fuzzers do not have expectations. They just try things. And software has a lot of assumptions baked into it that only surface when those assumptions get violated.

The bugs fuzzing catches that nothing else does

Let me walk through the actual categories of failures we find during fuzz testing engagements. These are real patterns from real projects.

Input type confusion. A registration form expects a string for the phone number field. The API handler parses it and passes it to a validation function that calls .match() on it. Send an integer instead of a string and the backend throws an unhandled TypeError. The server returns a 500 with a stack trace that includes the file path and line number. Now an attacker knows your framework, your file structure, and exactly where to probe next.

Unit tests rarely cover this because the developer wrote the test with the same mental model they used to write the code. They send a string because that is what the field is for.

Malformed JSON handling. We see this constantly. APIs that parse JSON request bodies without validating the structure first. Send {"user": {"name": {"name": {"name": ...}}}} nested 100 times and the server either runs out of memory or hits a recursion limit and crashes. Send JSON with a trailing comma (technically invalid) and some parsers accept it while others throw. Send a 10MB payload to an endpoint that expects 200 bytes and there is no size limit enforced.

These are not exotic attacks. They are basic robustness issues that every public-facing API should handle. Fuzzers find them in minutes.

File upload validation gaps. This one is a classic. An endpoint says it accepts PNG files. It checks the Content-Type header. It does not check the actual file content. So you can upload a PHP script, a shell script, or an SVG containing embedded JavaScript, and the server happily stores it. Depending on the server configuration, that file might be directly executable.

We tested a client's document upload feature and found that it validated the file extension in the filename but not the actual bytes. Rename malicious.php to malicious.php.png and it went straight through.

Error message information leakage. When software crashes on unexpected input, the error messages often contain information that should never reach the client. Database connection strings, internal IP addresses, full stack traces with dependency versions, SQL query fragments. Fuzzers trigger these crashes systematically, and each crash response becomes a reconnaissance opportunity for an attacker.

Integer overflows and boundary values. We worked on a payment processing system where fuzz testing found an integer overflow in the transaction amount field. The field was a 32-bit signed integer. Send a value just past 2,147,483,647 and the system wrapped around to a negative number. In a payment context, that could mean a credit instead of a debit. Standard tests sent amounts like 100, 500, 10000. Nobody tested what happens at the boundary of the data type itself.

Why your existing tests miss these

Your unit tests are written by the same people who wrote the code. They share the same assumptions about what valid input looks like. They test the happy path and a handful of known error cases.

Your integration tests verify that components work together correctly when given correct data. They rarely test what happens when component A sends garbage to component B.

Your end-to-end tests simulate real user behavior. Real users do not typically paste 50,000 characters into a phone number field or send raw bytes to a JSON endpoint. Attackers do.

Fuzzing fills the gap between "does it work correctly?" and "does it fail safely?" Those are two very different questions, and most test suites only answer the first one.

How we actually run fuzz tests

At BetterQA, fuzzing is part of our DAST (Dynamic Application Security Testing) work. We built an AI Security Toolkit with over 30 scanners, and fuzzing is integrated into the dynamic analysis pipeline.

Here is how a typical engagement works:

1. Map the attack surface. Before we fuzz anything, we need to know what exists. We crawl the application, identify all endpoints, document the expected input formats, and note which endpoints handle sensitive data (auth, payments, file uploads, admin functions).

2. Seed the fuzzer with valid inputs. Good fuzzing starts with valid data. We capture real requests from the application (with test accounts, never production data), and the fuzzer uses these as templates. It knows what a valid request looks like, so it can make targeted mutations rather than purely random noise.

3. Run mutation-based fuzzing. The fuzzer takes each valid input and generates thousands of variants. Wrong types, boundary values, encoding tricks, oversized payloads, special characters, null bytes, format string patterns. Each variant gets sent to the endpoint, and we capture the response code, response body, response time, and any server-side logs.

4. Triage the findings. Not every crash is a security vulnerability. Some are just robustness issues (the server returns a 500 but recovers cleanly). Some are actual security holes (the server leaks data, accepts the malformed input as valid, or enters an inconsistent state). We classify each finding by severity and exploitability.

5. Verify and document. Every finding gets manually verified. We reproduce the crash, confirm the root cause, and write up the fix. No false positives in the final report.

For web applications, we often use OWASP ZAP as one of the tools in this pipeline. For APIs, we combine custom fuzzing scripts with tools like Burp Suite's Intruder or purpose-built API fuzzers. For projects with unusual protocols (IoT devices, custom binary formats), we write targeted fuzzers from scratch.

When to fuzz (and when not to)

Fuzzing works best when:

You have a public-facing API that accepts user input
You process file uploads
You handle payment or financial data
You parse complex data formats (JSON, XML, CSV, binary protocols)
You have already done basic security testing and want to go deeper

Fuzzing is less useful when:

The application has no external input surface (purely internal batch processing)
You have not done basic input validation yet (fix the obvious stuff first, then fuzz)
The codebase changes so frequently that findings become stale before they are fixed

The best time to start fuzzing is after your first round of functional testing is stable but before you go to production. That is when the cost of fixing issues is lowest and the risk of missing something is highest.

The security testing reality in 2024

As Tudor Brad, BetterQA's founder, puts it: "It's a good versus evil game right now." AI is accelerating development speed, which means more code ships faster, which means more potential vulnerabilities reach production faster. Features that used to take months now take days. The testing has to keep pace.

Fuzzing is one of the few techniques that scales with code output. You do not need to manually write a test case for every possible malformed input. The fuzzer generates them. You just need to point it at the right targets and have someone who knows what they are looking at to triage the results.

If you have never run a fuzzer against your application, I would strongly suggest trying it on a staging environment. The results will probably surprise you. We have yet to fuzz a non-trivial application and find zero issues. Every single engagement has turned up something the existing test suite missed.

The question is never "does my software have these bugs?" The question is "do I find them before someone else does?"

More on security testing and QA practices on the BetterQA blog.

Payment testing: the card types that break in production

Tudor Brad — Thu, 09 Apr 2026 08:34:46 +0000

The bug that costs you money twice

Last year we tested a fintech client's checkout flow. Everything passed in Stripe test mode. Green across the board. Then they went live in Germany and 30% of transactions started failing silently. No error page. No retry prompt. Just... nothing happened when the user clicked "Pay."

The problem was 3D Secure. Their integration handled the initial charge request fine, but never implemented the redirect flow for SCA (Strong Customer Authentication). In test mode, Stripe skips 3D Secure unless you explicitly use the 4000002760003184 test card. Nobody on the dev team had used that card. So nobody knew the integration was broken for every European card that required authentication.

The client found out when chargebacks started hitting. That is the worst way to discover a payment bug: your payment processor tells you, your bank tells you, and your users have already left.

Why payment bugs are different from other bugs

A broken image on your landing page is embarrassing. A broken payment flow is expensive. Here is what makes payment bugs uniquely painful:

Direct revenue loss. Every failed transaction is money that almost entered your account and didn't. If 5% of your transactions fail due to a card type you never tested, that is 5% of revenue gone. Not "at risk." Gone.

Chargebacks compound the damage. When a payment goes through incorrectly (wrong amount, duplicate charge, currency mismatch), you don't just refund the money. You pay chargeback fees. Enough chargebacks and your payment processor raises your rates or drops you entirely.

User trust evaporates instantly. People are anxious about money. A single failed payment makes a user question whether your site is legitimate. They won't debug it for you. They will close the tab and buy from someone else.

Silent failures hide the problem. Unlike a 500 error that shows up in your monitoring, many payment failures happen at the processor level and return a generic decline. Your logs show "card_declined" but the real cause is that your integration doesn't handle the card network correctly.

This is why we treat payment testing as its own discipline, not just "form validation with a credit card field."

Card types that actually break things

Here are the specific card type issues we run into repeatedly when testing payment integrations for clients.

Amex and the 15-digit problem

American Express cards have 15 digits and a 4-digit CVV (called CID). Visa and Mastercard have 16 digits and a 3-digit CVV. This sounds trivial until you see how many integrations hardcode maxLength="16" on the card number input and maxLength="3" on the CVV field.

We tested a SaaS platform where Amex cards were being silently rejected. No error message. The form just wouldn't submit. The frontend validation required exactly 16 digits, so any 15-digit PAN was treated as incomplete. The user saw a disabled submit button and assumed they typed something wrong.

Test cards to use:

Amex:           3782 822463 10005    (15 digits, 4-digit CID)
Visa:           4242 4242 4242 4242  (16 digits, 3-digit CVV)
Mastercard:     5555 5555 5555 4444  (16 digits, 3-digit CVV)

What to check:

Card number field accepts 15, 16, and 19 digits
CVV field accepts both 3 and 4 digits
Card type detection updates dynamically (Amex logo appears when you type 37xx)
Backend validation matches frontend rules

UnionPay and 19-digit PANs

UnionPay cards can be 16, 17, 18, or 19 digits long. If your validation regex is ^\d{16}$, you are rejecting a card network used by over a billion people.

We see this constantly in integrations targeting Asian markets. The dev team builds and tests with Visa/Mastercard, launches in Singapore or Malaysia, and gets support tickets from users who "can't enter their card number."

UnionPay (19):  6200 0000 0000 0000 003
UnionPay (16):  6200 0000 0000 0005

The fix is straightforward: accept 13-19 digits and let the payment processor handle network-specific validation. Your frontend should not be the gatekeeper for PAN length.

Diners Club and the 14-digit edge case

Diners Club cards traditionally have 14 digits, though newer ones may have 16. If your system strips spaces and then checks length === 16, Diners Club users cannot pay.

Diners Club:    3056 9309 0259 04   (14 digits)

This one is less common globally but still matters if you operate in parts of South America or accept corporate cards. We have seen it break on subscription billing platforms where the initial charge worked (the card was tokenized by Stripe directly) but a later recurring charge failed because the platform's own validation ran during a card update flow.

3D Secure and SCA failures

This is the big one. 3D Secure (3DS) adds an authentication step where the card issuer verifies the cardholder, usually through a redirect or iframe popup. In the EU, SCA regulations make this mandatory for most online transactions.

The problem: Stripe's test mode does not trigger 3DS by default. You need to explicitly use test cards that simulate the 3DS flow:

3DS required:       4000 0027 6000 3184
3DS required (fail): 4000 0084 0000 1629
3DS optional:       4000 0025 0000 3155

What breaks:

The redirect URL is not configured, so the user gets sent to a blank page
The return handler does not check payment_intent.status after the redirect
Mobile webviews block the 3DS popup, so the authentication never completes
The webhook handler does not account for the requires_action status

We tested a client's mobile app where 3DS worked perfectly in the browser but failed 100% of the time in the iOS webview. The app's WKWebView had javaScriptEnabled set to true but blocked popups, which is how the 3DS challenge was presented. Every EU user on iOS could not complete a payment.

Currency and amount edge cases

Currency bugs are sneaky because they often produce a valid charge for the wrong amount. The user gets billed, the amount looks plausible, and nobody notices until reconciliation.

Common issues we test for:

Zero-decimal currencies. JPY, KRW, and several others do not use decimal subunits. If your system sends 1000 to Stripe for a 10.00 USD charge (correct, because Stripe uses cents), sending 1000 for a JPY charge means 1000 yen, not 10 yen. The amount field interpretation changes by currency.

# USD: $10.00 = 1000 (cents)
# JPY: 1000 yen = 1000 (no subunit)
# BHD: 10.000 BD = 10000 (three decimal places)

Rounding on conversion. If your platform shows prices in EUR but charges in USD after conversion, rounding differences can mean the user sees 9.99 EUR but gets charged 10.01 EUR equivalent. Small difference. Big trust problem.

Minimum charge amounts. Stripe requires a minimum of 50 cents USD (or equivalent). If your platform allows a 0.10 USD tip or a discount that reduces the charge below the minimum, the payment fails at the processor level with a generic error.

How we structure payment test suites

When we pick up a payment integration project, here is the sequence we follow. This is not theory. This is what we actually run.

Phase 1: Card type coverage matrix. We build a grid of every card network the client wants to support, crossed with every payment scenario (one-time charge, subscription, refund, partial refund, card update). Each cell gets tested. No assumptions that "if Visa works, Mastercard works."

Phase 2: Authentication flows. We test every 3DS path: success, failure, abandonment (user closes the popup), timeout, and network error during redirect. We test on desktop browsers, mobile browsers, and in-app webviews separately because they behave differently.

Phase 3: Error handling and messaging. We trigger every decline code Stripe can return (insufficient funds, expired card, incorrect CVV, processing error, card not supported) and verify the user sees a specific, actionable message. "Payment failed" is not acceptable. "Your card was declined. Please check your card details or try a different payment method" is the minimum.

Phase 4: Webhook reliability. We verify that payment confirmation does not depend solely on the client-side redirect. If the user closes their browser after 3DS but before the redirect completes, the webhook from Stripe should still update the order. We test this by intentionally killing the browser session mid-payment and confirming the backend processes the webhook correctly.

Phase 5: Currency and locale. We test with cards issued in different countries, in different currencies, with different locale settings on the browser. A Japanese user with a JPY card on a platform that prices in USD should see a coherent experience from price display through to their bank statement.

Stripe test cards quick reference

For developers setting up their own payment test suites, here are the cards we use most often:

Scenario	Card number	Notes
Success	`4242 4242 4242 4242`	Always succeeds
Generic decline	`4000 0000 0000 0002`	Always declined
Insufficient funds	`4000 0000 0000 9995`	Specific decline reason
Incorrect CVC	`4000 0000 0000 0127`	CVC check fails
Expired card	`4000 0000 0000 0069`	Expiry check fails
3DS required	`4000 0027 6000 3184`	Triggers authentication
3DS failure	`4000 0084 0000 1629`	Authentication fails
Amex	`3782 822463 10005`	15 digits, 4-digit CID
Dispute/chargeback	`4000 0000 0000 0259`	Triggers dispute

Use any future expiry date and any 3-digit CVC (4-digit for Amex). For full documentation, check Stripe's testing page.

The test mode trap

Here is the pattern we see over and over: a team builds a payment integration, tests it thoroughly in Stripe test mode, and ships it. Then production breaks in ways that test mode never revealed.

Test mode is not production. It does not enforce SCA. It does not check real BIN ranges. It does not apply real fraud detection rules. It does not connect to actual card networks. It is a simulation, and like all simulations, it has blind spots.

The gap between test mode and production is where payment bugs live. You can narrow that gap by using the right test cards, testing authentication flows explicitly, and verifying webhook handling under failure conditions. But you cannot eliminate it entirely without production monitoring.

We always recommend that clients set up real-time alerting on payment failure rates. A 2% failure rate on day one that creeps to 8% by day thirty means something changed at the processor or issuer level, and no amount of pre-launch testing catches that.

What we have learned from testing payments across clients

After testing payment integrations for fintech and e-commerce clients at BetterQA, a few things stand out:

Card type validation belongs at the processor level, not your frontend. Let Stripe or Adyen validate the PAN. Your job is to not block valid cards before they reach the processor.
3D Secure is not optional in Europe. If you sell to EU customers and your integration does not handle 3DS, you will lose transactions. Not might. Will.
Test the sad paths harder than the happy paths. A successful payment needs to work. A failed payment needs to communicate clearly. Most teams spend 90% of testing time on success and 10% on failure. We flip that ratio.
Webhooks are your safety net. Client-side confirmation is unreliable. Browsers crash, users close tabs, networks drop. Your backend must handle payment confirmation through webhooks independently of what happens in the browser.
Currency handling is a category of bugs, not a single check. Zero-decimal currencies, three-decimal currencies, conversion rounding, minimum amounts: each one is a distinct failure mode.

Payment bugs are expensive, embarrassing, and preventable. The card types and scenarios in this article are the ones we see break most often. Test them before your users find them for you.

More on how we approach QA for complex integrations: betterqa.co/blog

The automation mistakes we keep fixing on inherited test suites

Tudor Brad — Thu, 09 Apr 2026 08:29:18 +0000

I have inherited a lot of test suites. Some were built by contractors. Some were built by developers who drew the short straw. A few were started by QA engineers who left the company before anyone else learned how the framework worked.

They all break in the same ways.

At BetterQA, automation suite maintenance is a significant chunk of our work. We build suites from scratch, yes, but we also take over existing ones. And after years of doing this across dozens of clients and tech stacks, I can tell you the failure modes are remarkably consistent.

Here are the mistakes I keep seeing, what they actually cost, and how we fix them.

Hardcoded waits everywhere

This is the single most common problem. Open any inherited suite and you will find sleep(5000) or cy.wait(5000) or time.sleep(5) scattered through the code like confetti.

I understand why it happens. A test is flaky. The page takes a moment to load. Someone adds a wait, the test passes, the PR gets merged. Problem solved, right?

No. Problem deferred.

Here is what hardcoded waits actually cost you:

They make your suite slow. A 5-second wait runs for 5 seconds whether the element appeared in 200 milliseconds or 4.9 seconds. Multiply that across 300 tests and you have added 25 minutes of pure wasted time to every CI run. That is 25 minutes your developers sit waiting for a green check before they can merge.

They mask real problems. If your app genuinely takes 5 seconds to render a button, that is a performance bug. A hardcoded wait hides that bug. An explicit wait with a reasonable timeout surfaces it.

They are still flaky. The app loads in 5 seconds on your machine. On the CI runner with limited resources, it takes 7 seconds. Now the test fails again and someone bumps the wait to 10.

The fix is straightforward but requires discipline. Replace every hardcoded wait with an explicit condition: wait for the element to be visible, wait for the network request to complete, wait for the loading spinner to disappear. Playwright and Cypress both have built-in mechanisms for this. Use them.

// This is the problem
await page.waitForTimeout(5000);
await page.click('#submit');

// This is the fix
await page.waitForSelector('#submit', { state: 'visible' });
await page.click('#submit');

When we take over a suite, the first thing we do is search for sleep, wait, and timeout calls. Replacing those alone typically cuts suite runtime by 30-40%.

No page object pattern (or an abandoned one)

The second most common problem is raw selectors duplicated across dozens of test files. The login page selector #email-input appears in 40 different tests. The dashboard navigation selector .nav-item.active shows up in 60.

Then the frontend team renames a CSS class and 60 tests break simultaneously.

The page object pattern exists specifically to solve this. You define your selectors in one place, your tests reference the page object, and when the UI changes you update one file instead of 60.

What I see more often than no page objects at all is an abandoned page object pattern. Someone started it, created page objects for the login page and maybe the dashboard, and then the team got busy and started writing selectors inline again. Now you have a codebase with two patterns, and you have to check both places when something breaks.

If you are going to use page objects, commit to them. Every new test file should use them. If you are reviewing a PR that introduces a raw selector for a page that already has a page object, send it back.

We have also started using Flows, our Chrome extension that records browser interactions and generates self-healing test selectors. The self-healing part matters because it addresses the brittle selector problem directly: if your selector breaks because someone changed a class name, Flows detects the shift and adapts. That removes the most painful part of page object maintenance, which is keeping selectors current when the frontend moves fast.

Testing implementation details instead of behavior

This one is subtle and I still catch experienced engineers doing it.

A test that checks expect(component.state.isLoading).toBe(false) is testing implementation. A test that checks expect(screen.getByText('Dashboard')).toBeVisible() is testing behavior.

Why does the distinction matter? Because implementation changes constantly. Someone refactors the loading state from a boolean to an enum. Someone moves from local state to a global store. Someone replaces the custom spinner with a library component. Every one of those changes breaks the implementation test while the actual user-facing behavior stays identical.

Tests should answer one question: does the user see what they expect to see?

When I audit a suite, I look for tests that reference internal state, internal method names, or specific DOM structure beyond what the user actually sees. Those tests are maintenance liabilities. They will break during refactors that change zero user-facing behavior, and every false failure erodes the team's trust in the suite.

Write your test assertions the way a user would describe the expected result. "I click submit and I see a confirmation message." Not "I click submit and the Redux store's formSubmission.status field equals SUCCESS."

No cleanup between tests

Tests should be independent. Each test should set up its own preconditions and clean up after itself. This is testing 101 and it is violated constantly.

The symptom is test order dependence. Test A creates a user, Test B assumes that user exists, Test C deletes the user. Run them in order and everything passes. Run Test B alone and it fails. Run them in parallel and you get race conditions.

I once inherited a suite where the entire test run depended on the first test creating a specific database seed. If that first test failed for any reason, every subsequent test failed too. The team had been living with this for a year, re-running the suite whenever the first test had a hiccup, and treating it as normal.

That is not normal. That is a test suite that can only give you useful signal when conditions are perfect, which in CI environments is roughly never.

The fix involves two things:

Before-each hooks for setup. Every test (or test group) should create the data it needs. If test B needs a user, test B creates that user in a beforeEach block.

After-each hooks for teardown. Delete what you created. Reset the state. Log out the session. If you are using an API to create test data (which you should be for speed), use that same API to clean up.

// Each test owns its own data
beforeEach(async () => {
  testUser = await api.createUser({
    email: `test-${Date.now()}@example.com`
  });
});

afterEach(async () => {
  if (testUser) {
    await api.deleteUser(testUser.id);
  }
});

This adds a few seconds of setup per test but it eliminates an entire category of flakiness. The tradeoff is worth it every time.

Running everything sequentially when tests could run in parallel

Most test suites I inherit run every test in sequence. 400 tests, one after another, 45 minutes total. The team complains about slow CI. Nobody has tried parallelization.

If your tests are independent (and after fixing the cleanup problem above, they should be), there is no reason they cannot run in parallel. Playwright supports parallel execution out of the box. Cypress has parallelization through their dashboard or through CI matrix strategies. Even pytest can parallelize with pytest-xdist.

The objections I hear are usually:

"Our tests share a database." Then give each parallel worker its own database, or use unique prefixes per worker so the data does not collide.

"Some tests are slow and some are fast, so parallelization does not help much." Use test sharding based on historical run times, not naive splitting by file.

"We tried it and got flaky results." That means you have test isolation problems (see the cleanup section above). Fixing isolation fixes parallelization.

On a recent client project we took a suite from 52 minutes sequential to 11 minutes across 6 parallel workers. Same tests, same CI machine. The only changes were fixing test isolation and enabling Playwright's built-in parallelism.

The real cost of bad automation

A bad test suite is worse than no test suite.

That sounds extreme, but I mean it. A suite full of hardcoded waits, brittle selectors, and order-dependent tests produces two outcomes, both harmful:

First, it creates false failures. Tests break for reasons unrelated to actual bugs. Developers learn to ignore the failures, re-run the suite, and merge anyway when it passes on the second try. At that point the suite is not catching bugs. It is a random gate that sometimes blocks merges for no reason.

Second, it creates false confidence. Tests pass, so the team assumes the feature works. But the tests were checking implementation details that happen to still match, not actual user behavior that might have regressed. Bugs reach production despite a green test suite, and leadership starts questioning whether automation was worth the investment.

The fix is not to abandon automation. The fix is to treat your test suite as production code. It needs code review. It needs refactoring. It needs maintenance. It needs someone who knows what they are doing.

What a healthy suite looks like

After we clean up an inherited suite, the result usually has these properties:

Zero hardcoded waits. Every wait is explicit and condition-based.
Page objects for every page. Selectors live in one place.
Behavior-focused assertions. Tests describe what the user sees, not how the code works internally.
Full test isolation. Any test can run alone or in any order.
Parallel execution. Suite runtime is measured in minutes, not close to an hour.
Self-healing selectors where possible. Tools like Flows reduce maintenance when the UI changes frequently.

None of this is revolutionary. It is basic engineering discipline applied to test code. The problem is that test code rarely gets the same attention as application code, and the debt accumulates until someone inherits the suite and has to deal with it.

If you are building a suite from scratch, build it right from the start. If you have inherited one that has these problems, fix them incrementally: start with the waits, then add page objects for the most-referenced pages, then fix isolation one test group at a time.

And if you would rather hand that work to someone who has done it dozens of times before, that is literally what we do.

More on automation, testing strategy, and QA engineering at betterqa.co/blog.

Shift-left testing sounds great until you try to get invited to the meeting

Tudor Brad — Thu, 09 Apr 2026 08:29:13 +0000

I have never met a single person in software who disagrees with shift-left testing in theory. Earlier testing catches cheaper bugs. The data is clear. The logic is obvious.

And yet.

Try walking into a design review as a QA engineer. Try asking a product manager if you can sit in on sprint planning. Try suggesting that testing should start before a single line of code exists.

You will get polite resistance. You will get scheduling conflicts that are not really conflicts. You will get "we'll loop you in later" emails that never arrive.

I have been running QA teams for years, and the hardest part of shift-left has never been the testing. It has been the politics.

The math that everyone already knows

IBM published the numbers decades ago, and they have been validated repeatedly since. A bug found during requirements costs roughly $100 to fix. The same bug found in production costs $10,000 or more. That is a 100x multiplier.

The Systems Sciences Institute at IBM put specific ranges on this:

Requirements phase: $100 per defect
Design phase: $300-600 per defect
Implementation: $1,000-2,000 per defect
System testing: $3,000-5,000 per defect
Production: $10,000+ per defect

NIST backed this up with their own study estimating that software bugs cost the US economy $59.5 billion annually, with more than half of that cost attributable to bugs that could have been caught earlier.

These are not controversial numbers. Every engineering leader has seen some version of this chart. It shows up in conference talks, blog posts, and onboarding decks at half the tech companies on the planet.

So why does testing still start late?

The resistance nobody talks about

Here is what actually happens when you try to shift left.

Developers feel watched. When a tester shows up to a design meeting, some developers interpret it as distrust. "Why do we need QA here? We haven't even written anything yet." The subtext is: you are here to find problems with my thinking, and I did not sign up for that.

Product managers feel slowed down. Sprint planning already takes too long. Adding QA concerns means discussing edge cases, error states, and unhappy paths before anyone has committed to a direction. PMs want to move fast and refine later. QA wants to think through failure modes before the work begins. Those two instincts collide.

Testers feel unwelcome. After a few rounds of being the person who raises problems in meetings full of people who want solutions, many QA engineers stop pushing. They wait for the handoff. They test what they are given. The shift-left conversation dies quietly.

I have watched this pattern play out at dozens of organizations. The people are not wrong for feeling what they feel. The resistance is human, and pretending it does not exist is why most shift-left initiatives fail.

Two clients, two outcomes

I want to share two real situations because the contrast is stark.

Client A brought us in during the requirements phase. We sat in on product discussions. We reviewed wireframes and user stories before development started. We wrote test scenarios alongside acceptance criteria.

The result: their production bug rate dropped by roughly 50% within three months. Not because we were catching more bugs in testing, but because our questions during requirements eliminated entire categories of bugs before they were ever coded. Things like "what happens if the user has two active sessions?" or "does this flow work for users who skipped onboarding?" would surface in design, and the team would address them in the spec.

Client B brought us in at staging. We got builds after development was done, filed bugs, and watched them get fixed in the next sprint. Classic.

Their bug count in production stayed roughly flat quarter over quarter. They kept shipping the same types of bugs: missing validation on edge cases, broken flows for uncommon user paths, accessibility gaps nobody thought about. The bugs were not hard to find. They were predictable. They were the kind of bugs that disappear when someone asks the right questions during design.

Same QA team. Same processes. Same tools. The only difference was when we entered the picture.

What shift-left actually looks like in practice

Shift-left is not about running unit tests earlier, though that helps. It is about involving testing thinking earlier. There is a difference.

Here is what works:

Test case design during requirements. Before a story moves to development, write the test scenarios. Not automated scripts. Just plain-language descriptions of what you are going to verify. This forces everyone to agree on expected behavior before anyone starts building.

QA review of acceptance criteria. Testers are better at finding ambiguity in specs than developers are, because testers think about what could go wrong. A developer reads "user can update their profile" and thinks about the happy path. A tester reads the same story and asks: what fields are required? What happens with special characters? Can they update while another session is active? Is there a character limit?

Bug categorization and traceability. Track where bugs originate. If 60% of your production bugs trace back to unclear requirements, that is your argument for QA in the requirements phase. Hard numbers beat theoretical frameworks every time.

Pair sessions between QA and dev. Not formal meetings. Just a developer and a tester spending 20 minutes talking through a feature before implementation. These conversations catch misunderstandings that would otherwise become bugs.

Getting past the resistance

Here is the honest advice, based on what I have seen work.

Start small. Do not try to get QA invited to every meeting. Pick one feature or one team. Demonstrate value with a contained experiment. When the team sees fewer bugs coming back from that feature, they will ask for more QA involvement on their own.

Lead with questions, not criticism. The fastest way to get uninvited from design meetings is to be the person who says "that won't work." Instead, ask questions. "How should this behave when the API is slow?" is more welcome than "you haven't considered the timeout case." Same concern, different framing.

Show the cost data for your own project. The IBM numbers are nice but generic. What actually moves people is your project's own data. Pull the bug reports from the last quarter. Categorize them by root cause. Calculate the time spent fixing production bugs versus the time it would have taken to catch them in requirements. That is a number your PM will care about.

Accept that shift-left is a spectrum. You do not need QA at every design meeting to get value. Even moving from "QA starts at staging" to "QA reviews stories before sprint commitment" is a significant shift. Take the win.

Where automation fits

Test-driven development, automated regression suites, CI/CD pipelines with quality gates: all of these are shift-left tools. They move verification earlier by making it cheaper to run tests frequently.

But automation is the easy part. You can set up a CI pipeline in an afternoon. Getting a product manager to add 15 minutes of QA review to their sprint planning ceremony takes months of relationship building.

The teams that get the most out of shift-left are the ones that combine both: automated checks that run early and often, plus human testers who participate in the thinking that happens before code exists.

The uncomfortable truth

Shift-left testing works. The data is overwhelming. The case studies are consistent. Organizations that test earlier ship better software with fewer production incidents.

But it requires something that no framework or tool can provide: it requires developers and product managers to voluntarily share their planning process with people whose job is to find problems. That is an act of trust, and trust takes time.

If you are a QA leader trying to push shift-left, be patient. Build relationships before building processes. Demonstrate value in small doses. And accept that the human side of this change is harder than the technical side.

The bugs do not care about your feelings. They will be cheaper to fix early whether your team is ready to hear that or not. Your job is to make them ready.

We write about testing practices, QA strategy, and the realities of running independent QA teams at betterqa.co/blog.

We went 100% automation on a client project. Here's what broke.

Tudor Brad — Thu, 09 Apr 2026 08:23:34 +0000

Last year we had a client come to us after they'd fired their entire manual QA team. They'd invested six months into a Cypress suite with 400+ tests, hired two automation engineers, and felt confident they had testing covered.

Three weeks after the manual testers left, their support tickets tripled.

The automated suite was passing. Every single run: green. And their users were reporting bugs that no script had ever thought to check for.

I've seen this play out at BetterQA more times than I can count. A team gets excited about automation, treats it as a silver bullet, and then learns the hard way that a green CI pipeline is not the same thing as a working product.

This is the story of what actually happens when you go all-in on automation and abandon manual testing entirely.

The green suite problem

Here's the thing nobody tells you about a 100% automated test suite: it only checks for things you already thought of.

Every automated test starts as a human decision. Someone sat down, considered a scenario, and wrote a script to verify it. That script will faithfully run that same check forever. It will never wonder "what happens if I click this button twice really fast?" or "does this flow still make sense after the last redesign?"

On this particular project, the client's suite covered login flows, CRUD operations, payment processing, and a handful of API contract tests. Solid coverage on paper. But nobody was testing:

What happens when a user fills out a form, leaves for 20 minutes, and comes back
Whether the new dashboard layout actually makes sense to someone seeing it for the first time
How the mobile experience feels on a slow 3G connection
Whether the error messages help users recover or just confuse them

These are the kinds of things a manual tester catches in the first five minutes of an exploratory session. No script in the world is looking for them.

Automation is a regression tool, not a testing strategy

I need to be blunt about this because the industry has muddied the water: automation testing and software testing are not the same thing.

Automation is phenomenal at regression. You fixed a bug? Write a test so it never comes back. You have a critical payment flow? Automate it so every deploy verifies it still works. You need to run the same checks across 12 browser/device combinations? Automation saves you days of repetitive work.

But regression is only one slice of testing. Exploratory testing, usability evaluation, edge case discovery, accessibility review, "does this feature actually solve the user's problem" testing: none of that can be scripted. Not because the tools aren't good enough, but because the value of those activities comes from human judgment and creativity.

When our client killed their manual team, they didn't just lose testers. They lost the people who understood how real users interact with the product.

What broke (specifically)

Let me walk through the actual failures we saw on this project, because abstract arguments are easy to dismiss.

Usability regressions went unnoticed for weeks. The dev team shipped a redesigned settings page. The automation suite verified that every button and input worked. What it couldn't tell them was that the new layout was confusing: users couldn't find the save button because it was below the fold on most screens. Support tickets piled up. A manual tester would have caught this in one session.

Edge cases multiplied. The suite tested the happy path and a few known error states. But real users do unpredictable things. They paste formatted text from Word documents into plain text fields. They open the app in two tabs and edit the same record simultaneously. They use browser autofill in ways that break client-side validation. The automation engineers couldn't write scripts fast enough to cover the edge cases that a curious manual tester would stumble into organically.

False confidence from false negatives. The suite had several tests that were passing but not actually verifying what they claimed to verify. A selector had drifted after a UI update, so the test was clicking a different element and asserting on stale data. Green check mark, zero value. When we audited the suite, about 8% of the tests were essentially testing nothing. A manual tester running the same scenarios would have noticed immediately that the behavior was wrong.

Dynamic content broke silently. The app served personalized dashboards with data-driven layouts. The automation suite used hardcoded selectors and fixed test data. Every time the personalization engine changed what was displayed, tests either broke (noisy failures that got ignored) or passed incorrectly (silent failures that hid real issues). The team spent more time maintaining flaky tests than they saved by automating.

The maintenance tax nobody budgets for

This is the part that surprises teams the most. Automation isn't "write it once and forget it." It's a living codebase that needs maintenance, refactoring, and debugging just like your production code.

On this project, the two automation engineers were spending roughly 60% of their time maintaining existing tests and only 40% writing new coverage. Every UI change, every feature flag toggle, every API response format update meant updating test scripts.

Compare that to a manual tester who can adapt on the fly. The button moved? They find it. The API response changed shape? They notice the UI looks different and investigate. The feature flag is on? They test the new behavior. No script updates required.

I'm not saying maintenance is a reason to avoid automation. I'm saying that if you don't budget for it, your "cost savings" from firing the manual team evaporate fast.

Complex scenarios resist automation

Some testing scenarios are genuinely hard to automate well. Multi-step workflows that span multiple systems, tests that depend on timing or environmental conditions, scenarios that require judgment calls about whether the output "looks right."

We had one case where the client needed to test a document generation feature. The automation could verify that a PDF was produced and that it contained certain text strings. But it couldn't tell whether the formatting was correct, whether the layout was readable, or whether the generated content actually made sense in context. A human looks at the PDF and immediately knows if something is off.

This isn't a tooling limitation that better frameworks will solve. It's a fundamental constraint: some quality attributes require human perception.

What we actually recommend

When we onboarded this client, we didn't tell them to throw away their automation suite. That would have been equally wrong in the other direction.

We helped them build a balanced approach:

Automate regression and smoke tests. The things that need to pass on every deploy, the critical paths that must always work, the repetitive checks across environments and devices. This is where automation earns its keep.

Keep manual testers for exploratory work. Dedicate time for testers to explore new features without a script. Let them break things creatively. Give them the freedom to follow their instincts when something feels off. This is where you find the bugs that matter most to users.

Use manual testing for usability evaluation. Before any major release, have a real human go through the key flows and ask: does this make sense? Is this intuitive? Would I be frustrated if I were a customer? No automated tool can answer these questions.

Rotate who does exploratory testing. Don't limit it to QA. Developers, product managers, designers: fresh eyes catch things that familiar eyes skip. The person who built the feature is the worst person to evaluate whether it's intuitive.

Review your automation suite quarterly. Audit for false negatives, outdated selectors, tests that pass but don't verify anything meaningful. Prune ruthlessly. A smaller suite that actually catches bugs is worth more than a massive suite that gives you false confidence.

The AI question

I'll address the elephant in the room because everyone asks. With AI-powered testing tools getting better every month, does this change the equation?

Our founder Tudor Brad has a line I keep coming back to: "AI will replace development before it replaces QA."

His reasoning is sound. AI can generate code, but the act of evaluating whether that code does what users actually need requires human judgment. And with AI accelerating development speed (features that used to take months now take hours), the volume of things that need testing is exploding. You don't need less QA in an AI-accelerated world. You need more.

AI tools are great at generating test cases, identifying patterns in bug reports, and even doing some basic visual regression checking. But the core question of "does this product work for real humans in real situations" still requires a human in the loop.

The real lesson

The client I mentioned at the start? After we helped them rebuild their testing approach with a mix of automation and manual testing, their support ticket volume dropped back to pre-automation-only levels within six weeks. They kept their Cypress suite. They also brought back two manual testers.

The lesson isn't that automation is bad. The lesson is that automation is a tool, not a strategy. And the teams that treat it as the entire strategy are the ones who end up with a green pipeline and angry users.

If your test suite is passing and your users are still finding bugs, the suite isn't the problem. What's missing is the human who would have found those bugs first.

More on testing strategy and QA methodology on the BetterQA blog.

Manual testing isn't dying, but manual testers need to change

Tudor Brad — Thu, 09 Apr 2026 08:23:30 +0000

I run a QA company with 50-plus engineers spread across 24 countries. Roughly half of them do manual testing. Not because we're behind the times. Because that's what our clients need.

Every conference talk, every LinkedIn influencer, every bootcamp curriculum pushes the same story: automate everything, manual testing is a relic, if you're clicking through a UI in 2024 you're wasting money. I've heard this for years. And every year, the demand for skilled manual testers at BetterQA grows.

So let me say what I actually think. Manual testing isn't dying. But the version of manual testing that people imagine when they hear the phrase? That version probably should die.

The boring manual testing is already dead

Let me be clear about what I'm not defending.

If your manual testing process involves a tester opening a spreadsheet of 200 test cases, clicking through each one in sequence, writing "pass" or "fail" in a column, and repeating this before every release, then yes. Automate that. Automate it yesterday. That kind of work destroys morale, produces inconsistent results, and costs more per bug found than any reasonable automation framework.

We automated repetitive regression testing years ago. We built Flows, a Chrome extension that records browser interactions and replays them as tests with self-healing selectors. The entire point was to free our manual testers from the mechanical parts of the job so they could spend time on the work that actually requires a human brain.

When people say "manual testing is dying," they usually mean this repetitive, scripted, follow-the-checklist kind. And they're right. It should die. The problem is that they then leap to the conclusion that all manual testing should die, and that's where they're wrong.

What manual testers actually do now

The testers on my team who do manual work aren't clicking through login forms all day. Here's what their week actually looks like.

They spend time in exploratory testing sessions, deliberately trying to break things in ways nobody anticipated. They navigate the product the way a confused user would, not the way a specification document describes. They find bugs that no automation script would ever catch because no one thought to write a test for that scenario.

They review designs and requirements before a single line of code gets written. This is the cheapest place to find defects. A bug caught in a requirements review costs almost nothing to fix. The same bug found in production costs 100 times more. That's not an exaggeration. It's a well-documented cost multiplier that's held up across decades of software engineering research.

They do usability assessments. They sit with the product and ask questions like: would a real person understand this flow? Does this error message actually tell you what went wrong? Is the button where you'd expect it to be? Automation can tell you whether a button exists on the page. It cannot tell you whether the button makes sense.

They run accessibility checks. Not just automated scans (those miss roughly 60-70% of real accessibility barriers), but actual screen reader walkthroughs, keyboard-only navigation, cognitive load evaluation. A WCAG compliance tool will tell you that a form label exists. A manual tester will tell you that the label says "Field 3" and means nothing to anyone.

They probe for security issues. Not full penetration testing necessarily, but the kind of poking around that finds exposed data in API responses, broken authorization checks, session handling problems. With AI-generated code flooding into production, this kind of investigative work matters more than it did five years ago.

The "automate everything" pressure is real, and it's partially nonsense

I get why engineering leaders push for full automation. The pitch is seductive. Write the tests once, run them forever, get fast feedback, reduce headcount. What's not to like?

Here's what I've seen happen in practice.

A client moves to 100% automation. Their Selenium or Playwright suite covers all the happy paths beautifully. CI runs green. Everyone feels confident. Then they ship a feature where the shopping cart total displays correctly but the font is 4px and grey on grey. A human would catch that in seconds. The automation suite doesn't check font sizes because nobody thought to add that assertion. A customer screenshots it, posts it on Twitter, and suddenly "fully automated QA" looks a lot less impressive.

Another client automates their entire regression suite. Takes three months and costs a fortune. Then the product team redesigns the navigation. Forty percent of the automated tests break, not because of bugs but because the selectors changed. Now you have an automation maintenance backlog that's bigger than the original testing backlog. The team spends more time fixing tests than writing new ones.

Automation is powerful, genuinely powerful, for specific categories of testing. Cross-browser compatibility. Regression on stable features. Performance benchmarks. Data-driven tests where you need to run the same flow with 500 different input combinations. For those things, automation is not just better than manual testing, it's the only sane option.

But automation is terrible at answering "does this feel right?" It can't do creative exploration. It can't notice that the loading spinner is technically working but feels sluggish in a way that will irritate users. It can't look at a form and realize that the field order doesn't match the mental model a healthcare administrator has when processing patient intake.

What's actually changing for manual testers

Here's the honest part that the "manual testing is dead" crowd gets right, even if they get the conclusion wrong. The job description is changing fast.

Five years ago, a junior manual tester could get by with basic test case execution skills. Open the app, follow steps, report results. That's not enough anymore.

The manual testers who are thriving on our team have skills that overlap with product management, security analysis, and UX research. They understand API calls well enough to check what's happening under the hood when the UI looks fine. They use browser DevTools to inspect network requests, check response payloads, verify that sensitive data isn't leaking in places it shouldn't be. They understand enough about accessibility standards to do meaningful evaluations, not just run an axe scan and forward the results.

They're also comfortable working alongside automation. On most of our client projects, the same team handles both. A manual tester explores a new feature, finds the edge cases, documents them, and then works with the automation engineer to decide which paths are worth scripting for regression and which are one-time exploratory findings. That collaboration is where the real quality comes from. Not from one discipline replacing the other.

Our founder Tudor Brad has a line he uses a lot: "AI will replace development before it replaces QA." It sounds provocative, and he means it to be. But the core point is serious. AI tools can generate code. They can even generate test scripts. What they cannot do is understand whether a product feels right to use, whether a workflow makes sense for the specific humans who will use it, or whether a security boundary that technically exists is actually robust enough. That requires judgment, creativity, and domain knowledge that nobody has automated yet.

The vibe coding problem

This part is new, and it matters.

We're seeing more client projects where significant chunks of the codebase were generated by AI tools. GitHub Copilot, Claude, ChatGPT, whatever the flavour of the month is. The code works, mostly. It passes the unit tests that the AI also generated. And it ships with subtle bugs that only surface when a real person uses the product in ways the AI didn't anticipate.

I've seen AI-generated form validation that checked email format but not length, allowing a 10,000-character email to crash the backend. I've seen AI-generated pagination that worked perfectly for pages 1 through 10 but returned duplicate results on page 11. These aren't exotic edge cases. They're the kind of thing a manual tester finds in their first hour with the feature because they naturally try inputs that a generated test suite doesn't consider.

As AI-assisted development accelerates the speed of feature delivery, the demand for people who can thoughtfully evaluate those features goes up, not down. Features that took three months to build now take three hours. That same speed produces more surface area for defects. You need testing that can match that pace, and skilled exploratory testers are faster at covering new ground than any test automation framework.

What I tell testers who are worried about their careers

If you're a manual tester and you're nervous about automation replacing you, I understand the anxiety. But I think the threat is misidentified.

The thing that will make you irrelevant is not automation. It's refusing to evolve what "manual testing" means for you personally.

Learn how to use browser DevTools. Understand enough about APIs to read a response payload. Get comfortable with accessibility testing beyond just running a scanner. Develop a specialty: security probing, or usability evaluation, or data integrity analysis. Understand CI/CD pipelines well enough to know when and where your testing fits in the release process.

You don't need to become a programmer. But you need to be more than someone who follows a test script. The testers on my team who are most in demand with clients are the ones who can sit in a sprint planning meeting, hear a feature described, and immediately start asking questions that expose gaps in the requirements. That's not a skill automation replaces. It's a skill that makes automation more effective because it ensures the right things get automated in the first place.

The honest answer

Manual testing isn't dying. What's dying is the job description that says "execute pre-written test cases and record results." That work is being absorbed by automation, and it should be.

What's growing is the need for people who can think critically about software quality, who can explore products with creativity and suspicion, who can translate technical findings into business risk, and who can evaluate whether something that technically works actually works well for the people who'll use it.

The boring repetitive stuff? Automate it and don't look back.

The creative investigative work? That's more valuable now than it's ever been. And I don't see that changing anytime soon.

If you're interested in how we approach testing at BetterQA, or you want to see more of our thinking on QA in the AI era, check out betterqa.co/blog.

The test case mistakes we see on every new client project

Tudor Brad — Thu, 09 Apr 2026 08:16:21 +0000

I lead QA onboarding at BetterQA. When a new client signs on, one of the first things I do is audit their existing test suite. I open it up, scroll through a few hundred test cases, and within about twenty minutes I can tell you exactly how much of it is useful.

Usually? About half.

That might sound harsh, but after doing this across dozens of client projects with a team of 50+ engineers, the patterns are so consistent it's almost boring. The same mistakes, the same dead weight, the same "we wrote these two years ago and nobody's touched them since."

The worst version of this is inheriting a 2,000-test suite where the team proudly tells you their pass rate is 97%. Then you look closer and realize 600 of those tests have no real assertions. Another 300 are duplicates with slightly different names. A hundred are flaky and get re-run until they pass. The 97% number is meaningless. It just makes everyone feel good while bugs keep shipping to production.

Here's what I keep finding.

Tests that test the framework, not the app

This is the single most common problem, and it's the sneakiest one because the tests look legitimate. They run. They pass. They show up green in the CI pipeline. Everyone's happy.

But the test isn't actually verifying that your application does something correctly. It's verifying that React renders a component. Or that a form element exists on the page. Or that clicking a button fires an event handler.

I saw a suite last year where someone had written 40 tests for a checkout flow. Every single one was checking that UI elements rendered. Not one test verified that an order was actually created, that inventory was decremented, or that the payment was processed. The checkout could have been completely broken and all 40 tests would still pass.

The fix is simple but requires discipline: every test needs to assert something about your business logic, not about whether your framework is doing its job. If you're testing that a button exists, that's a framework test. If you're testing that clicking the button creates an order with the correct line items, that's an application test.

Tests with no meaningful assertions

Related but distinct from the framework problem. These tests go through a whole flow, click things, fill out forms, navigate between pages, and then... nothing. No assertion at the end. Or a single assertion that checks something trivial, like the page title.

I opened a Cypress suite for a client last quarter and found 15 tests that navigated to various pages and asserted cy.url().should('include', '/dashboard'). That was it. The tests confirmed you could reach the dashboard. They said nothing about whether the dashboard was showing the right data, whether the charts loaded, whether the filters worked.

The tester who wrote them probably had good intentions. They were probably under pressure to increase test coverage numbers. So they wrote tests that technically covered pages without actually verifying anything useful.

If your test doesn't have an assertion that would fail when the feature breaks, it's not a test. It's a page visit.

Copy-pasted tests with wrong expected values

This one physically hurts when I find it. Someone writes a solid test for Scenario A. Then they need a similar test for Scenario B, so they copy-paste and change a few things. But they forget to update the expected values. Now you have a test for Scenario B that's asserting Scenario A's expected output, and it's been passing for months because the assertion is loose enough to match both.

We onboarded a fintech client where this was happening in their pricing calculation tests. Three variants of a discount test all expected the same final price, even though the discount percentages were different. Nobody noticed because the tests passed. The actual discount logic had a bug that made all three discounts produce the same result, which was wrong, but the tests said everything was fine.

Copy-paste is fine. But you have to treat every pasted test as a new test. Read the expected values. Ask yourself if they make sense for this specific scenario. Better yet, calculate them independently rather than copying them from the original.

Flaky tests that nobody fixes

Every team has them. Tests that fail randomly, pass on retry, and gradually erode everyone's trust in the suite. The typical lifecycle goes like this:

Test starts failing intermittently
Someone adds a retry mechanism
Retries mask the flakiness
Team stops investigating failures because "it's probably just flaky"
Real bugs start slipping through because failures get dismissed

I've seen teams with 30-40 known flaky tests that they just re-run whenever CI fails. At that point, your CI pipeline isn't catching bugs. It's a slot machine that eventually gives you a green build if you pull the lever enough times.

The painful truth is that flaky tests are usually flaky for a reason: timing dependencies, shared state between tests, hardcoded test data that conflicts with other tests, or assumptions about the order things load. These are fixable problems. They just require someone to sit down and actually diagnose them instead of adding another retry.

At BetterQA, when we inherit a flaky suite, the first thing we do is quarantine the flaky tests. Move them out of the main pipeline. Run them separately. Then fix them one by one. It's tedious work but it's the only way to make the suite trustworthy again.

No separation between smoke, regression, and edge cases

When every test has the same priority and runs in the same pipeline, you end up with 45-minute CI runs where critical path tests are mixed in with obscure edge case validations. A developer pushes a one-line CSS fix and waits 45 minutes to find out if it broke anything.

The result is predictable: people start skipping CI, merging without waiting for tests, or just ignoring red builds because "it's probably that one slow test again."

A healthy suite has layers. Smoke tests that run in under 5 minutes and cover the critical paths. Regression tests that run on merge to main. Edge case and exploratory tests that run nightly or on-demand. When everything is lumped together, nothing gets the attention it deserves.

Test data that's hardcoded and brittle

Hardcoded IDs, specific usernames, dates that assume a certain timezone, URLs that point to a staging server that got decommissioned six months ago. I see all of these constantly.

The worst case I encountered was a test suite that had a user's actual production email address hardcoded in 200+ tests. The tests were hitting a staging API, but if anyone accidentally pointed them at production, they'd spam a real customer with test emails. Beyond the safety issue, those tests broke every time the staging database got refreshed because the hardcoded user no longer existed.

Test data should be created by the test, used by the test, and cleaned up by the test. If your test depends on something that already exists in the database, it's one environment reset away from failing.

Tests that verify implementation, not behavior

This is a subtler problem but it kills test suite longevity. When tests are tightly coupled to implementation details (specific CSS selectors, internal component state, exact API response shapes), any refactoring breaks them even if the behavior is identical.

I've watched teams avoid refactoring because "it would break too many tests." That's backwards. Tests should give you confidence to refactor. If they're blocking refactors, they're testing the wrong things.

Test the behavior the user sees. The login form accepts credentials and redirects to the dashboard. The search returns relevant results. The export generates a file with the correct data. If you refactor the internals and those behaviors still work, your tests should still pass.

No traceability between tests and requirements

This is the organizational problem underneath all the technical ones. When tests aren't linked to requirements, user stories, or bug reports, nobody knows which tests matter and which are leftovers from features that were redesigned or removed.

We built BugBoard partly because of this problem. When you can see which tests are actually catching bugs versus which ones have been passing quietly for two years without ever failing, you start to understand the real health of your suite. A test that has never failed might be rock-solid validation of a stable feature. Or it might be testing nothing useful. Without traceability, you can't tell the difference.

How we fix this when onboarding clients

When we take over a test suite, the process looks like this:

Audit pass: read every test, tag it with what it actually validates, flag the ones with weak or missing assertions
Quarantine flaky tests: pull them out of the main pipeline, track them separately
Prioritize by risk: map tests to features ranked by business impact, find the gaps where critical features have no coverage
Kill the dead weight: delete tests that test framework behavior, have no assertions, or duplicate other tests
Fix what remains: stabilize the flaky tests, update hardcoded data, decouple from implementation details

It's not glamorous work. It takes time. But the difference between a 2,000-test suite with 50% useful coverage and a 900-test suite with 95% useful coverage is enormous. The smaller suite runs faster, fails for real reasons, and actually catches bugs before they ship.

The uncomfortable math

If you have 1,000 tests and 400 of them are noise, every developer on your team is waiting for those 400 useless tests to run on every CI build. Multiply that wait time by the number of builds per day, the number of developers, and the number of working days in a year. You're burning weeks of engineering time on tests that provide zero value.

That's before you count the cognitive cost. When developers see tests fail and their first reaction is "it's probably flaky," you've already lost. The test suite has become background noise instead of a safety net.

Start with honesty

The hardest part of fixing a test suite is admitting it needs fixing. Nobody wants to hear that the 2,000 tests they spent months writing are half useless. But the alternative is continuing to invest in something that gives you false confidence while bugs keep reaching production.

If you want to see the patterns I've described in your own suite, start with one question: for each test, what specific bug would this catch? If you can't answer that clearly, the test needs work or removal.

We write about testing patterns, QA team structure, and what we learn from client projects on our blog: betterqa.co/blog

Your team is confusing bug severity with priority, and it's costing you sprints

Tudor Brad — Thu, 09 Apr 2026 08:16:16 +0000

I've sat through hundreds of sprint planning sessions where someone says "this is a P1" and someone else says "no, it's a sev-3" and then the whole room argues for fifteen minutes about a tooltip that renders wrong on Firefox. Nobody ships anything. The standup runs long. Half the team checks out mentally because they've had this exact argument before.

The root problem is simple: most teams use "severity" and "priority" interchangeably, and that confusion creates real damage. Bugs get fixed in the wrong order. Critical issues sit in backlog while someone polishes a cosmetic fix that a stakeholder complained about in Slack.

At BetterQA, we triage thousands of bugs across dozens of client projects every month. This confusion shows up constantly, and it's one of the first things we fix when onboarding a new team.

Severity is about impact, priority is about urgency

That's the whole distinction. Once you internalize it, triage gets dramatically faster.

Severity answers: how broken is this? How much damage does the bug cause to the system, the data, or the user's ability to do their job?

Priority answers: how soon do we need to fix it? Given everything else on our plate, where does this land in the queue?

These two axes are independent. They correlate sometimes, but treating them as the same thing is where teams lose sprint capacity.

The examples that make it click

I use two examples when explaining this to new QA engineers, and they tend to stick.

Low severity, high priority: the CEO's bio typo. Someone misspelled the CEO's name on the company About page. The system works perfectly fine. No functionality is broken. No data is corrupted. Severity? Low. But the CEO noticed it, sent a message to the VP of Product, and now three people are asking when it will be fixed. Priority? High. Fix it today.

High severity, low priority: the edge case crash. There's a bug where the app crashes if a user enters exactly 47 special characters into a phone number field during registration. The app completely dies. Severity? High, it's a full crash. But it affects roughly 0.1% of users in a flow that has a validation fallback anyway. Nobody has actually reported it in production. Priority? Low. Log it, schedule it for a future sprint, move on.

If your bug tracker doesn't let you set these independently, you'll default to whatever field you have and lose the nuance. This is exactly why we built BugBoard with separate severity and priority fields. The distinction matters for triage, and collapsing them into one dimension forces bad decisions.

What severity levels actually look like

I've seen teams use three levels, five levels, even seven. The number matters less than consistency. Here's a practical five-level scale that works across most projects:

Critical (sev-1)

The system is down, data is being lost or corrupted, or a core workflow is completely blocked for all users. Payment processing fails. Login is broken. The database is returning errors. There is no workaround.

If you're debating whether something is sev-1, ask: "Can users do the primary thing they came here to do?" If the answer is no, it's sev-1.

Major (sev-2)

A significant feature is broken or behaving incorrectly, but the system is still usable. Users can work around it, but the workaround is painful or non-obvious. Think: search returns wrong results, file uploads fail intermittently, or a key report generates incorrect numbers.

Moderate (sev-3)

Something is clearly wrong but the impact is contained. A secondary feature misbehaves. A form doesn't validate one edge case properly. Sorting works on most columns but breaks on date fields. Users notice it but can still get their work done.

Minor (sev-4)

Cosmetic issues, UI inconsistencies, or small deviations from the spec that don't affect functionality. A button is slightly misaligned. A success message uses the wrong shade of green. Text truncates awkwardly at one specific viewport width.

Trivial (sev-5)

Issues so minor that most users would never notice them. A tooltip appears 200ms late. There's extra whitespace at the bottom of a page that only shows on one browser. The "about" link in the footer points to a slightly outdated version of the page.

What priority levels actually look like

Priority is a business decision, not a technical one. That's why product managers, project leads, or client stakeholders typically set priority, while QA engineers set severity. The people closest to the technical impact assess severity. The people closest to the business impact assess priority.

Immediate (P1)

Drop what you're doing and fix this now. The fix goes into the current sprint, possibly as a hotfix outside the normal release cycle. Reserved for situations where the bug is actively causing business damage: lost revenue, broken SLAs, security vulnerabilities being exploited.

High (P2)

Fix this in the current sprint. It's important enough to bump something else out of the sprint if needed. Stakeholders are watching. Customers have noticed.

Medium (P3)

Schedule this for the next sprint or two. It needs to get done, but it's not urgent enough to disrupt current work. Most bugs land here, and that's fine.

Low (P4)

Fix it when you have time. Put it in the backlog and revisit during grooming. If it never gets fixed because higher-priority work keeps coming in, that might be acceptable.

Won't fix / defer (P5)

The team acknowledges the bug exists but has decided not to fix it, at least not in the foreseeable future. Maybe the feature is being deprecated. Maybe the cost of fixing it outweighs the impact. Document the decision and move on.

The four quadrants that matter for triage

When you separate severity and priority into two independent fields, you get a 2x2 matrix that makes triage decisions almost mechanical:

High severity + high priority: Fix immediately. System crash affecting many users, critical security hole, data corruption in a production workflow. This is your "all hands on deck" category.

High severity + low priority: Schedule carefully. The bug is technically severe but the real-world impact is low because of how rarely it occurs or because a workaround exists. Don't ignore it, but don't let it hijack your sprint either.

Low severity + high priority: Fix fast, but keep perspective. The CEO's typo. The client's logo rendered in the wrong color. A cosmetic issue on a landing page right before a big marketing push. Quick fix, high visibility, low technical risk.

Low severity + low priority: Backlog it. Minor UI polish, edge case behaviors that almost nobody encounters, small inconsistencies that don't affect usability. Groom these periodically and close the ones that are no longer relevant.

Where teams actually lose time

The damage isn't theoretical. I've watched it happen across projects.

Scenario 1: Everything is P1. A product owner marks every bug as high priority because they want everything fixed. The dev team has thirty P1 tickets and no way to distinguish between a broken payment flow and a misaligned icon. So they pick based on what seems easiest, or what they were already working near. The truly critical bugs get fixed by accident, not by design.

Scenario 2: Severity drives priority by default. The team uses a single field, or treats them as synonyms. A sev-1 crash that happens once a month in an internal admin tool gets treated with the same urgency as a sev-1 crash in the customer-facing checkout flow. One affects three people who already know the workaround. The other loses revenue every hour it's live.

Scenario 3: Nobody updates priority after initial triage. A bug was P3 when it was filed two months ago. Since then, the feature it affects has become the primary onboarding flow for a new enterprise client. It's now effectively P1, but nobody re-triaged it. The new client hits it on day one.

How we handle this at BetterQA

When we onboard a new client's QA process, one of the first things we audit is how they categorize bugs. More often than not, we find a single "priority" dropdown doing double duty, or severity levels that nobody on the team can define consistently.

We standardize on two separate fields with clear definitions that the whole team agrees on. QA sets severity based on technical impact. Product sets priority based on business context. When the two conflict, that conflict is the conversation worth having in triage, not "is this a P1 or a P2?"

In BugBoard, we enforce this separation at the tool level. Every bug has both fields. Reports can be filtered and sorted by either dimension independently. When you look at your backlog and filter for "high severity, low priority," you get a clear view of the technical debt that's accumulating quietly. When you filter for "low severity, high priority," you see the political fires that need quick attention.

Practical steps to fix this on your team

If your team is currently mixing these up, here's what I'd do:

1. Add both fields to your bug tracker. If your tool only has one, add a custom field. Every bug gets both a severity and a priority rating.

2. Define who owns each field. QA owns severity. Product or project management owns priority. If someone wants to change the other team's rating, that's a conversation, not a unilateral edit.

3. Write down your definitions. Put your severity scale and priority scale somewhere the whole team can reference. One page, plain language, with examples. Revisit it quarterly.

4. Use the 2x2 in triage. When reviewing new bugs, plot them mentally on the severity/priority grid. The quadrant tells you what to do. Stop debating feelings and start making decisions based on two clear dimensions.

5. Re-triage periodically. Priorities change. A P4 bug in January might be a P2 by March because the product roadmap shifted. Build re-triage into your grooming cadence.

It's a small distinction with a big payoff

Getting severity and priority right won't make your bugs disappear. But it will make your triage meetings shorter, your sprint planning more accurate, and your team less frustrated. When everyone agrees on what "this is critical" actually means, you stop arguing about vocabulary and start fixing the right things in the right order.

That's the whole point.

For more on QA practices, bug reporting, and how independent testing teams handle triage at scale, check out the BetterQA blog.

API testing with Cypress: the part most teams skip

Tudor Brad — Thu, 09 Apr 2026 08:09:02 +0000

I spend most of my working hours writing Cypress tests. UI flows, login forms, dashboards, the usual. But the tests that have saved me the most time and headaches over the past few years are the ones that never open a browser at all.

They hit the API directly with cy.request().

And almost nobody writes them.

The bug that only the API knew about

A few months ago I was testing a project management app for a client. The UI looked perfect. You could create a task, assign it, mark it done. All green in Cypress. Ship it.

Except the API was returning a 500 on every third POST request when the task description contained special characters. The frontend was silently swallowing the error and showing a success toast anyway because the developer had wrapped everything in a try-catch that defaulted to "ok."

The user would create a task, see a success message, and the task would simply not exist. No error. No feedback. Just gone.

I caught it by accident when I added a cy.request() test for the create endpoint. The UI tests had been green for weeks.

That's the problem. If you only test through the UI, you're testing the frontend's ability to hide failures. You're not testing whether the backend actually works.

Why cy.request() and not Postman?

Fair question. At BetterQA we use both. Postman is great for exploratory API testing and for sharing collections with developers. But when I need API tests running in the same pipeline as my UI tests, using the same config, the same env variables, the same reporting, cy.request() wins.

No extra tooling. No separate runner. No "well the Postman tests passed in Newman but the Cypress tests failed" confusion.

If your team already has Cypress installed, you have an API testing framework. You're just not using it yet.

The basics: hitting an endpoint and checking what comes back

Here's what a real API test looks like. Nothing fancy.

describe('Users API', () => {
  it('returns a list of users with the expected shape', () => {
    cy.request('GET', '/api/users').then((response) => {
      expect(response.status).to.eq(200);
      expect(response.body).to.be.an('array');
      expect(response.body.length).to.be.greaterThan(0);

      const user = response.body[0];
      expect(user).to.have.property('id');
      expect(user).to.have.property('email');
    });
  });

  it('creates a user and gets back a real ID', () => {
    cy.request({
      method: 'POST',
      url: '/api/users',
      body: {
        name: 'Test User',
        email: `test-${Date.now()}@example.com`,
      },
    }).then((response) => {
      expect(response.status).to.eq(201);
      expect(response.body.name).to.eq('Test User');
      expect(response.body.id).to.be.a('number');
    });
  });
});

Notice the Date.now() in the email. I learned this the hard way: if your test creates data, make it unique every run. Otherwise your second pipeline run fails with a "duplicate email" error and you waste 20 minutes debugging a test problem that isn't a test problem.

Authentication: the part people get stuck on

Most real APIs need auth. Here are the two patterns I use constantly.

Bearer tokens (JWT, OAuth, etc.):

describe('Protected endpoints', () => {
  let authToken;

  before(() => {
    cy.request({
      method: 'POST',
      url: '/api/auth/login',
      body: {
        email: Cypress.env('TEST_USER_EMAIL'),
        password: Cypress.env('TEST_USER_PASSWORD'),
      },
    }).then((response) => {
      authToken = response.body.token;
    });
  });

  it('returns profile data with valid token', () => {
    cy.request({
      method: 'GET',
      url: '/api/profile',
      headers: {
        Authorization: `Bearer ${authToken}`,
      },
    }).then((response) => {
      expect(response.status).to.eq(200);
      expect(response.body.email).to.eq(Cypress.env('TEST_USER_EMAIL'));
    });
  });

  it('rejects requests with no token', () => {
    cy.request({
      method: 'GET',
      url: '/api/profile',
      failOnStatusCode: false,
    }).then((response) => {
      expect(response.status).to.eq(401);
    });
  });
});

API key in a header:

cy.request({
  method: 'GET',
  url: '/api/data',
  headers: {
    'X-API-Key': Cypress.env('API_KEY'),
  },
});

Keep your credentials in cypress.env.json (and add that file to .gitignore right now if you haven't). In CI, pass them as environment variables prefixed with CYPRESS_.

One thing that bites people: failOnStatusCode: false. Without it, Cypress treats any non-2xx status as a test failure and throws. When you're intentionally testing a 401 or 404, you need this flag. I forget it about once a month.

Schema validation: the test that catches breaking changes

This is where API tests really earn their keep. Backend developers change response structures. They rename a field from created_at to createdAt. They drop a property. They add a nested object where there used to be a string.

Your UI might still work because JavaScript is forgiving. But your mobile client breaks. Or your integration partner's webhook stops parsing. Or the data is silently wrong.

it('user response has the required fields and types', () => {
  cy.request('GET', '/api/users/1').then((response) => {
    expect(response.body).to.have.all.keys(
      'id', 'name', 'email', 'created_at', 'role'
    );
    expect(response.body.id).to.be.a('number');
    expect(response.body.name).to.be.a('string');
    expect(response.body.email).to.match(/^[^\s@]+@[^\s@]+\.[^\s@]+$/);
    expect(response.body.role).to.be.oneOf(['admin', 'user', 'editor']);
  });
});

This test takes about 50 milliseconds to run. It will catch a breaking API change before your users do. That's a trade-off I will take every single time.

For bigger projects, look into chai-json-schema for full JSON Schema validation. But honestly, the simple assertions above cover 80% of what I need.

Testing the unhappy paths

Every junior tester writes tests for when things go right. The tests that matter are the ones for when things go wrong.

describe('Error handling', () => {
  it('returns 404 for a user that does not exist', () => {
    cy.request({
      method: 'GET',
      url: '/api/users/999999',
      failOnStatusCode: false,
    }).then((response) => {
      expect(response.status).to.eq(404);
      expect(response.body).to.have.property('error');
    });
  });

  it('returns 400 when required fields are missing', () => {
    cy.request({
      method: 'POST',
      url: '/api/users',
      body: { name: '' },
      failOnStatusCode: false,
    }).then((response) => {
      expect(response.status).to.eq(400);
      expect(response.body.errors).to.be.an('array');
    });
  });
});

I check three things on every error response: correct status code, an error message that makes sense, and that the body does not leak internal details (stack traces, database errors, file paths). You'd be surprised how many APIs return a full Node.js stack trace on a 500.

Combining API and UI tests

This is where Cypress really shines compared to standalone API tools. You can set up data through the API and then verify it in the UI:

it('shows a newly created task on the dashboard', () => {
  // Create data via API (fast, reliable)
  cy.request({
    method: 'POST',
    url: '/api/tasks',
    headers: { Authorization: `Bearer ${authToken}` },
    body: { title: 'Fix login bug', priority: 'high' },
  }).then((response) => {
    const taskId = response.body.id;

    // Verify it shows up in the UI
    cy.visit('/dashboard');
    cy.contains('Fix login bug').should('be.visible');
    cy.get(`[data-task-id="${taskId}"]`).should('exist');
  });
});

This pattern is faster and more reliable than creating data through the UI. Click-based setup is fragile. API-based setup gives you a known state in milliseconds.

Mocking APIs with cy.intercept()

Sometimes you need to test how the frontend handles a broken backend. That's where cy.intercept() comes in:

it('shows an error message when the API is down', () => {
  cy.intercept('GET', '/api/users', {
    statusCode: 500,
    body: { error: 'Internal Server Error' },
  }).as('getUsers');

  cy.visit('/users');
  cy.wait('@getUsers');
  cy.contains('Something went wrong').should('be.visible');
});

it('shows empty state when there is no data', () => {
  cy.intercept('GET', '/api/users', {
    statusCode: 200,
    body: [],
  }).as('getUsers');

  cy.visit('/users');
  cy.wait('@getUsers');
  cy.contains('No users found').should('be.visible');
});

I use this to test every error state the designer put in the mockups. If there's an empty state in the Figma file, there should be a test that forces that state through cy.intercept().

Organizing your tests so they don't become a mess

Once you have more than five or six API test files, structure matters.

cypress/
  e2e/
    api/
      users.cy.js
      auth.cy.js
      orders.cy.js
      payments.cy.js
    ui/
      login.cy.js
      dashboard.cy.js
  support/
    commands.js

And pull repeated API calls into custom commands:

// cypress/support/commands.js
Cypress.Commands.add('apiLogin', (email, password) => {
  return cy.request({
    method: 'POST',
    url: '/api/auth/login',
    body: { email, password },
  }).then((response) => {
    window.localStorage.setItem('token', response.body.token);
    return response.body.token;
  });
});

// In your tests:
beforeEach(() => {
  cy.apiLogin(Cypress.env('TEST_USER_EMAIL'), Cypress.env('TEST_USER_PASSWORD'));
});

I create a custom command for every API operation I call more than twice. Login, create user, create resource, cleanup. This keeps test files short and readable.

Running API tests in CI

API tests are fast because they skip the browser rendering. A suite of 50 API tests finishes in under 10 seconds. Add them to your pipeline:

# GitHub Actions
api-tests:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - run: npm install
    - run: npx cypress run --spec "cypress/e2e/api/**"
      env:
        CYPRESS_BASE_URL: ${{ secrets.API_BASE_URL }}
        CYPRESS_TEST_USER_EMAIL: ${{ secrets.TEST_EMAIL }}
        CYPRESS_TEST_USER_PASSWORD: ${{ secrets.TEST_PASSWORD }}

Run them on every PR. They're cheap and they catch real bugs.

When Cypress is not the right API testing tool

I'm not going to pretend Cypress is always the answer. Use Postman or a dedicated API framework when:

You're testing APIs that have no frontend at all
You need to generate load or stress test endpoints
You want API documentation generated from your test definitions
Your API tests need to run outside a Node.js environment

For everything else, especially when your team already has Cypress in the repo, just write the cy.request() tests. You already have the tool. Use it.

What I'd add to any test suite tomorrow

If I had to pick three API tests to add to a project that has zero, these are the ones:

Health check test - Hit /api/health or your main endpoint. Confirm it returns 200. This is your canary. If this fails, something is very wrong.
Auth rejection test - Hit a protected endpoint with no token. Confirm you get 401, not 200. You would not believe how many APIs return data to unauthenticated requests.
Schema test on your most-used endpoint - Pick the endpoint the frontend calls most. Assert every field name and type. This catches breaking changes before they reach production.

Three tests. Maybe 15 minutes to write. They'll save you hours.

We write tests like these on client projects every week at BetterQA, usually alongside Postman collections and full E2E suites. If you want to read more about how we approach testing, check out betterqa.co/blog.

Selenium vs Cypress: what we actually use and why

Tudor Brad — Thu, 09 Apr 2026 08:08:57 +0000

We have about 50 engineers across 24 countries working on client QA projects. On any given week, some of those projects run Cypress, some run Selenium, and a few run both. We did not pick sides. The client's stack, timeline, and constraints pick for us.

This is what we have learned from running both frameworks in production across dozens of projects. Not a feature matrix you can find on either tool's website. The actual pains and gains we deal with.

Where Cypress wins and why we reach for it

Cypress is the faster path to a working test suite on most modern web apps. That is the single biggest gain.

On a React or Vue SPA, a new tester can have a Cypress test running within an hour of cloning the repo. Install it, write a spec, run it. No driver downloads, no browser binaries to manage, no WebDriver protocol quirks. The test runner shows you what happened at each step with DOM snapshots. When a test fails, you can time-travel through the state to see exactly what went wrong.

For teams that write JavaScript and build SPAs, Cypress removes a pile of friction:

No driver management. Selenium needs ChromeDriver, GeckoDriver, etc., and they break every time Chrome auto-updates. Cypress bundles its own browser management.
Automatic waiting. Cypress retries assertions until they pass or time out. In Selenium, you write explicit waits or sleep statements, and you still get flaky tests.
Network stubbing built in. Intercepting API calls, mocking responses, testing error states: all native. In Selenium, you need a proxy tool like BrowserMob or mitmproxy.
Readable test output. The Test Runner GUI is genuinely useful for debugging. Selenium's output is a stack trace and a prayer.

We use Cypress on most greenfield SPA projects unless the client has a specific reason not to. It is the default recommendation when someone asks "what should we automate with?"

Where Cypress hurts

Here is the part that Cypress's marketing does not put on the homepage.

Single browser tab only. Cypress runs inside the browser. It cannot open a second tab. If your app opens a link in a new tab, sends you to an OAuth provider in another window, or does anything involving multiple browser contexts, you are stuck. We have had to rewrite application code to work around this on two separate client projects. That is not a testing framework problem, that is a testing framework creating an application problem.

Cross-origin is painful. Cypress historically blocked cross-origin navigation entirely. They added cy.origin() to handle it, but it is clunky. If your login flow redirects through an identity provider on a different domain, expect to spend time fighting Cypress rather than testing your app.

JavaScript only. Your test code must be JavaScript or TypeScript. If the team writes Python or Java and nobody knows JS, Cypress is not "easy to learn." It is easy to learn if you already know the language it requires. We have had QA engineers comfortable with Python spend weeks getting productive in Cypress because the language was the barrier, not the framework.

No mobile testing. Cypress tests web browsers. Period. If you need to test a native mobile app, or even a responsive site in an actual mobile browser, you need a different tool. We pair Cypress with Appium on projects that have both web and mobile, which means maintaining two frameworks anyway.

iframes are a headache. Cypress and iframes have a long, troubled history. The cy.iframe() command from community plugins works sometimes. Payment forms (Stripe, Braintree) that embed in iframes are consistently annoying to test with Cypress.

No parallel by default. Cypress's free tier runs tests sequentially. Parallel execution requires Cypress Cloud (paid) or a third-party orchestrator. On a project with 400+ tests, sequential runs took over 40 minutes. That kills CI feedback loops.

Where Selenium wins and why it survives

Selenium is 20+ years old and looks it. The API is verbose. The documentation sprawls across multiple projects. Setting up a grid for parallel execution is an infrastructure project in itself. Nobody loves writing Selenium tests.

But Selenium handles things Cypress cannot:

Any browser, any language. Java, Python, C#, Ruby, JavaScript, Kotlin. Chrome, Firefox, Safari, Edge, even IE if you have been cursed. A QA team can use whatever language they already know.
Multiple tabs and windows. driver.switchTo().window() just works. OAuth flows, popup windows, payment redirects: all testable without workarounds.
Cross-origin is not special. Selenium controls the browser from outside. It does not care what domain you navigate to.
Mobile testing via Appium. Appium is built on the WebDriver protocol. Skills and patterns transfer directly from Selenium to Appium. Your page object models work in both.
Mature ecosystem. Selenium Grid, Docker images, cloud providers (BrowserStack, Sauce Labs, LambdaTest) all support Selenium natively. The infrastructure is battle-tested.
Non-browser automation. With Appium's desktop drivers, you can automate Windows and macOS desktop apps using the same WebDriver API. Cypress cannot touch anything outside a browser.

We use Selenium on projects with complex auth flows, multi-window interactions, legacy browser requirements, or mixed web-and-mobile testing needs. It is also our choice when the QA team already has Java or Python expertise and there is no budget to retrain.

The pains we live with on Selenium

Selenium's problems are real and we deal with them weekly:

Flaky tests from timing issues. Selenium does not auto-wait. You write explicit waits, implicit waits, fluent waits. You still get StaleElementReferenceException at 2 AM in CI. Every Selenium project accumulates a utility class of retry helpers, and every team writes them slightly differently.

Driver version mismatches. Chrome 124 ships, ChromeDriver 124 is not ready yet, CI breaks. Selenium Manager (added in Selenium 4.6) helps, but we still see this on projects with locked-down CI environments that cannot auto-download drivers.

Verbose test code. A simple "click this button and check the text" test is 15 lines in Selenium and 3 lines in Cypress. Over hundreds of tests, that verbosity adds up. Code reviews take longer. New team members need more ramp-up time.

Grid management overhead. Running Selenium Grid (even with Docker) is operational work. Someone has to maintain the images, handle node scaling, debug session allocation. Cloud providers solve this but cost money.

No built-in visual feedback. When a Selenium test fails, you get a stack trace. Maybe a screenshot if you configured the teardown to capture one. There is no interactive debugger, no time-travel, no DOM snapshot. You read logs and re-run.

What about Playwright?

We would be dishonest if we did not mention Playwright here. Microsoft's framework has taken over a significant chunk of new projects since 2023. It handles multi-tab, cross-origin, and multiple browsers natively. It auto-waits like Cypress. It supports JavaScript, TypeScript, Python, Java, and C#.

On new projects where the team has no existing framework investment, we now recommend Playwright over both Selenium and Cypress more often than not. But Playwright is not the point of this article, and the reality is that most of our active client projects still run Selenium or Cypress because switching frameworks mid-project rarely makes business sense.

What we built to deal with both

One problem we kept hitting: QA engineers who were strong at manual testing but struggled to write automation code in either framework. We built Flows, a Chrome extension that records browser interactions visually and exports them as executable tests.

Flows does not replace either framework. It gives manual testers a way to create automated tests without writing code, and it gives automation engineers a starting point they can refine. When a recorded flow captures a complex user journey, the engineer can export it and clean it up rather than writing every step from scratch.

We built it because we were tired of the same bottleneck on every project: too many manual test cases, too few automation engineers, and a backlog of "we should automate this" tickets that never got done.

How we decide on each project

Our actual decision process is not complicated:

Pick Cypress when:

The app is a JavaScript/TypeScript SPA
The team knows JS
There are no multi-tab or cross-origin flows
No mobile testing requirement
The client wants fast CI feedback on a small-to-medium test suite

Pick Selenium when:

The team knows Java, Python, or C# and does not want to learn JS
The app has multi-window flows, OAuth redirects, or iframe-heavy payment forms
Mobile testing is also needed (Appium integration)
The client requires Safari or legacy browser coverage
There is an existing Selenium suite that works

Consider Playwright when:

Starting fresh with no existing framework
Need multi-browser, multi-tab, and cross-origin support
Team can work in JS/TS, Python, or Java
The client is open to a newer tool

Use Flows when:

Manual testers need to contribute to automation
There is a large backlog of manual test cases to convert
The team wants visual test recording regardless of the target framework

Neither framework solves your real problem

The honest answer nobody wants to hear: the framework choice matters less than most teams think. We have seen terrible test suites in Cypress and excellent ones in Selenium. The difference was never the tool. It was whether the team had clear test strategies, maintained their tests, and ran them consistently.

A Cypress suite that nobody maintains after sprint 3 is worse than no automation at all. It gives false confidence. A Selenium suite with proper page objects, good waits, and regular maintenance catches real bugs in production.

Pick the tool that fits your team and your app. Invest the time you save on setup into writing tests that actually matter. If you are spending more time debating frameworks than writing tests, you have already lost.

We write about testing from the perspective of a team that does it for a living across dozens of client projects. More at betterqa.co/blog.

We built an accessibility tool because spreadsheet audits were killing us

Tudor Brad — Thu, 09 Apr 2026 07:45:28 +0000

There's a specific kind of despair that comes from opening the ninth spreadsheet in a WCAG audit, the one you're pretty sure somebody duplicated from the wrong version two weeks ago, and finding that step 14 of the login journey is marked "Fail" in your copy and "Not Tested" in the reviewer's copy.

Which one is right? Nobody knows. The tester who originally logged it is on PTO. The screenshot is somewhere in Slack, probably in a thread that got buried under a deployment argument.

That was us, about two years ago, during a healthcare accessibility audit in the US. The client was strict, the regulations were strict, everything was strict except our tooling, which was held together with Google Sheets, manual dates, and hope.

We had nine spreadsheets open. One tracked testers. One tracked severity. One tracked notes. One was apparently a backup of another one, but with different data. Screenshots lived in Slack channels, sometimes in DMs, sometimes attached to Jira tickets that referenced a different version of the WCAG criteria.

And then we found the conflicting reports. Same user flow, same step, two different testers, two different results. One said Fail with a note about missing alt text. The other said Not Tested. Both had been submitted to the client in the same week.

That was the moment we stopped patching the process and started building something.

The tool is called Auditi

It lives at auditi.ro. We built it at BetterQA because nothing else matched how we actually test accessibility: by user journeys, broken into steps, with everything traceable back to a specific tester, date, platform, and WCAG criterion.

The core idea is simple. You model journeys the way a user experiences them. Login flow. Checkout flow. Onboarding. Each journey has steps. Each step gets an audit result: pass, fail, or not applicable. Every result has a tester name, severity, notes, evidence files, and a timestamp.

That sounds obvious. It isn't. In spreadsheet world, you're tracking all of that across columns, tabs, and files. Somebody renames a column. Somebody adds rows in the middle. Somebody filters by severity and forgets to unfilter before sending the report. I've watched an experienced QA engineer spend forty minutes rebuilding a pivot table that broke because Excel decided to reinterpret dates.

Auditi gives you filters by journey, tester, status, severity, platform, WCAG level, device, and date. If you've ever tried to find "that one iOS Safari fail from last Tuesday" in a spreadsheet, you understand why this matters.

What we actually built

Assignment dialogs and review queues, so work gets distributed without a Slack message chain. Pass/fail/N-A toggles per step, because that's the atomic unit of an accessibility audit. Notifications for deadlines and invites, because relying on people to check a spreadsheet daily doesn't work.

Then analytics. Pass rate over time. A WCAG compliance matrix. Breakdown by tester, by platform, by severity. This is the part that managers actually care about, and the part that's almost impossible to maintain in a spreadsheet without a dedicated person updating charts.

Reports export to Excel, PDF, and CSV. We kept that because the people who receive accessibility reports often live in those formats. Auditi generates Overview, Detailed, and Matrix reports.

We also added an AI-powered Smart Report that produces an executive summary, scores by WCAG level, flags top issues by priority, and suggests fixes. I'll be honest about this: AI summarization is useful here because it's compressing structured data, not making judgment calls. The tester still decides what passes and what fails. The AI just writes the summary you'd otherwise spend an hour drafting.

If you've tried to make a React app WCAG compliant

Here's where I want to talk to the developers reading this, because the auditing side is only half the problem. The other half is actually fixing things.

We run thirteen products in the BetterQA ecosystem. Different stacks: Vite/React SPAs, Next.js apps, a Laravel app, a WordPress site. Earlier this year we decided to do an accessibility sweep across all of them using our own scanner tool, which runs axe-core via Playwright.

The results were humbling.

Eight of our thirteen sites had accessibility scores below 60. The single biggest offender? Color contrast. Specifically, Tailwind's purple-400 on a white background.

Every Vite/React SPA in our ecosystem used text-purple-400 for links, badges, labels, secondary text. It's a nice color. It also has a contrast ratio of about 3.3:1 against white. WCAG AA requires 4.5:1 for normal text. We were failing dozens of contrast checks per page, across eight different sites, and nobody had noticed because the pages looked fine to us.

The fix: switch to purple-600 (#9333ea), which gives you 4.6:1. Just barely over the threshold, but it passes. We made the change across all eight sites. Some of them needed 24 individual class updates. One site, BetterFlow (a Laravel/Blade app), had the same pattern in Blade templates.

/* Before: 3.3:1 contrast - fails WCAG AA */
.text-purple-400 { color: #a855f7; }

/* After: 4.6:1 contrast - passes WCAG AA */
.text-purple-600 { color: #9333ea; }

That got us from the 50s to the 70s and 80s in accessibility scores. But it only caught the low-hanging fruit.

The deeper fixes taught us more

After the color contrast sweep, we went deeper. Sites like jrny.ro had icon-only buttons with no accessible name. Three buttons that a screen reader would announce as just "button." Fix: add aria-label attributes.

On menute.ro, we found eight form inputs and selects with no labels. A sighted user sees the placeholder text and understands the field. A screen reader user hears nothing useful. Fix: aria-label on each input.

The one that taught us the most was nis2manager.ro. The site uses CSS custom properties for its primary color. The original --primary value was set to an oklch lightness of 0.65. Changing it to 0.48 fixed over seventy contrast violations in one line of CSS. Seventy. From a single variable change.

/* One variable, seventy fixes */
--primary: oklch(0.48 0.2 270);  /* was 0.65 */

That's the lesson we keep coming back to: if your design system uses CSS custom properties or Tailwind theme colors, check the contrast of your base tokens first. You can hunt individual elements for hours, or you can fix the source and watch dozens of violations disappear.

What automated tools actually catch

I want to be direct about this because I've seen too many articles claim that automated accessibility testing solves the problem. It doesn't. Not even close.

Automated tools like axe-core catch maybe 30-40% of WCAG issues. They're good at color contrast, missing alt text, missing form labels, duplicate IDs, and broken ARIA attributes. They're bad at everything that requires context: whether alt text is actually meaningful, whether focus order makes sense, whether a custom widget is operable with a keyboard, whether content is understandable when read linearly by a screen reader.

WCAG has roughly 80 success criteria across levels A, AA, and AAA. Automated tools can reliably check maybe 25-30 of them. The rest need a human who understands the user flow, the intent of the content, and what the experience is like without a mouse.

That's why Auditi is structured around human-driven audits with journeys and steps, not around automated scan results. The automation is useful for catching regressions. It's not a substitute for a tester who actually navigates the site with a screen reader.

What we got wrong

A few things, since we're being honest.

The first version of Auditi was too complex. We modeled every WCAG criterion as a separate audit point, which meant testers had to click through dozens of criteria per step. Most of those were not applicable. We simplified it to let testers mark what matters and skip the rest.

We also underestimated how important the export format is. Early exports were clean but didn't match what compliance officers expected. We had to add specific report layouts that mapped to the documentation formats our healthcare and government clients were already using.

And our own ecosystem sweep revealed that we'd been shipping inaccessible products while building an accessibility tool. That stung. We fixed it, but it's a good reminder that building a tool and actually using it consistently are two different things.

The honest numbers

Our ecosystem scores before and after the sweep:

Site	Before	After	Primary fix
betterqa.co	54	84	Plugin color updates
betterflow.eu	55	84	24 Blade template fixes
auditi.ro	54	82	purple-400 to purple-600
electricworks.ro	53	80	Tailwind primary classes
psysign.ro	53	80	Same pattern
nis2manager.ro	51	76	CSS custom property (one-liner)
factos.ro	47	72	Same pattern

Not perfect scores. Not even close. But a 25-30 point jump across eight sites, and a process we can repeat.

Where this matters for your stack

If you're running a React or Vite SPA and you haven't run axe-core against it, do that first. You'll probably find contrast issues, missing labels, and button-name violations. Those are fixable in an afternoon.

After that, the harder work begins. Keyboard navigation. Focus management in modals and dynamic content. Screen reader announcements for state changes. That's where spreadsheets fall apart and you need actual audit tracking.

We built Auditi because we couldn't do that work well with the tools we had. It's at auditi.ro if you want to look at it.

For more about how we approach QA across different domains, there's the BetterQA blog.

Your staging environment is lying to you

Tudor Brad — Thu, 09 Apr 2026 07:45:24 +0000

I got a call from a client on a Tuesday morning. Their checkout flow was broken in production. Users couldn't complete purchases. Revenue was bleeding.

The thing is, their staging regression suite had passed. Every test green. The deployment went through without a hitch. And yet real users were hitting a payment confirmation page that spun forever, because a third-party webhook URL had been updated in production but not in the staging environment config.

Their regression tests checked that the checkout flow worked. They didn't check that the checkout flow worked with the actual production webhook endpoint, because staging had its own endpoint, and that one was fine.

This is not a rare story. I run QA operations across teams in 24 countries, and this exact pattern shows up every few weeks. A team invests serious effort into staging regression tests, those tests pass, and production breaks anyway. Not because the tests were wrong, but because the tests were answering the wrong question.

The "test it in staging" mindset

Most dev teams have a version of this workflow. Code gets written. It goes through code review. It lands in staging. Someone runs the test suite. If it's green, it ships.

The problem is that staging is a simulation. It's supposed to mirror production, but it never fully does. Different data volumes. Different third-party configurations. Different network conditions. Sometimes different infrastructure entirely.

When you make staging the primary place where quality gets verified, you've placed a bet that your simulation is accurate enough to catch real problems. And that bet fails more often than anyone likes to admit.

I've seen a team lose three days debugging a production outage caused by a database migration that worked perfectly in staging against 500 test records but locked the table for 40 minutes against 2.3 million production records. Their migration test passed. The test was useless because it tested the wrong scale.

What regression tests actually verify

Let me be specific about what regression tests in staging do well. They verify that previously working features still work after new code is introduced. If your login page worked last sprint and still works this sprint, that's regression testing doing its job.

But here's what regression tests in staging don't verify:

They don't verify that new features work under real user conditions. They check happy paths and known edge cases, not the creative ways actual humans interact with your product.

They don't verify that your environment configuration matches production. Staging has its own secrets, its own endpoints, its own feature flags. Any mismatch is invisible to your test suite.

They don't verify user workflows end-to-end across service boundaries. A user doesn't click one button and stop. They navigate through five screens, interact with three services, hit two payment providers, and receive an email. Your regression tests probably don't cover that full chain.

And they certainly don't verify what happens when things go wrong. What does your app do when the payment provider returns a timeout instead of a success? Your staging tests probably don't simulate that.

The real cost of late discovery

Bugs found in production cost 10-100x more to fix than bugs found during development. That's not a made-up number. IBM published this data decades ago and it's been validated repeatedly since.

But the dollar cost isn't even the worst part. The worst part is the context switch. When a production bug surfaces, the developer who wrote the code three weeks ago has to stop what they're doing, reload all the context for that feature, reproduce the issue, fix it, and ship a hotfix. That developer loses half a day minimum, and whatever they were working on gets delayed.

Multiply this by the average number of production bugs per sprint and you're looking at a real drag on velocity that nobody tracks because it looks like "unplanned work" in the sprint metrics.

One client tracked this explicitly for a quarter. They found that production bug fixes consumed 22% of their engineering capacity. Nearly a quarter of their team's time was spent fixing things that should have been caught before release.

Where bugs actually come from

When we do root cause analysis on production bugs that passed staging regression, the same categories come up repeatedly.

Environment differences. Config values, feature flags, API endpoints, database sizes. Staging says yes, production says no.

Untested user workflows. The regression suite tests individual features. Nobody tested the workflow where a user starts on mobile, switches to desktop midway through a multi-step form, and submits. That workflow broke because session handling differed between the two.

Integration timing. Service A calls Service B. In staging, Service B responds in 50ms. In production under load, Service B responds in 3 seconds. The calling code had a 2-second timeout that nobody tested against realistic latency.

Data shape surprises. Your tests use clean, well-formed test data. Production data has nulls where you don't expect them, Unicode characters in names, addresses with 8 lines, phone numbers with country codes your validation doesn't handle.

None of these are caught by running the same regression suite one more time.

Testing earlier, not just testing more

The fix isn't to write more regression tests and run them more often. The fix is to test different things at different stages.

In development, before code even reaches staging, you should be running integration tests against realistic data. Not the full suite, just the tests relevant to the change being made. If a developer changes the checkout flow, they should run the checkout integration tests locally, with production-scale data if possible, before pushing the PR.

During code review, someone should be asking: "What user workflow does this change affect, and have we tested that workflow end-to-end?" Not "does this function return the right value," but "can a user still complete their purchase after this change?"

In staging, yes, run the regression suite. But also run exploratory tests. Have a human actually use the feature the way a customer would. Click around. Try unexpected inputs. Navigate away mid-process and come back. These are the tests that catch the bugs automation misses.

Capturing what users actually do

One of the hardest parts of testing user workflows is knowing what those workflows are. Developers and testers don't use the product the same way customers do. They know the shortcuts. They avoid the rough edges. They don't make the mistakes that real users make every day.

This is why we built Flows, a browser test recorder that captures real user interactions. Instead of guessing which workflows matter, you record them. A tester walks through the actual user journey, Flows captures every click, every navigation, every form input, and turns it into a repeatable test. When someone says "test the checkout flow," you're not testing a developer's idea of the checkout flow. You're testing what users actually do.

The difference matters. We've had cases where the developer-written test covered 6 steps and the recorded user workflow covered 14 steps, because users do things like checking their cart twice, editing quantities, applying a coupon code, removing the coupon, adding a different one, and then checking out. The 6-step test passed. The 14-step test found a state management bug that corrupted the cart after coupon removal.

Tracking what escapes

The other half of this is tracking what gets past your testing. If a bug makes it to production, that's data. Not just "fix it and move on" data, but "why did this escape and how do we prevent the next one" data.

We built BugBoard partly for this reason. When a production bug gets reported, it goes into BugBoard with full context: what the user was doing, what they expected, what happened instead. But more importantly, we tag escape analysis on it. Did we have a test for this scenario? If yes, why didn't it catch it? If no, should we?

Over time, this builds a picture of your testing gaps. You stop seeing production bugs as random bad luck and start seeing them as predictable failures in specific categories. Maybe your team consistently misses accessibility regressions. Maybe edge cases in multi-currency handling always slip through. The pattern tells you where to invest your testing effort.

What staging should actually be for

I'm not saying staging is useless. Staging is valuable when it's treated as a final validation step, not the first real quality check.

By the time code reaches staging, you should already be confident that it works. Unit tests passed. Integration tests passed. A human walked through the user workflow at least once. Code review caught the obvious architectural problems.

Staging should be confirming what you already believe: this is ready to ship. It should be catching the rare environmental issues and last-minute integration problems. It should not be the place where you discover that a core feature is broken. If that happens regularly, your upstream testing has gaps that no amount of staging regression can fill.

Think of it like proofreading. If you hand a document to a proofreader and they find it's missing three chapters, something went wrong long before the proofreading stage. Proofreading catches typos and formatting issues. It assumes the content is already complete and coherent. Staging should work the same way.

A checklist for teams stuck in the staging trap

If your team keeps finding significant bugs in staging or, worse, in production after staging regression passes, here's where to start.

Audit your environment parity. List every configuration difference between staging and production. API keys, feature flags, database sizes, third-party endpoints. Any difference is a potential blind spot.

Map your user workflows. Not your test cases, your actual user workflows. Talk to support. Read the tickets. Watch session recordings if you have them. The gap between what you test and what users do is where production bugs live.

Test with production-scale data. If your staging database has 500 records and production has 5 million, your staging tests are performance theater. Either scale up your test data or run specific performance checks against production-like volumes.

Track your escapes. Every production bug should trigger a brief retrospective. Not a blame session, just a question: could we have caught this earlier, and if so, how?

Move testing left. Not as a buzzword, as a calendar event. Integration tests before code review. User workflow tests before staging. Staging becomes confirmation, not discovery.

Your staging regression suite passing is not evidence that your product works. It's evidence that your staging environment is internally consistent. Those are different things, and the difference shows up in production.

We work with dev teams who keep finding bugs too late in the cycle. More about how we approach testing at betterqa.co/blog.