You're about to launch that feature you've been working on for weeks. Everything looks good before deployment. But once you hit the deploy button, then ... boom.🥺
A critical bug surfaces on the live site. Customers start complaining, your team panics to fix it, and you are watching your CI pipeline, and all that wasted time is costing the company money.
Sound familiar? Let's tackle these problems with 5 testing patterns. At the end, you will get my favourite tool that will handle testing issues.
You don't need a massive QA team to catch the bugs that actually matter. You just need to be smarter about where you focus.
1. Smart Test Prioritization
A painful truth that I learned the hard way, running every test on every change is Overkill, wasteful, and surprisingly ineffective.
Think about it, if you're working on an e-commerce website, a bug in your payment processing is going to hurt more than a typo in your footer section. But most teams test everything with the same intensity.
What actually works is getting smart about the priorities of the test. Start tracking which tests catch real bugs versus which ones just eat up CI resources.
In most apps, you'll find that a small percentage of tests catch the majority of critical issues. Usually anything touching core user flows like authentication, payments, or data processing.
The best solution would be, tag your tests based on business impact and failure probability. Run your highest-risk tests on every pull request.
Save the medium risk ones for when you merge to main. Everything else can wait for overnight runs when CI costs are lower.
To the business point of view, your signup form breaking is worse than your admin dashboard loading 200ms slower. Test accordingly.
Launchable: The AI That Picks Your Tests
What it does: Launchable uses machine learning to analyze your codebase and predict Which tests are most likely to fail based on your recent changes? Instead of running all 2000 tests, it might tell you to run just 200 tests that have an 80% chance of catching any of the critical issues.
Why I love it: After using Launchable for 6 months, our CI pipeline went from taking 45 minutes to just 12 minutes on average. The AI gets smarter over time, learning from your team's patterns and becoming more accurate with predictions.
Getting Started:
# Install the CLI
pip install launchable
# Initialize your project (run this once)
launchable verify
# Record a build and its test results
launchable record build --name "build-$(git rev-parse --short HEAD)"
launchable record tests --build "build-$(git rev-parse --short HEAD)" \
junit test-results.xml
# Get smart test predictions for your next build
launchable subset --target 80% --build "build-$(git rev-parse --short HEAD)" \
--rest remaining-tests.txt \
npm test
The beauty is in the simplicity. Once set up, Launchable runs in the background, quietly learning which tests matter most for your specific codebase.
2. Progressive Test Coverage
Low test coverage makes everyone feel guilty. The spontaneous reaction is to write tests for everything until you hit that magical 90% coverage number. But there's a catch, not all code is created equal.
A better approach is to build coverage in focused phases:
Start with the money-makers: Unit tests for your core business logic such as: user registration, authentication, checkout processes. These are the features that directly impact company revenue.
Add integration coverage: Tests for your API endpoints and database interactions. A few well-placed integration tests often catch more real-world issues than dozens of isolated unit tests.
Finish with smoke tests: A handful of end-to-end tests that verify your critical user journeys work from start to finish. Think of as
is the site basically functional?
The key insight would be that perfect coverage is the enemy of useful coverage; rather have bulletproof tests for revenue generating features than tests for every edge case.
Codecov: Intelligent Coverage Analysis
What it does: Codecov tracks your test coverage over time and shows you exactly which lines of code aren't covered. But more importantly, it helps you understand coverage trends and identifies when new code is being pushed without appropriate testing.
Why it's a game-changer: Rather than obsessing over that overall percentage, Codecov shows you coverage for each pull request. You can set rules like new code must have 85% coverage while allowing legacy code to stay at 60%. This prevents technical debt from blocking progress.
Setup: GitHub Actions:
# .github/workflows/test.yml
name: Tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Setup Node
uses: actions/setup-node@v3
with:
node-version: '18'
- name: Install dependencies
run: npm ci
- name: Run tests with coverage
run: npm test -- --coverage --watchAll=false
- name: Upload to Codecov
uses: codecov/codecov-action@v3
with:
token: ${{ secrets.CODECOV_TOKEN }}
Smart Configuration:
# codecov.yml - Place this in your repository root
coverage:
status:
project:
default:
target: 75% # Overall project target
threshold: 2% # Allow 2% decrease
patch:
default:
target: 85% # New code must have higher coverage
ignore:
- "tests/"
- "**/*.test.js"
- "build/"
What I love most is the pull request comments. Codecov automatically comments on PRs showing exactly which lines need test coverage, making code reviews much more focused.
3. Environment-Specific Strategies
Not every environment needs the same level of testing intensity. Running your full test suite locally before every commit is like wearing a bulletproof vest to check the mailbox technically safer, but probably overkill.
- Locally: Just unit tests. Fast feedback, quick iterations.
- Staging CI: The full monty integration tests, E2E tests.
- Production: Tiny health checks that confirm we're not totally broken.
This isn't just about speed, it's about matching your testing intensity to the risk level of each environment.
MSW: One Mock Setup, Every Environment
What it does: MSW intercepts your API calls at the network level and returns mock responses. The best part is that your application code doesn't know it's talking to mocks instead of real APIs. Same code, different environments.
Why I switched to MSW: Before MSW, we had different mocking approaches for tests, development, and storybook. Now we use one set of mock definitions everywhere. When the backend team changes an API endpoint, we update the mock once and it works across all environments.
Basic Setup:
// src/mocks/handlers.js
import { rest } from 'msw'
export const handlers = [
// Mock user registration
rest.post('/api/users/register', (req, res, ctx) => {
const { email, password } = req.body
// Simulate different scenarios
if (email === 'existing@example.com') {
return res(
ctx.status(400),
ctx.json({ error: 'Email already exists' })
)
}
return res(
ctx.status(201),
ctx.json({
id: Math.random().toString(36),
email,
created_at: new Date().toISOString()
})
)
}),
// Mock login
rest.post('/api/auth/login', (req, res, ctx) => {
return res(
ctx.status(200),
ctx.json({
token: 'mock-jwt-token',
user: { id: '123', email: req.body.email }
})
)
})
]
For Tests:
// src/setupTests.js
import { setupServer } from 'msw/node'
import { handlers } from './mocks/handlers'
export const server = setupServer(...handlers)
beforeAll(() => server.listen())
afterEach(() => server.resetHandlers())
afterAll(() => server.close())
For Development:
// src/index.js (development only)
if (process.env.NODE_ENV === 'development') {
const { worker } = await import('./mocks/browser')
worker.start()
}
The setup is surprisingly simple, but the impact is huge. No more the backend is down blocking frontend development. No more flaky tests because external APIs are slow.
4. Automated Maintenance
It's better to have no tests at all than to have tests that don't work right. If your team starts ignoring tests because they fail for random reasons, you have a problem. Soon, real problems will be ignored because everyone is used to seeing failure warnings.
The fix is simple but requires discipline: set a failure threshold and stick to it. If a test fails more than 10% of the time over a reasonable period, it gets quarantined immediately. Either fix it or delete it, no exceptions.
Setting up automated tracking for flaky tests and regular team check-ins about test health can help maintain this discipline. The goal is to make test failures meaningful again.
Testomat.io: Test Health Monitoring Made Simple
What it does: Testomat.io automatically tracks which tests are flaky, how often they fail, and provides insights into test suite health. It integrates with your existing CI/CD pipeline and sends alerts when test reliability drops below your threshold.
Why it solved our flaky test problem: We had tests that would randomly fail once a week, and everyone just re-ran them. Testomat.io showed us that 15 specific tests were responsible for 80% of our CI failures. We fixed 5, deleted 10, and our build reliability went from 70% to 95%.
Integration Setup:
# Install the reporter
npm install @testomatio/reporter --save-dev
# Configure your test runner (Jest example)
# package.json
{
"scripts": {
"test": "jest --testLocationInResults --json --outputFile=report.json",
"test:report": "npm test && testomatio-reporter report.json"
}
}
Configuration:
// testomatio.config.js
module.exports = {
api_key: process.env.TESTOMATIO_API_KEY,
project: process.env.TESTOMATIO_PROJECT,
// Flaky test detection
flaky_detection: {
threshold: 0.1, // Flag tests failing >10% of the time
window_size: 50, // Over last 50 test runs
auto_disable: true // Automatically skip flaky tests
},
// Slack notifications
notifications: {
slack_webhook: process.env.SLACK_WEBHOOK,
notify_on: ['flaky_detected', 'build_failed']
}
}
CI Integration - GitHub Actions:
- name: Run tests and report
run: |
npm test
npx @testomatio/reporter report.json --env="CI"
env:
TESTOMATIO_API_KEY: ${{ secrets.TESTOMATIO_API_KEY }}
The weekly reports are incredibly helpful. Testomat.io shows you trends like "test reliability improved 15% this month" or "3 new flaky tests introduced". It turns test maintenance from reactive firefighting into proactive health monitoring.
5. Outsourced Execution with Bug0
Sometimes you reach a point where you need broader test coverage than your team can realistically maintain. Maybe you're scaling fast, adding features constantly, or your testing is becoming a development bottleneck.
This is where outsourced QA solutions can provide the extensive coverage you need without the maintenance overhead.
Bug0: AI QA Engineer Platform
One best option is Bug0, which acts as your AI QA Engineer while also being an AI QA platform that automatically explores your staging environment and creates comprehensive browser tests. The combination of automated discovery and human QA review can catch edge cases that internal teams often miss.
Bug0 AI QA Agent – acting as your AI QA Engineer inside GitHub
How it works: Bug0's AI crawler explores your web application like a user would, clicking buttons, filling forms, and following navigation paths. It automatically generates test cases and runs them across different browsers and devices, then provides detailed reports of any issues found.
Why it became the best safety net: We had a nasty bug slip through to production because it only happened on Safari mobile with a specific screen resolution. Our team couldn't realistically test every browser/device combination for every release. Bug0 caught 12 similar issues in the next month that we would have missed.
The real-world impact was in the first report Bug0 generated, which found 23 issues in our staging environment that we didn't know existed, such as broken links, form validation problems, and mobile layout issues. Some were minor, but 5 were bugs that would have definitely made it to production.
The weekly health reports track your application's stability over time and alert you when new deployments introduce regressions. This allows your engineering team to focus on building features instead of managing extensive test suites, while still maintaining comprehensive coverage across browsers and devices.
Keep It Simple, Keep It Focused
The best testing strategy isn't the most comprehensive one. It's the one your team will actually follow consistently. Smart testing is about being strategic with your resources and focusing on what matters most.
Here's what I've learned from implementing these patterns:
- Start small. Pick one pattern and one tool. Get it working well before moving to the next.
- Focus on pain points. If flaky tests are your biggest problem, start with Testomat.io. If slow CI is killing productivity, try Launchable.
- Measure the impact. Track metrics like CI time, test reliability, and bugs caught before production. These tools should make your life measurably better.
For fast-moving startups, adopting AI QA, with an AI QA Engineer like Bug0, is often the simplest way to ship faster without breaking the budget.
Start with the bugs that hurt your business most. Build coverage where it provides the highest return on investment. And remember, you're trying to catch problems before they reach your users, not create the perfect test suite.
The next time you're persuaded to test everything, ask yourself, what's the worst that could realistically go wrong? Then test for that.
Have you been dealing with similar testing headaches? I'm curious to hear how other teams handle this stuff. Drop me your thoughts!
Meet your AI QA Engineer.
Top comments (2)
Thanks for sharing.
Great Write