DEV Community: Marx Jenes

Sandbox Testing Tools for Teams Testing Against Real Dependencies

Marx Jenes — Wed, 16 Sep 2026 11:06:29 +0000

A team I worked near a while back had a "sandbox" environment that was really just a second production database with fewer permissions. Tests ran against it, passed, and nobody noticed the sandbox had drifted out of sync with what the real payment provider actually returned until a release broke checkout for a subset of users whose cards triggered an error path the sandbox had never simulated. The sandbox wasn't fake in the sense of being obviously unrealistic. It was fake in a quieter way — it just hadn't kept up.

That's usually the actual failure mode with sandbox testing, not "the sandbox is missing" but "the sandbox stopped matching reality months ago and nobody was checking." Worth going through what a sandbox needs to actually do its job, then a rundown of the tools teams commonly reach for.

What a sandbox testing tool actually needs to provide

Before comparing any sandbox testing tool, it's worth being specific about what "good" means here, because a lot of tools that call themselves sandboxes only solve part of this:

Isolation — tests shouldn't touch real production systems, real user data, or real third-party services with real consequences (real charges, real emails sent, real rate limits consumed).
Realism — the simulated environment needs to behave like the real one, including error cases, latency, and edge cases, not just the happy path someone remembered to configure.
Freshness — this is where most sandboxes quietly fail. A sandbox configured once and never updated drifts from the real dependency's actual behavior, the same way a hand-written mock does.
Reproducibility — the same test run against the sandbox should produce the same result every time, which is harder than it sounds once real network calls or shared state are involved.
Low setup cost — a sandbox that takes a day to configure per service doesn't get maintained, and an unmaintained sandbox is worse than no sandbox, because it gives false confidence.

With that as the bar, here's how the commonly used tools stack up.

Container-based sandboxes: Docker Compose, Testcontainers

Testcontainers and plain Docker Compose setups are probably the most widely used approach for sandboxing infrastructure dependencies — spinning up real instances of a database, message queue, or cache in a container, scoped to a test run, then tearing it down. This gets isolation and reproducibility right, since you're running the actual software, not a simulation of it.

Where this approach is weaker: it sandboxes infrastructure you control, but doesn't help much with third-party APIs or internal services you can't just spin up in a container — a payment provider, an external auth service, another team's API you don't own.

API mocking/simulation platforms: WireMock, Mockoon

For external APIs, WireMock and Mockoon are common choices — they let you define stub responses for HTTP endpoints and run them as a lightweight standalone sandbox server. Strong on setup simplicity and control; you can hand-craft exactly the responses and error cases you want to test against.

The weakness is the same one hand-written mocks always have: someone has to define every scenario manually, and nothing keeps those definitions in sync as the real API changes. A WireMock sandbox is only as realistic as the person who last updated it remembered to make it.

Contract-based sandboxes: Pact

Pact takes a different angle — instead of manually defining sandbox responses, it generates them from contracts agreed between a service and its consumers. This solves part of the drift problem, since a contract change is at least visible and versioned, but it depends on both sides maintaining accurate contracts, which is its own discipline that not every team keeps up consistently.

Traffic-capture-based sandboxes: Keploy

Keploy takes an approach closer to the "freshness" problem directly: instead of hand-defining sandbox behavior or relying on maintained contracts, it captures real API and database traffic at the network layer using eBPF and turns that captured traffic into a sandbox environment, along with test cases and mocks generated from it.

The practical difference this makes: the sandbox reflects what a dependency actually did, including real error responses, real latency characteristics, and edge cases that occurred in practice rather than ones someone thought to configure. Because capture happens at the network layer, it also works across different languages and frameworks without requiring code instrumentation in the service being tested, which matters for teams with a mixed-language stack where writing per-language stub definitions gets expensive to maintain consistently.

This doesn't make it strictly better in every situation — a Testcontainers-based sandbox is still the right call for infrastructure you control and want to run for real rather than simulate, and a hand-crafted WireMock stub is still faster to set up for a single, simple, rarely-changing endpoint. Where the traffic-capture approach earns its place is specifically the freshness and coverage problem: sandboxes that would otherwise need constant manual upkeep to stay accurate, especially across a distributed system with a lot of interdependent services.

Choosing between them

A rough way to think about it: use container-based sandboxes for infrastructure you own and can run directly. Use hand-defined mocking tools for external dependencies with a small, stable set of scenarios you're comfortable maintaining by hand. Use contract-based tools where you have real cross-team discipline around API contracts already. And where the actual problem is a sandbox drifting out of sync with a dependency's real behavior over time, especially across several services, a traffic-capture-based approach is worth prioritizing over one that depends on someone remembering to keep definitions updated.

None of these are strictly better than the others across every situation. The mistake worth avoiding is the one from that payment sandbox story: picking an approach once, without revisiting whether it's actually still matching reality six months later.

Dependency Mocking vs Service Virtualization: What Actually Replaces Hand-Written Mocks in Integration Tests

Marx Jenes — Thu, 10 Sep 2026 12:46:30 +0000

Somewhere around our third microservice, our hand-written mocks stopped being an asset and started being a liability nobody wanted to own. Every mock lived in its own file, written by whoever needed it at the time, matching whatever the dependency returned on the day it was written. Six months later, a payments service changed its error response shape, and every mock that simulated it kept returning the old shape forever, because nothing told the mocks the world had moved on. Our tests stayed green. Production did not.

That's the point where most teams start asking whether there's something better than hand-writing mocks one file at a time, and the two answers that usually come up, dependency mocking (tooling) and service virtualization, sound similar enough that people use them interchangeably. They're not the same thing, and picking the wrong one for your situation costs you real time.

What hand-written mocks actually cost you

Before comparing tools, it's worth being honest about why hand-written mocks break down at scale:

They drift. A mock is a snapshot of a dependency's behavior at the moment someone wrote it. Nothing keeps it in sync as the real dependency changes.
They're incomplete by construction. Someone writes a mock for the cases they thought to handle - success, one or two errors. Real dependencies fail in ways nobody imagined until it happens in production.
They don't scale with team size. Each new mock is a new maintenance surface. Multiply that across a team writing dozens of integration tests a week and you get an unmanaged pile of stale fixtures nobody trusts.
They're disconnected from real traffic. A mock represents what someone thinks a dependency does, not what it actually does across the range of real requests hitting it in production.

Both dependency mocking and service virtualization tools exist to fix some version of this, but they fix different parts of it.

Dependency mocking (as a tooling category)

Dependency mocking tools automate the creation and maintenance of mocks for individual dependencies - a specific API, a database call, a message queue. Instead of hand-writing a mock object, you use a library or framework that intercepts calls to a dependency and returns configured or recorded responses.

The better tools in this category address the drift problem directly by generating mocks from observed behavior rather than someone's manual guess - recording a dependency's actual responses and replaying them, rather than hand-typing what someone assumes the response looks like. This is a meaningfully different approach from classic hand-rolled mocking, even though both get called "mocking."

Where this fits best: testing one service's interaction with one or a few specific dependencies, in unit or narrow integration tests, where you want fast, deterministic, isolated tests and don't need a full simulated environment.

Where it falls short: it typically operates one dependency at a time. It doesn't give you a full simulated environment where multiple interdependent services behave consistently with each other - which is exactly the situation distributed systems testing runs into.

A few tools commonly used in this category, each taking a different approach to generating or managing mocks: WireMock (HTTP-level stubbing, widely used in the Java ecosystem), Mockoon (a desktop app for designing and running mock APIs manually), Pact (contract-testing focused, mocks generated from consumer-defined contracts rather than recorded traffic), and Keploy (generates mocks automatically from real API and database traffic captured at the network layer via eBPF, rather than hand-defined stubs or contracts).

Service virtualization

Service virtualization operates at a different scope: instead of mocking individual calls, it simulates entire dependent systems or environments - a whole downstream service, or a cluster of them, behaving consistently as a unit, the way the real system would across a full transaction.

This matters most in distributed systems, where a single test scenario might touch five or six services, some of which are unavailable in a test environment, rate-limited, expensive to call repeatedly, or owned by another team entirely. Service virtualization gives you a realistic stand-in for that whole dependency graph, not just one endpoint.

The stronger implementations of this approach are also built from recorded, real traffic rather than someone's hand-built simulation - capturing the actual request/response behavior of a dependency (or a whole set of them) and replaying it faithfully, including realistic latency, error rates, and edge cases that were never intentionally designed into a hand-written mock because nobody thought to.

On the tooling side, this space includes established enterprise platforms like Broadcom's Service Virtualization (formerly CA Service Virtualization) and Parasoft Virtualize, both of which let teams build simulated environments manually or from recorded interactions, and Hoverfly, an open-source option for capturing and simulating HTTP service behavior. Keploy also applies here on the recorded-traffic side specifically - since it captures real inter-service traffic via eBPF, it can generate the kind of realistic, environment-wide behavior simulation service virtualization is meant to provide, without needing a separately maintained virtual environment built by hand.

Where record-and-replay changes the comparison

The most useful development across both categories, in practice, is the shift away from hand-authored simulation entirely and toward capturing real traffic and replaying it as test infrastructure. This addresses the core weaknesses of manual mocking directly:

Drift stops being a maintenance burden when mocks are regenerated from actual captured traffic rather than manually updated by a person who has to remember to do it.
Coverage improves because recorded traffic includes the edge cases real usage produces - malformed payloads, unusual timing, rare error responses - not just the cases a person thought to write by hand.
CI determinism is preserved because replay is exact: the same recorded interaction plays back the same way every run, without depending on a live, possibly-flaky external dependency actually being reachable during a test.

This is the part of the space Keploy specifically focuses on: capturing real API traffic and database interactions directly at the network layer using eBPF, then generating both mocks and full integration/regression test cases from that captured traffic automatically, rather than requiring either hand-written mock objects or a separately maintained virtualization environment. Because the capture happens at the network layer, it works across languages and frameworks without instrumenting application code, and because the mocks are generated from what a dependency actually returned, they don't carry the "someone's best guess" problem that hand-written mocks always have.

The practical effect is that it blurs the line between the two categories in this comparison: it produces artifacts that function like dependency mocks (deterministic, fast, scoped per test) but are generated the way good service virtualization is built (from real recorded system behavior, not manual authorship), which is a meaningfully different position than tools that only do one half of that.

A rough way to choose

If you're testing a single service's handling of a small number of specific dependencies, and you mostly need fast, deterministic, isolated tests - dependency mocking tooling is usually the right scope, especially if it supports generating mocks from recorded traffic rather than manual definitions.

If you're testing across a distributed system where multiple services need to behave consistently together, and hand-maintaining that many mocks isn't realistic - you're in service virtualization territory, and the traffic-capture-based approaches are worth prioritizing over ones that require you to hand-build the virtual environment yourself.

Either way, the actual upgrade from hand-written mocks isn't really "mocking tool vs virtualization tool" as a binary choice. It's moving away from manually authored simulation of any kind and toward tests grounded in what your dependencies actually do, captured from real behavior instead of guessed at a keyboard.

Software Testing Basics: The One Habit That Matters More Than Any Tool

Marx Jenes — Thu, 27 Aug 2026 12:10:48 +0000

A junior dev I mentored a while back asked me which testing framework he should learn first. Reasonable question. I told him it didn't matter much yet, and he looked at me like I'd dodged it. I hadn't. The honest answer is that the framework was the least important thing standing between him and writing tests that actually caught bugs.

The thing that actually mattered - the one habit I've seen separate people who write tests that catch real problems from people who write tests that just exist - is stupidly simple: write down what you expect to happen before you write the test that checks it. Not "test the login function." What, specifically, should happen when the password is wrong. What should happen when the account doesn't exist. What should happen when someone hits submit twice.

Why this sounds too obvious to matter

Most people already think they do this. They don't, not really. What actually happens most of the time is: write the code, run it, see what it does, write a test that confirms it does that. That's not testing. That's documentation of current behavior, and current behavior includes your bugs.

The habit is inverted from that. You write down the expected result before you look at what the code actually does, ideally before the code is even fully written. It feels slower. It's the entire difference between a test suite that catches regressions and one that just quietly agrees with whatever's already there.

Where I've seen this go wrong

Early in my career I tested a password reset flow by running it, watching it work, and writing an assertion that matched what I saw. Six months later someone changed an error message and my test broke - correctly, technically, since the output changed, but I couldn't tell if the new behavior was fine or actually wrong, because I'd never written down what "right" was supposed to look like. I'd only ever recorded what "current" looked like. Three of us argued about it in a PR review for twenty minutes before someone just asked product what the message was supposed to say in the first place, which is the question I should have answered before writing the test at all.

This is the same failure mode as confusing verification and validation - checking that code matches what it currently does isn't the same as checking that it does what it's actually supposed to do. The habit of writing the expected result down first is what forces that distinction to actually show up, instead of staying invisible until something breaks in a confusing way.

What this looks like in practice

Before writing a test, three quick questions, in this order:

What should happen here, in plain language, no code? ("Login fails with a clear error and no session is created" - not "returns 401.")
What's the one thing that would make this wrong even if it technically ran without crashing? (A silent failure. A vague error. Data left in a bad state.)
Would someone unfamiliar with this code understand what failed and why, just from the test name and assertion?

If you can't answer #1 without looking at the code first, that's the actual signal something's off - not the framework, not the tooling, not the coverage number.

Why this matters more than tool choice

Frameworks, runners, assertion libraries - these are genuinely interchangeable in a way that surprises people early on. Jest and Vitest and pytest all do fundamentally the same job with different syntax. Nobody's test suite got meaningfully better because they switched frameworks. Suites get better when the people writing them get sharper about defining "correct" before checking it.

This is also, not coincidentally, the actual foundation under most of what gets called software testing basics- the types of testing, the levels, the terminology. All of it assumes you already know what you're checking for. None of the vocabulary helps if that part's fuzzy. I found Keploy's rundown of the fundamentals genuinely useful for tying that vocabulary together in one place, if you want the fuller map of levels, types, and lifecycle beyond just this one habit.

Where it pays off later

This habit compounds in a way tool choice never does. A test suite where every test was written against a clearly stated expectation is a suite people trust - when something turns red, the team knows immediately whether it's a real regression or an intentional behavior change, instead of spending twenty minutes in a PR thread trying to reconstruct what "right" was supposed to mean.

Learn a framework whenever you need to. Learn to write the expected result down first, before you touch the code, and you'll write better tests in whatever framework you're handed next.

Verification vs Validation in Software Testing: A Practical Breakdown

Marx Jenes — Thu, 20 Aug 2026 08:06:12 +0000

A few years back I worked on a reporting feature that hit every acceptance criterion in the ticket. Every field matched the spec, every calculation was correct according to the documented formula, every edge case in the ticket was handled. QA signed off. Then it shipped, and the first support ticket came in within a day: the numbers were "technically right" but useless, because the spec itself had been written against an old version of how the business actually calculated that metric. Nobody had checked with finance before writing the ticket.

That's verification and validation in one story. We verified the software matched the spec perfectly. Nobody validated that the spec matched what the business actually needed. Both matter, they're not interchangeable, and mixing them up is one of the more expensive mistakes a team can make without realizing it's happening.

Verification: are we building it right?

Verification asks whether the software conforms to its specification, requirements, or design documents. It's internal-facing - you're checking the product against what was written down, not against what anyone actually wants.

This covers most of what people think of as "testing" day to day:

Code reviews checking implementation against a design doc
Unit and integration tests checking behavior against documented requirements
Static analysis, linting, spec-conformance checks
QA walking through a ticket's acceptance criteria one by one

Verification is good at catching a specific kind of bug: the gap between what was specified and what got built. It's terrible at catching a different kind of bug: the gap between what was specified and what should have been specified in the first place. My reporting story is a verification success and a validation failure, and verification alone had no way of flagging that.

Validation: are we building the right thing?

Validation asks whether the software actually satisfies the real-world need it's meant to serve, regardless of what the spec says. It's external-facing - checking against reality, users, and business intent, not against a document.

This looks more like:

User acceptance testing with actual users, not just QA reading a checklist
Beta releases and real usage data
Product and stakeholder review against the actual problem being solved, not just the ticket
Watching what happens when real traffic hits the thing, not just the scenarios someone thought to write down

Validation is where "it passed every test" and "it's ready to ship" turn out to be different sentences. A feature can be verified into oblivion - every acceptance criterion green, and still fail validation the moment a real user tries to do something the spec author never considered.

Why the mix-up is expensive, not just academic

Teams that only verify tend to over-trust their test suite. A hundred passing tests feels like safety, but if every one of those tests was written against a spec that had a gap, you've built a very well-tested version of the wrong thing. The bug isn't in the code. It's upstream, in the requirements, and no amount of additional unit testing against those same requirements will ever catch it.

This is also why "we have great test coverage" and "our users are happy" aren't the same claim, and teams get bitten when they treat the first as proof of the second. Coverage measures verification. It says nothing about validation.

The verification toolbox, and what each piece actually catches

It's worth breaking "verification" down further, because the different techniques catch genuinely different classes of mismatch:

Static verification happens before anything runs: code reviews, design reviews, static analysis, linting, type checking. These catch structural problems - a function that doesn't match its documented contract, a data flow that violates an assumption elsewhere in the system, a design that technically satisfies a requirement but contradicts another one nobody cross-checked. Cheap to run, cheap to fix at this stage, and the earlier a mismatch is caught here, the less it costs later.

Dynamic verification is everything that runs the code against expected behavior: unit tests, integration tests, contract tests, regression suites. This is where most engineering time goes, and for good reason - it's fast, repeatable, and automatable in a way static review isn't. But it's bounded by the same limitation every time: it can only check the code against what someone wrote down as "expected." A dynamic test suite with high branch coverage against an incomplete spec is still just thoroughly verifying the wrong thing.

Formal verification - proving a program correct against a formal specification using mathematical methods, sits at the far end and is mostly reserved for places where a bug is catastrophic: aerospace, medical devices, cryptographic protocols, safety-critical embedded systems. Most teams will never need it, but it's worth knowing it exists, because it makes the boundary clearer: even a mathematically proven-correct program is only proven correct relative to its spec. If the spec encodes the wrong requirement, formal verification will prove the wrong thing flawlessly.

The pattern across all three: verification techniques get more rigorous as you move from static review to formal proof, and none of them, at any level of rigor, can tell you whether the specification itself reflects reality. That gap only closes through validation.

The validation side, and why it resists automation

Validation techniques look different because they're checking against something inherently fuzzier than a spec:

User acceptance testing (UAT) puts the software in front of actual users or close proxies (product owners, domain experts) and asks whether it does what they need - not whether it matches a document. UAT frequently surfaces requirements gaps that were invisible during verification, because the people testing aren't checking a checklist, they're trying to accomplish something.

Beta programs and staged rollouts extend this to real usage at scale, which matters because individual UAT sessions still can't replicate the variety of how a broad user base actually behaves. A feature can sail through UAT with five friendly testers and still collapse against the messiness of real users doing things nobody anticipated.

Production monitoring and incident analysis is validation that never stops. Every bug report, every confused support ticket, every unexpected usage pattern is a live signal about whether the system satisfies the real need - arguably a more honest signal than any pre-release testing phase, because it's coming from reality instead of a simulation of it.

Domain and stakeholder review - going back to the people who understand the actual problem, not just the ticket, and asking whether this solves what they actually needed - is the least technical and most frequently skipped form of validation, and it's often the one that would have caught my reporting-feature story before it reached a user.

The common thread: validation techniques resist full automation because they're checking against a moving, real-world target rather than a fixed document. You can automate a regression suite. You can't fully automate whether something still makes sense to the humans using it, and teams that try to substitute more verification for that keep rediscovering the gap the hard way.

A rough way to think about where each one belongs

Verification tends to live close to the code - the earlier and more automatable, the better, because it's checking against something explicit and stable (a spec, a contract, a set of requirements). This is where automated test suites earn their keep: fast, repeatable checks that the implementation still matches what was agreed on.

Validation tends to need something verification can't provide on its own: exposure to reality. Real users, real traffic, real edge cases nobody thought to write into a ticket. This is also where a lot of teams under-invest, because it's harder to automate and harder to feel "done" with - there's no green checkmark for "definitely building the right thing," only ongoing signal from actual usage.

Verification and validation in software testing, does not substitute each other. A team that's excellent at verification and weak at validation ships well-tested products nobody needed exactly as specified. A team that's strong on validation but sloppy on verification ships the right idea built on a shaky, bug-prone foundation. The healthiest setups treat them as two different questions, asked continuously, rather than one "testing" phase that's supposed to cover both.

Next time a feature passes every test and still gets a confused reaction from users, it's worth asking which of the two actually failed. Usually it's not the tests.

How to Build a First Test Suite From Scratch for a New Project?

Marx Jenes — Wed, 12 Aug 2026 06:24:22 +0000

The worst test suite I ever inherited had 400 tests, and I trusted about six of them. The rest were either testing implementation details nobody cared about, duplicating each other, or so tightly coupled to internal function names that a harmless refactor broke thirty tests for no real reason. Reading that codebase taught me more about what not to do than any greenfield project ever has.

So when you're starting from zero, the goal isn't "write a lot of tests fast." It's building a suite you'll still trust a year from now. If you're new to this, getting the software testing basics right early matters more than covering everything - learning how to build a first test suite from scratch teaches you what to prioritize in a way that inheriting someone else's bloated suite never will. Here's roughly how I'd approach it.

Start with what would actually hurt if it broke

Before writing a single test, list the handful of things that would be genuinely bad if they silently broke - checkout completing, auth working, the core thing your product does actually happening. Not every function, not every branch. Just the stuff where a silent failure costs you money, users, or trust.

This list is usually shorter than people expect. Five to ten flows for most early-stage products. That's your actual test suite's job in the first few months, not "100% coverage."

Unit tests for logic, not for plumbing

Unit tests are for things with actual decision-making in them - pricing calculations, validation rules, state transitions, anything where "given this input, is the output correct" is a real question with a wrong answer possible. They're fast, they're cheap, and they should make up the bulk of your suite.

Skip unit-testing pure plumbing: a function that just calls another function and returns its result doesn't need its own test. That's the kind of test that pads a coverage number without catching anything real, and it's exactly the kind of test that made that 400-test suite so hard to trust.

Integration tests for the seams

This is where most early suites are too thin. Unit tests tell you your pricing function is correct in isolation. They don't tell you the checkout flow actually calls it correctly, passes the right currency, or handles what happens when the payment provider returns an error you didn't expect.

The seams between your code and everything external - your database, third-party APIs, other internal services are where bugs actually like to hide, because that's where two sets of assumptions meet and don't always agree. This is also the hardest category to get right early on, because hand-writing every plausible external response (success, timeout, malformed payload, rate limit) is a lot of manual work, and most teams just don't do it, which is part of why integration bugs are so common in production even in codebases with decent unit coverage. Tools that can record real API traffic and replay it as test cases - Keploy is one - exist specifically to close that gap without hand-writing every scenario, which is worth knowing about before you spend a week writing mocks by hand.

Don't test the framework

A depressing amount of early test-writing time goes into testing things the framework already guarantees - does this React component render, does this route return a 200. If your framework is reasonably mature, trust it. Test your logic, not its plumbing.

Make flaky tests a same-week problem, not a someday problem

The first flaky test in a new suite is a fork in the road. Fix it now, while the codebase is small enough that the cause is easy to find, or let it slide and watch the team's trust in the whole suite erode as more join it. I've written about this before - flaky tests are almost never actually about the test itself, they're usually pointing at shared state, a race condition, or an environment assumption nobody wrote down.

Let the suite grow with real usage, not just imagination

Whatever you write on day one will cover the bugs you could imagine. It won't cover the malformed payload from a partner's outdated client, or the auth token format some old integration still sends. As the project gets real traffic, treat production incidents as test-suite input - every bug that reaches production and wasn't caught is a gap in the suite, and turning it into a regression test is the cheapest way that gap ever gets closed.

The suite you actually want in six months

Not the biggest one. The one where a red result still means something, where the team fixes failures instead of quietly muting them, and where adding a new feature doesn't mean an afternoon fighting tests that were testing the wrong thing to begin with. Start small, aim it at what actually matters, and let it grow from real usage rather than a coverage target.

What AI Actually Helps With in Test Automation

Marx Jenes — Mon, 10 Aug 2026 04:30:11 +0000

What AI Actually Helps With in Test Automation (and Where It Falls Apart)

A teammate spent an afternoon last month feeding our checkout flow into an AI coding assistant and asking it to generate a test suite. Twenty minutes later he had forty tests. Good coverage on paper, clean syntax, decent naming. Then we ran them against a known bug we'd fixed the week before, and every single test passed anyway. The AI had written tests that matched what the code currently did, not what the code was supposed to do. It had no way of knowing the difference.

That's the honest version of where AI sits in test automation right now, and it's a lot less dramatic than most of what gets written about it.

Where it's genuinely useful

Scaffolding, fast. Writing the boilerplate for a new test file, setting up fixtures, wiring up assertions for a straightforward CRUD endpoint - this is exactly the kind of repetitive, low-judgment work AI is good at. It saves real time, especially early in a project when you're writing a lot of similar tests.

Spotting edge cases you didn't think of. Describe a function's inputs and ask what could break it, and you'll usually get a decent list back: empty strings, null values, boundary numbers, unicode weirdness. Not exhaustive, and not always relevant to your actual domain, but a reasonable starting checklist that's faster than staring at a blank test file.

Explaining existing test failures. Pointing an AI assistant at a stack trace and a failing assertion and asking "what's likely going on here" is one of the more underrated uses. It's fast at pattern-matching common failure modes - off-by-one errors, type mismatches, obvious null pointer issues, even if it can't always diagnose the deeper cause.

Refactoring test code itself. Cleaning up duplicated setup logic, converting a pile of copy-pasted tests into a parameterized suite, modernizing old assertion syntax - mechanical work that AI handles well because it's about the shape of the code, not what the code is supposed to prove.

Where it falls apart

It doesn't know what "correct" means for your system. This is the checkout example above. AI can write a test that passes against your current code, but it has no independent way of knowing whether your current code is right. It's not testing against your business logic, it's testing against whatever's already there - which means it's very good at locking in bugs, not just catching them.

It doesn't know your actual traffic patterns. A generated test suite tends to cover the inputs a human (or an AI) can imagine, not the inputs your system actually receives. Real API traffic is messier than anyone's imagination: malformed payloads from a partner's outdated client, an auth token format from three integrations ago that somehow still works, timing patterns nobody would think to write by hand. Software test automation built entirely from imagined scenarios misses a category of bug that only shows up under real usage.

Maintenance is still a human problem. AI can generate forty tests in twenty minutes. It doesn't show up six months later when the API contract changes and half of them need updating. Automated testing has always had a maintenance cost, and generating tests faster just means you're generating maintenance debt faster too, unless something is tracking how the underlying system actually behaves over time and flagging what's drifted.

It can't tell you what to test, only how. The genuinely hard part of test automation was never syntax. It's deciding what matters: which flows are business-critical, which failure modes are acceptable, where the risk actually lives. That's a judgment call rooted in what your product does and who depends on it. No amount of prompting hands that over.

Where this is heading

The realistic near-term picture isn't "AI writes your test suite," it's AI doing more of the mechanical work - scaffolding, edge-case suggestions, failure triage - while the actual test strategy, the ground-truth of what correct behavior looks like, and the ongoing maintenance stay firmly a team's responsibility. The tools that will matter most in this space long-term probably aren't the ones that generate the most tests fastest, but the ones that keep tests grounded in what a system genuinely does in production, rather than what someone (human or AI) guessed it should do.

Worth remembering next time a demo makes AI-driven test automation look like a solved problem: the demo is testing against a scenario someone already knew the answer to. Production isn't that generous.

Why Flaky Tests Are Rarely About the Test

Marx Jenes — Thu, 06 Aug 2026 12:42:52 +0000

We had a checkout test at my last job that everyone called "the coin flip." Green for a week, red twice on a Tuesday, green again. Someone eventually wrapped it in a retry and it sat like that for eight months before anyone looked at it again. Turned out the real bug was a webhook that occasionally fired before the order record finished writing to the DB - a two-hundred-millisecond gap that only showed up under load. The test wasn't broken. It was the only thing in the entire pipeline that noticed.

That's usually the story. Someone blames the test - bad selector, missing wait, a sleep(2) some intern left in there three years ago, and half the time they're right. But when a test flakes repeatedly and nobody can explain why, the test is rarely the actual problem. It's just the part of the system rude enough to say something.

A few places I keep finding the real cause hiding.

Tests that quietly depend on each other

Test A writes a row, Test B reads it and never knew it needed to. Run B by itself, it passes. Run the suite in a different order, or in parallel, and B fails for no reason anyone can point to. I've lost a full afternoon to this exact thing more than once - a cache value from Test 12 leaking into Test 47.

The actual fix is annoying and unglamorous: every test gets its own fixtures, its own scoped data, no assumptions about what ran before it. If your suite only goes green in one specific order, you don't have a flaky test. You have an undocumented dependency graph, and it's going to bite someone eventually.

The app is racing, not the test

Click a button, immediately assert on the result - that's a bet that the UI update lands the instant the click handler returns. It usually does, on your machine, on a good day. Add a debounce, a background job, or just enough network latency and that bet stops paying off.

This one's frustrating because the test isn't being paranoid. The app genuinely has a race condition. The test just runs the interaction often enough, across enough machines, that it eventually catches the app mid-race - something a human clicking through the same flow once or twice would probably never notice.

Clocks, timezones, and CI boxes that aren't your laptop

Date.now() and anything DST-adjacent will happily pass for months and then fail on one specific day of the year, or the moment your CI runner's timezone doesn't match what the original author assumed (usually their own laptop, usually not documented anywhere). Same idea with tests that were written assuming reasonably fast hardware - throw them on a throttled CI box and timing assumptions that never mattered locally suddenly do.

Mocks that stopped matching reality

This is the sneaky one, because it doesn't look like flakiness at all - it looks like a test that's passing. Someone mocks a third-party API, gets it right on day one, and then the real API changes: a field gets renamed, an error shape changes, rate limiting gets added. The mock has no idea. It keeps returning exactly what it always returned, the test stays green, and production quietly starts failing in a way nothing in CI can see.

If anything this is worse than a red test, because it erodes trust in the wrong direction - teams end up more confident in a mock-backed test than they should be, right up until it fails somewhere that actually matters.

The twenty-minute question

It's easy to treat a flaky test as noise to manage - quarantine it, slap a retry on it, mute it and move on. But almost every one of them is pointing at something real: state nobody's tracking, a race condition in the app itself, an environment assumption that was never written down, or a mock that fell out of sync with the thing it's supposed to represent.

Muting it doesn't make any of that go away. It just makes the pipeline quieter while whatever's actually wrong keeps happening, unwatched, in production.

Next time a test earns a reputation, it's worth the twenty minutes to ask why before reaching for a retry decorator. Most of the time it wasn't lying to you.