DEV Community: Marx Jenes

Parasoft Virtualize vs Modern Alternatives: How Service Virtualization Has Changed

Marx Jenes — Mon, 05 Oct 2026 13:02:00 +0000

A friend who's maintained a Parasoft Virtualize setup for the better part of a decade put it well: it was built for a world where a release happened every quarter, a dependency map fit on one slide, and the person configuring a virtual service was often a dedicated specialist with the time to do it properly. That world still exists in plenty of large enterprises, and Parasoft still serves it well. But a lot of teams evaluating service virtualization today aren't in that world anymore, and the gap between what they need and what the original generation of virtualization tooling assumed is exactly why parasoft alternatives for service virtualization has become such a common query.

Worth being specific about what actually changed, rather than treating this as a simple old-vs-new story.

What Parasoft Virtualize was built to solve, and still solves

Parasoft Virtualize is a mature, enterprise-grade platform for simulating dependent systems - APIs, databases, mainframes, message queues - in environments where those dependencies are expensive, unavailable, or risky to call directly. It supports a wide range of protocols beyond plain REST, which matters a lot to organizations running legacy systems alongside modern services, and it gives teams fine-grained manual control over exactly how a virtual service behaves, which some regulated or highly specific testing scenarios genuinely require.

Its strengths are real: broad protocol support, mature tooling for complex stateful simulations, and a track record in large, compliance-heavy environments where that maturity matters more than setup speed. None of what follows is an argument that it's a bad tool. It's an argument that it was designed for a specific shape of team and workflow, and a lot of teams today don't have that shape anymore.

What changed underneath it

Three shifts explain most of why teams go looking for alternatives:

Release cadence. Quarterly releases gave teams time to hand-configure a virtual service carefully and let it live for months before anything needed to change. Teams shipping multiple times a week don't have that slack - a virtual service that takes a day to update becomes a bottleneck, not a convenience.

Service count. A dependency map that used to fit on a slide now often spans dozens of microservices, each with its own API surface. Manually defining and maintaining that many virtual services, by hand, in a GUI-driven workflow, becomes its own full-time job at a certain scale.

Who's doing the configuring. The earlier generation of tooling assumed a dedicated specialist, often in a central QA or environments team. Modern teams increasingly expect individual engineers to own their own service's test environment, without needing to learn a separate platform's configuration language to do it.

None of this makes the older model wrong. It makes it a worse fit for teams whose release cadence, service count, and ownership model look different from what it was designed around.

The alternatives, and what each one changes

WireMock is a common lightweight starting point - open source, HTTP-focused, and simple enough that an individual engineer can stand up a stub server without needing a platform. It trades Parasoft's broad protocol coverage and enterprise tooling for speed and developer-level control, which suits teams whose dependencies are mostly REST APIs and who want virtualization to live close to the code rather than in a separate system.

Keploy addresses the maintenance side of the problem directly. Instead of manually defining virtual service behavior, it captures real API and database traffic at the network layer using eBPF and generates mocks and test cases from that captured traffic automatically. Because capture happens at the network layer, it works across different languages and frameworks without instrumenting application code, which matters for teams running a genuinely mixed stack. The practical effect is that updating a virtual service becomes a matter of recording traffic again rather than someone revisiting a configuration by hand, which is the specific pain point teams with frequent releases and many services tend to hit hardest with older tooling.

Hoverfly is another open-source option focused on HTTP service simulation, with support for capturing and replaying real traffic as well as manually defined scenarios. It sits between WireMock's simplicity and a fuller platform's feature set, and is often chosen by teams that want capture-based behavior without committing to a broader tool.

Testcontainers, while not a virtualization tool in the strict sense, is worth mentioning because it solves a related problem differently - running real instances of infrastructure dependencies (databases, queues) in containers rather than simulating them at all. For dependencies a team actually controls, this sidesteps the simulation-fidelity question entirely by using the real thing, scoped to a test run.

How to actually choose

The honest answer depends more on your situation than on which tool is "better" in the abstract:

Heavy legacy/mainframe protocol needs, regulated environment, dedicated environments team - Parasoft's maturity and protocol breadth are hard to replace with newer tooling.
Mostly REST APIs, small number of stable dependencies, want fast manual control - WireMock or Hoverfly fit well without much overhead.
Many services, frequent releases, a recurring problem with virtual services going stale between updates - a capture-based approach like Keploy addresses the maintenance burden specifically, rather than just making manual configuration faster.
Dependencies you actually own and can run in CI - Testcontainers sidesteps virtualization by using the real thing.

The actual shift

The real story isn't that newer tools are simply better than Parasoft Virtualize. It's that service virtualization has split into different tools optimized for different constraints - protocol breadth and regulatory maturity on one end, developer-level speed and low maintenance overhead on the other, with traffic-capture approaches specifically targeting the staleness problem that hand-configured virtual services have always had. Which one fits depends on how often your dependencies change, how many of them there are, and who's actually responsible for keeping the simulation honest.

What Makes an API Mock Trustworthy

Marx Jenes — Wed, 30 Sep 2026 08:49:47 +0000

A mock earns its place in a test suite the same way any other piece of infrastructure does: by being right often enough that nobody thinks to question it. That's also exactly what makes a bad mock dangerous. Nobody double-checks the thing they've stopped noticing. A mock that quietly stopped matching reality six months ago doesn't announce itself. It just keeps returning green, right up until the gap it's been hiding shows up somewhere much more expensive than a test run.

Worth being specific about what separates a mock that's actually doing its job from one that's just occupying the space where a real dependency test should be.

It reflects what the dependency actually does, not what someone assumed it does

The most common origin story for a mock is someone reading documentation, or remembering roughly how an API behaves, and writing a response that seems reasonable. That's a fine starting point, and it's also the exact spot where trustworthiness starts to erode, because documentation and memory both drift from reality in ways nobody notices until something breaks.

A trustworthy mock is built from observed behavior: an actual response the dependency actually returned, at some point, under real conditions. This matters more than it sounds, because real APIs are full of small inconsistencies that no one writes into a spec — a field that's sometimes null and sometimes just absent, a timestamp format that varies by endpoint, an error response shaped differently than the success response's documentation implies. A mock built from what actually happened carries all of that. A mock built from what someone assumed happened carries none of it.

It includes failure, not just success

Most hand-built mocks are optimized for the case that's easiest to imagine: the request that works. That's understandable, since the success case is usually what the person writing the mock was actually trying to test in the first place. It's also the least useful part of a mock to get right, because the success case is rarely where production bugs live.

The valuable part of a mock is what happens when the dependency times out, returns a malformed payload, hits a rate limit, or fails in some specific way particular to that service. These are the cases most likely to be missing from a mock, precisely because they require someone to think of them in advance and deliberately write them in. A mock's trustworthiness is really a question of how much of the dependency's actual behavior it captures, and failure modes are usually the majority of that behavior that goes uncaptured.

It doesn't quietly go stale

A mock is a snapshot. The moment it's written, it starts drifting away from whatever the real dependency is doing, at a pace nobody's tracking. This is the failure mode that's hardest to notice, because a stale mock doesn't look broken. It looks exactly like it did the day it was written. Tests built on it keep passing, confidence keeps building, and the actual dependency keeps changing underneath, unmonitored.

The trustworthy version of this isn't "write a good mock once." It's having a clear answer to who refreshes it, how often, and what would actually trigger that refresh - a scheduled re-check, a contract version bump, a captured-traffic pipeline that updates automatically rather than depending on someone remembering. Mocks that get treated as permanent fixtures are the ones most likely to be silently wrong by the time anyone looks at them again.

It's scoped to what it's actually simulating

A trustworthy mock does one job clearly: standing in for a specific dependency's specific behavior. Mocks lose credibility when they start absorbing logic that belongs somewhere else - business rules baked into a mock's response, conditional behavior that exists only to make a particular test pass, special cases added over time by different people for different reasons until nobody's sure what the mock is actually simulating anymore.

The cleanest mocks are the ones that map directly to something a real dependency did, with nothing added and nothing simplified away. The moment a mock needs its own internal comments explaining why it behaves the way it does, that's usually a sign it's drifted from representing the dependency into representing whatever made a specific test pass at some point.

It's reproducible

A mock that behaves differently between runs, whether from race conditions, uncontrolled randomness, or hidden state carried over from a previous test, undermines the entire reason mocks exist. The whole premise of replacing a real dependency with a mock is predictability: the same input should produce the same output, every time, regardless of what ran before it or what environment it's running in. A flaky mock is arguably worse than no mock at all, because it produces the appearance of determinism while quietly not providing it, and debugging a flaky test caused by a flaky mock is a particularly frustrating way to lose an afternoon.

Where this actually comes from in practice

Most of these properties point in the same direction: a mock's trustworthiness comes from how closely it's tied to real, observed behavior, and how deliberately that connection is maintained over time. Hand-written mocks can achieve all of this, but only with real ongoing discipline, since every property above requires someone to keep paying attention after the mock is first written.

This is the specific gap that dependency mocking as a tooling category exists to close - generating and maintaining mocks from real recorded behavior instead of leaving that entirely to manual upkeep, so freshness and failure-case coverage come from the process itself rather than from whether someone remembered to go back and update a stub file.

The actual test

A simple way to check whether a mock in your suite still deserves trust: could you explain, right now, where its behavior came from and how recently that source was checked. If the honest answer is "someone wrote it a while ago based on the docs," that's not automatically wrong, but it's worth treating as a question rather than an assumption. The mocks worth trusting are the ones where that question has a specific, recent, confident answer.

Testing Against a Dependency You Can't Call: When Service Virtualization Is the Only Option

Marx Jenes — Mon, 28 Sep 2026 05:46:06 +0000

A team I know integrated with a partner's inventory system that had exactly one environment: production. No staging, no sandbox, no test credentials. Their only way to "test" the integration was to send real requests to a live system that held real stock counts for real customers. For the first few months they did what most teams do in that spot. They wrote a mock by reading the partner's PDF documentation, and they trusted it. Then the partner returned a response with a field the PDF never mentioned, and an order flow that had been green in CI for months failed on its first live day.

Most service virtualization content treats it as an optimization, a faster or tidier way to run tests you could technically run against the real thing. This piece is about the other situations, the ones where calling the real dependency isn't merely inconvenient. It's off the table.

The situations where you genuinely can't call it

The dependency charges per call. Some third-party APIs bill for every request: credit checks, identity verification, geolocation lookups, SMS delivery. A regression suite that runs a few hundred times a day against one of these turns testing into a line item on the budget.

There is no test environment. Partner systems, government services, and older internal platforms often exist in production only. The other side has no interest in maintaining a sandbox for your integration, and you can't create one yourself.

The dependency is rate limited or quota bound. Running a full suite in parallel against a service that allows a handful of requests per minute gives you failures caused by throttling, not by your code. Those failures are noise, and noise trains teams to ignore red builds.

Another team owns it and it's unstable. If the service you depend on is mid-rewrite, down every other afternoon, or deployed on a schedule you don't control, your pipeline inherits its instability. Your tests fail for reasons that have nothing to do with your change.

The real thing has side effects you can't undo. Sending an actual email, triggering an actual payment, or creating an actual shipment during a test run is a problem no cleanup script fully solves.

In all five cases the choice isn't "virtualize or call the real service." It's "virtualize or don't test this integration at all," and the second option is how integration bugs reach production.

What a virtual service has to get right

Once service virtualization is the only route, the quality of the virtual service decides whether your tests mean anything. A few properties matter more than people expect.

Fidelity to real responses. A virtual service built from documentation reproduces what the documentation says. Real systems return things documentation leaves out: extra fields, inconsistent casing, empty strings where the spec promises nulls. The inventory story above is a textbook case. The closer the virtual service is to observed behavior, the fewer surprises reach production.

Error behavior, not just success. Happy-path responses are easy to simulate and rarely where integrations break. Timeouts, malformed payloads, partial failures, and odd status codes are what your error handling gets tested against, and they're also what a hand-built virtual service is least likely to include, because nobody thought to write them down.

State across a sequence. Many real interactions span several calls: create a record, fetch it, update it, fetch it again. A virtual service that returns the same canned response regardless of history will pass tests that a real system would fail.

Staying current. A virtual service is a snapshot. When the real dependency changes and the snapshot doesn't, tests keep passing against a system that no longer exists. This is the quiet failure mode of the whole category, and it's worth deciding up front who is responsible for refreshing it and how often.

Build it by hand, or derive it from real behavior

There are two broad ways to produce a virtual service. The first is to define behavior manually: read the documentation, write out the responses, and script whatever logic the interaction needs. This is quick to start and gives full control, and it inherits every weakness above. It reflects what the author believed the dependency does on the day it was written.

The second is to derive the virtual service from observed traffic: capture real requests and responses, then replay them. This addresses fidelity and error coverage directly, since the recorded behavior includes whatever the dependency actually did, including the odd cases nobody documented. Refreshing it becomes a matter of recording again instead of editing definitions by hand.

Keploy is one tool that works this way. It captures API and database traffic at the network layer using eBPF, without changes to application code, and turns that traffic into replayable mocks and test cases, so the simulated dependency reflects real recorded interactions rather than an author's assumptions.

The practical catch with any capture-based approach is that you need traffic to record. For a dependency you truly can't call from a test environment, that usually means recording from an environment that already talks to it, such as a development setup with limited access, or production traffic captured carefully and scrubbed of sensitive data before it becomes a test fixture. That scrubbing step deserves real attention. Recorded traffic can contain tokens, personal data, and account identifiers that must not end up committed to a repository.

A short checklist before you rely on one

Before trusting a virtual service in CI, it's worth asking a few plain questions:

Where did its behavior come from: documentation, someone's memory, or observed traffic?
Does it include failure cases, or only successes?
Can it handle a multi-step sequence, or does it return the same answer every time?
Who refreshes it when the real dependency changes, and how would anyone know it had drifted?
Is there any sensitive data baked into its recorded responses?

If most of those answers are vague, the virtual service is closer to a guess than a test double.

The point of all this

When you can't call a dependency, a virtual service stops being a convenience and becomes the only evidence you have that your integration works. That raises the bar on how it's built. The inventory integration at the start of this piece didn't fail because the team skipped testing. It failed because the thing standing in for the partner was a reasonable guess, and a reasonable guess is a fragile thing to build a release on.

Regression Test That Would Have Caught It, If Anyone Had Written It

Marx Jenes — Wed, 23 Sep 2026 13:45:27 +0000

We fixed a discount-stacking bug once - a promo code and a loyalty discount were combining in a way finance definitely hadn't approved - shipped the fix, closed the ticket, moved on to the next sprint. Eleven months later, a refactor of the pricing service reintroduced the exact same bug. Same root cause, same wrong output, same customers getting a discount they shouldn't have. Nobody on the team remembered the first incident. The person who fixed it originally had left the company. The only record that it had ever happened was a closed Jira ticket nobody thought to search before merging the refactor.

That's the entire argument for regression testing in one story, and it's a much narrower argument than "run the tests again before release." A regression test isn't a general-purpose safety net. It's a specific, permanent record that says: this exact thing broke once, here's proof it's not allowed to break the same way again.

Why the fix alone was never enough

Fixing a bug closes the incident. It doesn't prevent the next person from reintroducing it, because a fix by itself leaves no trace in the codebase that says why the old behavior was wrong. Six months later, someone refactoring that code has no way of knowing a previous version of this exact logic caused a real problem - they just see code that looks reasonable, changes it in a way that looks reasonable, and ships it.

A regression test is what turns "we fixed this" into "this can't happen again without someone finding out immediately." It's not really about testing in the abstract sense. It's closer to institutional memory that runs in CI instead of living in a person's head, which is exactly why it survives someone leaving the team and a ticket getting buried in a backlog nobody searches.

Why it gets skipped anyway

Nobody skips this step because they don't understand its value. They skip it because writing the regression test is extra work at the exact moment the incident feels resolved - the fix is deployed, the customer's been refunded, everyone wants to move on to whatever got deprioritized while the fire was being put out. Writing a test that specifically encodes "promo codes plus loyalty discounts shouldn't stack past X%" takes real thought: what's the minimal case that reproduces it, what's the assertion, does it need its own fixture or does it fit into an existing suite.

Under deadline pressure, that thirty minutes loses to whatever's next on the sprint board almost every time. The bug gets fixed. The proof that it was ever a bug doesn't get written down anywhere durable. This is an extremely normal way for teams to operate, and it's also exactly how the same bug comes back a year later wearing a different diff.

What actually closes the gap

The teams that do this well tend to treat "write the regression test" as part of the incident's definition of done, not an optional follow-up. Not every bug fix needs one - a typo in an error message doesn't need a permanent regression test guarding it forever - but anything involving money, data correctness, security, or a customer-facing failure earns one as a rule, no exceptions for how busy the sprint is.

The other thing that helps, less obvious but just as important: the regression test should be built from what actually happened, not a cleaned-up, idealized version of it. If the real bug involved a specific combination of a promo code, a loyalty tier, and a checkout retry after a timeout, the test should reproduce that specific combination, not a simplified "discounts shouldn't exceed X%" check that misses the exact conditions that caused the real failure. A regression test that tests a simplified version of the bug can pass while the actual bug is still very much alive.

This is where a lot of regression suites quietly lose their value over time - not because nobody's writing tests, but because the tests that do get written are approximations of the incident rather than reconstructions of it, written from memory a few days later instead of from what actually happened on the wire when it broke.

The suite that actually earns trust

A regression suite built this way - one entry per real incident, grounded in what genuinely happened, added as a non-negotiable step rather than a someday task - becomes something a team actually trusts, because every test in it has a story behind it. Not "we thought this might be a problem," but "this exact thing happened, and here's the proof it can't happen silently again."

The discount bug that came back eleven months later wasn't a failure of testing philosophy. It was a gap between "we fixed it" and "we made sure it can't happen again the same way," and that gap is exactly where regression testing is supposed to live. Closing it takes thirty minutes at the time it's most annoying to spend them. Not closing it costs a lot more than thirty minutes, just later, and usually to someone who wasn't even there the first time.

Playwright API Testing Plus E2E: Testing the Same User Flow at Two Layers, on Purpose

Marx Jenes — Mon, 21 Sep 2026 12:48:39 +0000

We had a signup flow that passed its E2E suite every single time a backend bug shipped that made new accounts default to the wrong pricing tier. The UI never noticed, because the UI didn't care what tier got assigned, it just cared that signup completed and redirected somewhere that looked like a dashboard. It took a support ticket from an annoyed customer to catch what an API-level assertion would have flagged in about four seconds.

That's the case for testing the same flow twice, on purpose, at two different layers. Not out of paranoia, and not because either layer is unreliable on its own, but because each one is structurally blind to a different category of bug.

Two layers, two different questions

An E2E test asks: does this work the way a person experiences it? Click, type, submit, does the right thing show up on screen, in the right state, without anything visibly breaking. It's the closest thing to ground truth for "does this feature actually work," and nothing else really substitutes for it.

An API test asks a narrower, sharper question: given this exact request, is the response correct? Right status code, right shape, right data, right side effects in whatever it touched. It doesn't care what any of that looks like rendered. It just cares whether the backend did the right thing.

The pricing-tier bug above is a clean example of the gap. The UI layer genuinely couldn't have caught it, because the UI never displayed the tier during signup at all — that information only showed up three screens later, in a part of the flow the E2E test didn't walk through. The API response, on the other hand, included the tier on every signup call. An assertion there would have caught it same-day.

Why one flow, tested twice, isn't redundant

The instinct to avoid this is understandable — testing the same user journey twice feels like doubling maintenance for no reason. In practice it's closer to doubling your blind-spot coverage than doubling your effort, because the two layers rarely fail for the same reason.

A flow can be API-correct and UI-broken: the backend does exactly the right thing, and a frontend bug displays it wrong, loses it in a bad render state, or fails to trigger the next step. Only the E2E test catches that.

A flow can be UI-correct and API-wrong: the frontend renders a perfectly reasonable-looking success state built on a response that's subtly incorrect underneath — wrong tier, wrong currency, a field that's silently null where it shouldn't be. Only the API test catches that, and usually catches it faster, because it's checking data directly instead of waiting for a person (or an E2E assertion) to notice something looks off on screen.

What actually changes when you build both, deliberately

This isn't "run every test twice." It's picking the handful of flows where the consequences of being wrong are actually expensive — signup, checkout, anything involving money or account state — and being intentional about covering each one from both directions instead of assuming one layer implies the other.

In practice this usually means the API-level version of the test goes deeper into the specific values and edge cases (what happens with an invalid promo code, what happens if a webhook fires twice, what the response looks like for every plan tier) while the E2E version stays focused on the handful of states that actually matter visually — does the confirmation show up, does the redirect happen, is there any state where the user gets stuck or sees something clearly broken.

Playwright api testing makes this pairing easier to actually maintain than it would be with two separate tools, since request-level API calls and full browser interaction live in the same test runner, the same fixtures, sometimes even the same file — which matters, because the version of this that never gets written is the one that requires switching tools and mental models every time.

The part that's easy to skip

The temptation, once both layers exist, is to let the API test quietly become the E2E test's setup step — using API calls to fast-forward through login or account creation so the E2E test can jump straight to the interesting part. That's a legitimate and common pattern for speed, but it's worth being honest that it's a different use of the API layer than testing the API layer as its own subject. A signup API call used to skip ahead in an E2E test isn't verifying the signup response is correct. It's assuming it is, so the UI test can move faster.

Both are useful. They're just not interchangeable, and conflating them is how a team ends up with heavy API usage across their suite and still no actual API-level assertions checking that the backend does the right thing on its own terms.

Where this earns its keep

The signup bug at the start of this piece is the shape of thing this pattern catches early instead of catching from a support ticket: something correct enough to pass the layer someone happened to test, and wrong enough to matter in the layer nobody checked. Testing a flow once tells you it works from one angle. Testing it deliberately from two tells you which angle, if either, is lying to you when something eventually breaks.

Sandbox Testing Tools for Teams Testing Against Real Dependencies

Marx Jenes — Wed, 16 Sep 2026 11:06:29 +0000

A team I worked near a while back had a "sandbox" environment that was really just a second production database with fewer permissions. Tests ran against it, passed, and nobody noticed the sandbox had drifted out of sync with what the real payment provider actually returned until a release broke checkout for a subset of users whose cards triggered an error path the sandbox had never simulated. The sandbox wasn't fake in the sense of being obviously unrealistic. It was fake in a quieter way — it just hadn't kept up.

That's usually the actual failure mode with sandbox testing, not "the sandbox is missing" but "the sandbox stopped matching reality months ago and nobody was checking." Worth going through what a sandbox needs to actually do its job, then a rundown of the tools teams commonly reach for.

What a sandbox testing tool actually needs to provide

Before comparing any sandbox testing tool, it's worth being specific about what "good" means here, because a lot of tools that call themselves sandboxes only solve part of this:

Isolation — tests shouldn't touch real production systems, real user data, or real third-party services with real consequences (real charges, real emails sent, real rate limits consumed).
Realism — the simulated environment needs to behave like the real one, including error cases, latency, and edge cases, not just the happy path someone remembered to configure.
Freshness — this is where most sandboxes quietly fail. A sandbox configured once and never updated drifts from the real dependency's actual behavior, the same way a hand-written mock does.
Reproducibility — the same test run against the sandbox should produce the same result every time, which is harder than it sounds once real network calls or shared state are involved.
Low setup cost — a sandbox that takes a day to configure per service doesn't get maintained, and an unmaintained sandbox is worse than no sandbox, because it gives false confidence.

With that as the bar, here's how the commonly used tools stack up.

Container-based sandboxes: Docker Compose, Testcontainers

Testcontainers and plain Docker Compose setups are probably the most widely used approach for sandboxing infrastructure dependencies — spinning up real instances of a database, message queue, or cache in a container, scoped to a test run, then tearing it down. This gets isolation and reproducibility right, since you're running the actual software, not a simulation of it.

Where this approach is weaker: it sandboxes infrastructure you control, but doesn't help much with third-party APIs or internal services you can't just spin up in a container — a payment provider, an external auth service, another team's API you don't own.

API mocking/simulation platforms: WireMock, Mockoon

For external APIs, WireMock and Mockoon are common choices — they let you define stub responses for HTTP endpoints and run them as a lightweight standalone sandbox server. Strong on setup simplicity and control; you can hand-craft exactly the responses and error cases you want to test against.

The weakness is the same one hand-written mocks always have: someone has to define every scenario manually, and nothing keeps those definitions in sync as the real API changes. A WireMock sandbox is only as realistic as the person who last updated it remembered to make it.

Contract-based sandboxes: Pact

Pact takes a different angle — instead of manually defining sandbox responses, it generates them from contracts agreed between a service and its consumers. This solves part of the drift problem, since a contract change is at least visible and versioned, but it depends on both sides maintaining accurate contracts, which is its own discipline that not every team keeps up consistently.

Traffic-capture-based sandboxes: Keploy

Keploy takes an approach closer to the "freshness" problem directly: instead of hand-defining sandbox behavior or relying on maintained contracts, it captures real API and database traffic at the network layer using eBPF and turns that captured traffic into a sandbox environment, along with test cases and mocks generated from it.

The practical difference this makes: the sandbox reflects what a dependency actually did, including real error responses, real latency characteristics, and edge cases that occurred in practice rather than ones someone thought to configure. Because capture happens at the network layer, it also works across different languages and frameworks without requiring code instrumentation in the service being tested, which matters for teams with a mixed-language stack where writing per-language stub definitions gets expensive to maintain consistently.

This doesn't make it strictly better in every situation — a Testcontainers-based sandbox is still the right call for infrastructure you control and want to run for real rather than simulate, and a hand-crafted WireMock stub is still faster to set up for a single, simple, rarely-changing endpoint. Where the traffic-capture approach earns its place is specifically the freshness and coverage problem: sandboxes that would otherwise need constant manual upkeep to stay accurate, especially across a distributed system with a lot of interdependent services.

Choosing between them

A rough way to think about it: use container-based sandboxes for infrastructure you own and can run directly. Use hand-defined mocking tools for external dependencies with a small, stable set of scenarios you're comfortable maintaining by hand. Use contract-based tools where you have real cross-team discipline around API contracts already. And where the actual problem is a sandbox drifting out of sync with a dependency's real behavior over time, especially across several services, a traffic-capture-based approach is worth prioritizing over one that depends on someone remembering to keep definitions updated.

None of these are strictly better than the others across every situation. The mistake worth avoiding is the one from that payment sandbox story: picking an approach once, without revisiting whether it's actually still matching reality six months later.

Dependency Mocking vs Service Virtualization: What Actually Replaces Hand-Written Mocks in Integration Tests

Marx Jenes — Thu, 10 Sep 2026 12:46:30 +0000

Somewhere around our third microservice, our hand-written mocks stopped being an asset and started being a liability nobody wanted to own. Every mock lived in its own file, written by whoever needed it at the time, matching whatever the dependency returned on the day it was written. Six months later, a payments service changed its error response shape, and every mock that simulated it kept returning the old shape forever, because nothing told the mocks the world had moved on. Our tests stayed green. Production did not.

That's the point where most teams start asking whether there's something better than hand-writing mocks one file at a time, and the two answers that usually come up, dependency mocking (tooling) and service virtualization, sound similar enough that people use them interchangeably. They're not the same thing, and picking the wrong one for your situation costs you real time.

What hand-written mocks actually cost you

Before comparing tools, it's worth being honest about why hand-written mocks break down at scale:

They drift. A mock is a snapshot of a dependency's behavior at the moment someone wrote it. Nothing keeps it in sync as the real dependency changes.
They're incomplete by construction. Someone writes a mock for the cases they thought to handle - success, one or two errors. Real dependencies fail in ways nobody imagined until it happens in production.
They don't scale with team size. Each new mock is a new maintenance surface. Multiply that across a team writing dozens of integration tests a week and you get an unmanaged pile of stale fixtures nobody trusts.
They're disconnected from real traffic. A mock represents what someone thinks a dependency does, not what it actually does across the range of real requests hitting it in production.

Both dependency mocking and service virtualization tools exist to fix some version of this, but they fix different parts of it.

Dependency mocking (as a tooling category)

Dependency mocking tools automate the creation and maintenance of mocks for individual dependencies - a specific API, a database call, a message queue. Instead of hand-writing a mock object, you use a library or framework that intercepts calls to a dependency and returns configured or recorded responses.

The better tools in this category address the drift problem directly by generating mocks from observed behavior rather than someone's manual guess - recording a dependency's actual responses and replaying them, rather than hand-typing what someone assumes the response looks like. This is a meaningfully different approach from classic hand-rolled mocking, even though both get called "mocking."

Where this fits best: testing one service's interaction with one or a few specific dependencies, in unit or narrow integration tests, where you want fast, deterministic, isolated tests and don't need a full simulated environment.

Where it falls short: it typically operates one dependency at a time. It doesn't give you a full simulated environment where multiple interdependent services behave consistently with each other - which is exactly the situation distributed systems testing runs into.

A few tools commonly used in this category, each taking a different approach to generating or managing mocks: WireMock (HTTP-level stubbing, widely used in the Java ecosystem), Mockoon (a desktop app for designing and running mock APIs manually), Pact (contract-testing focused, mocks generated from consumer-defined contracts rather than recorded traffic), and Keploy (generates mocks automatically from real API and database traffic captured at the network layer via eBPF, rather than hand-defined stubs or contracts).

Service virtualization

Service virtualization operates at a different scope: instead of mocking individual calls, it simulates entire dependent systems or environments - a whole downstream service, or a cluster of them, behaving consistently as a unit, the way the real system would across a full transaction.

This matters most in distributed systems, where a single test scenario might touch five or six services, some of which are unavailable in a test environment, rate-limited, expensive to call repeatedly, or owned by another team entirely. Service virtualization gives you a realistic stand-in for that whole dependency graph, not just one endpoint.

The stronger implementations of this approach are also built from recorded, real traffic rather than someone's hand-built simulation - capturing the actual request/response behavior of a dependency (or a whole set of them) and replaying it faithfully, including realistic latency, error rates, and edge cases that were never intentionally designed into a hand-written mock because nobody thought to.

On the tooling side, this space includes established enterprise platforms like Broadcom's Service Virtualization (formerly CA Service Virtualization) and Parasoft Virtualize, both of which let teams build simulated environments manually or from recorded interactions, and Hoverfly, an open-source option for capturing and simulating HTTP service behavior. Keploy also applies here on the recorded-traffic side specifically - since it captures real inter-service traffic via eBPF, it can generate the kind of realistic, environment-wide behavior simulation service virtualization is meant to provide, without needing a separately maintained virtual environment built by hand.

Where record-and-replay changes the comparison

The most useful development across both categories, in practice, is the shift away from hand-authored simulation entirely and toward capturing real traffic and replaying it as test infrastructure. This addresses the core weaknesses of manual mocking directly:

Drift stops being a maintenance burden when mocks are regenerated from actual captured traffic rather than manually updated by a person who has to remember to do it.
Coverage improves because recorded traffic includes the edge cases real usage produces - malformed payloads, unusual timing, rare error responses - not just the cases a person thought to write by hand.
CI determinism is preserved because replay is exact: the same recorded interaction plays back the same way every run, without depending on a live, possibly-flaky external dependency actually being reachable during a test.

This is the part of the space Keploy specifically focuses on: capturing real API traffic and database interactions directly at the network layer using eBPF, then generating both mocks and full integration/regression test cases from that captured traffic automatically, rather than requiring either hand-written mock objects or a separately maintained virtualization environment. Because the capture happens at the network layer, it works across languages and frameworks without instrumenting application code, and because the mocks are generated from what a dependency actually returned, they don't carry the "someone's best guess" problem that hand-written mocks always have.

The practical effect is that it blurs the line between the two categories in this comparison: it produces artifacts that function like dependency mocks (deterministic, fast, scoped per test) but are generated the way good service virtualization is built (from real recorded system behavior, not manual authorship), which is a meaningfully different position than tools that only do one half of that.

A rough way to choose

If you're testing a single service's handling of a small number of specific dependencies, and you mostly need fast, deterministic, isolated tests - dependency mocking tooling is usually the right scope, especially if it supports generating mocks from recorded traffic rather than manual definitions.

If you're testing across a distributed system where multiple services need to behave consistently together, and hand-maintaining that many mocks isn't realistic - you're in service virtualization territory, and the traffic-capture-based approaches are worth prioritizing over ones that require you to hand-build the virtual environment yourself.

Either way, the actual upgrade from hand-written mocks isn't really "mocking tool vs virtualization tool" as a binary choice. It's moving away from manually authored simulation of any kind and toward tests grounded in what your dependencies actually do, captured from real behavior instead of guessed at a keyboard.

Software Testing Basics: The One Habit That Matters More Than Any Tool

Marx Jenes — Thu, 27 Aug 2026 12:10:48 +0000

A junior dev I mentored a while back asked me which testing framework he should learn first. Reasonable question. I told him it didn't matter much yet, and he looked at me like I'd dodged it. I hadn't. The honest answer is that the framework was the least important thing standing between him and writing tests that actually caught bugs.

The thing that actually mattered - the one habit I've seen separate people who write tests that catch real problems from people who write tests that just exist - is stupidly simple: write down what you expect to happen before you write the test that checks it. Not "test the login function." What, specifically, should happen when the password is wrong. What should happen when the account doesn't exist. What should happen when someone hits submit twice.

Why this sounds too obvious to matter

Most people already think they do this. They don't, not really. What actually happens most of the time is: write the code, run it, see what it does, write a test that confirms it does that. That's not testing. That's documentation of current behavior, and current behavior includes your bugs.

The habit is inverted from that. You write down the expected result before you look at what the code actually does, ideally before the code is even fully written. It feels slower. It's the entire difference between a test suite that catches regressions and one that just quietly agrees with whatever's already there.

Where I've seen this go wrong

Early in my career I tested a password reset flow by running it, watching it work, and writing an assertion that matched what I saw. Six months later someone changed an error message and my test broke - correctly, technically, since the output changed, but I couldn't tell if the new behavior was fine or actually wrong, because I'd never written down what "right" was supposed to look like. I'd only ever recorded what "current" looked like. Three of us argued about it in a PR review for twenty minutes before someone just asked product what the message was supposed to say in the first place, which is the question I should have answered before writing the test at all.

This is the same failure mode as confusing verification and validation - checking that code matches what it currently does isn't the same as checking that it does what it's actually supposed to do. The habit of writing the expected result down first is what forces that distinction to actually show up, instead of staying invisible until something breaks in a confusing way.

What this looks like in practice

Before writing a test, three quick questions, in this order:

What should happen here, in plain language, no code? ("Login fails with a clear error and no session is created" - not "returns 401.")
What's the one thing that would make this wrong even if it technically ran without crashing? (A silent failure. A vague error. Data left in a bad state.)
Would someone unfamiliar with this code understand what failed and why, just from the test name and assertion?

If you can't answer #1 without looking at the code first, that's the actual signal something's off - not the framework, not the tooling, not the coverage number.

Why this matters more than tool choice

Frameworks, runners, assertion libraries - these are genuinely interchangeable in a way that surprises people early on. Jest and Vitest and pytest all do fundamentally the same job with different syntax. Nobody's test suite got meaningfully better because they switched frameworks. Suites get better when the people writing them get sharper about defining "correct" before checking it.

This is also, not coincidentally, the actual foundation under most of what gets called software testing basics- the types of testing, the levels, the terminology. All of it assumes you already know what you're checking for. None of the vocabulary helps if that part's fuzzy. I found Keploy's rundown of the fundamentals genuinely useful for tying that vocabulary together in one place, if you want the fuller map of levels, types, and lifecycle beyond just this one habit.

Where it pays off later

This habit compounds in a way tool choice never does. A test suite where every test was written against a clearly stated expectation is a suite people trust - when something turns red, the team knows immediately whether it's a real regression or an intentional behavior change, instead of spending twenty minutes in a PR thread trying to reconstruct what "right" was supposed to mean.

Learn a framework whenever you need to. Learn to write the expected result down first, before you touch the code, and you'll write better tests in whatever framework you're handed next.

Verification vs Validation in Software Testing: A Practical Breakdown

Marx Jenes — Thu, 20 Aug 2026 08:06:12 +0000

A few years back I worked on a reporting feature that hit every acceptance criterion in the ticket. Every field matched the spec, every calculation was correct according to the documented formula, every edge case in the ticket was handled. QA signed off. Then it shipped, and the first support ticket came in within a day: the numbers were "technically right" but useless, because the spec itself had been written against an old version of how the business actually calculated that metric. Nobody had checked with finance before writing the ticket.

That's verification and validation in one story. We verified the software matched the spec perfectly. Nobody validated that the spec matched what the business actually needed. Both matter, they're not interchangeable, and mixing them up is one of the more expensive mistakes a team can make without realizing it's happening.

Verification: are we building it right?

Verification asks whether the software conforms to its specification, requirements, or design documents. It's internal-facing - you're checking the product against what was written down, not against what anyone actually wants.

This covers most of what people think of as "testing" day to day:

Code reviews checking implementation against a design doc
Unit and integration tests checking behavior against documented requirements
Static analysis, linting, spec-conformance checks
QA walking through a ticket's acceptance criteria one by one

Verification is good at catching a specific kind of bug: the gap between what was specified and what got built. It's terrible at catching a different kind of bug: the gap between what was specified and what should have been specified in the first place. My reporting story is a verification success and a validation failure, and verification alone had no way of flagging that.

Validation: are we building the right thing?

Validation asks whether the software actually satisfies the real-world need it's meant to serve, regardless of what the spec says. It's external-facing - checking against reality, users, and business intent, not against a document.

This looks more like:

User acceptance testing with actual users, not just QA reading a checklist
Beta releases and real usage data
Product and stakeholder review against the actual problem being solved, not just the ticket
Watching what happens when real traffic hits the thing, not just the scenarios someone thought to write down

Validation is where "it passed every test" and "it's ready to ship" turn out to be different sentences. A feature can be verified into oblivion - every acceptance criterion green, and still fail validation the moment a real user tries to do something the spec author never considered.

Why the mix-up is expensive, not just academic

Teams that only verify tend to over-trust their test suite. A hundred passing tests feels like safety, but if every one of those tests was written against a spec that had a gap, you've built a very well-tested version of the wrong thing. The bug isn't in the code. It's upstream, in the requirements, and no amount of additional unit testing against those same requirements will ever catch it.

This is also why "we have great test coverage" and "our users are happy" aren't the same claim, and teams get bitten when they treat the first as proof of the second. Coverage measures verification. It says nothing about validation.

The verification toolbox, and what each piece actually catches

It's worth breaking "verification" down further, because the different techniques catch genuinely different classes of mismatch:

Static verification happens before anything runs: code reviews, design reviews, static analysis, linting, type checking. These catch structural problems - a function that doesn't match its documented contract, a data flow that violates an assumption elsewhere in the system, a design that technically satisfies a requirement but contradicts another one nobody cross-checked. Cheap to run, cheap to fix at this stage, and the earlier a mismatch is caught here, the less it costs later.

Dynamic verification is everything that runs the code against expected behavior: unit tests, integration tests, contract tests, regression suites. This is where most engineering time goes, and for good reason - it's fast, repeatable, and automatable in a way static review isn't. But it's bounded by the same limitation every time: it can only check the code against what someone wrote down as "expected." A dynamic test suite with high branch coverage against an incomplete spec is still just thoroughly verifying the wrong thing.

Formal verification - proving a program correct against a formal specification using mathematical methods, sits at the far end and is mostly reserved for places where a bug is catastrophic: aerospace, medical devices, cryptographic protocols, safety-critical embedded systems. Most teams will never need it, but it's worth knowing it exists, because it makes the boundary clearer: even a mathematically proven-correct program is only proven correct relative to its spec. If the spec encodes the wrong requirement, formal verification will prove the wrong thing flawlessly.

The pattern across all three: verification techniques get more rigorous as you move from static review to formal proof, and none of them, at any level of rigor, can tell you whether the specification itself reflects reality. That gap only closes through validation.

The validation side, and why it resists automation

Validation techniques look different because they're checking against something inherently fuzzier than a spec:

User acceptance testing (UAT) puts the software in front of actual users or close proxies (product owners, domain experts) and asks whether it does what they need - not whether it matches a document. UAT frequently surfaces requirements gaps that were invisible during verification, because the people testing aren't checking a checklist, they're trying to accomplish something.

Beta programs and staged rollouts extend this to real usage at scale, which matters because individual UAT sessions still can't replicate the variety of how a broad user base actually behaves. A feature can sail through UAT with five friendly testers and still collapse against the messiness of real users doing things nobody anticipated.

Production monitoring and incident analysis is validation that never stops. Every bug report, every confused support ticket, every unexpected usage pattern is a live signal about whether the system satisfies the real need - arguably a more honest signal than any pre-release testing phase, because it's coming from reality instead of a simulation of it.

Domain and stakeholder review - going back to the people who understand the actual problem, not just the ticket, and asking whether this solves what they actually needed - is the least technical and most frequently skipped form of validation, and it's often the one that would have caught my reporting-feature story before it reached a user.

The common thread: validation techniques resist full automation because they're checking against a moving, real-world target rather than a fixed document. You can automate a regression suite. You can't fully automate whether something still makes sense to the humans using it, and teams that try to substitute more verification for that keep rediscovering the gap the hard way.

A rough way to think about where each one belongs

Verification tends to live close to the code - the earlier and more automatable, the better, because it's checking against something explicit and stable (a spec, a contract, a set of requirements). This is where automated test suites earn their keep: fast, repeatable checks that the implementation still matches what was agreed on.

Validation tends to need something verification can't provide on its own: exposure to reality. Real users, real traffic, real edge cases nobody thought to write into a ticket. This is also where a lot of teams under-invest, because it's harder to automate and harder to feel "done" with - there's no green checkmark for "definitely building the right thing," only ongoing signal from actual usage.

Verification and validation in software testing, does not substitute each other. A team that's excellent at verification and weak at validation ships well-tested products nobody needed exactly as specified. A team that's strong on validation but sloppy on verification ships the right idea built on a shaky, bug-prone foundation. The healthiest setups treat them as two different questions, asked continuously, rather than one "testing" phase that's supposed to cover both.

Next time a feature passes every test and still gets a confused reaction from users, it's worth asking which of the two actually failed. Usually it's not the tests.

How to Build a First Test Suite From Scratch for a New Project?

Marx Jenes — Wed, 12 Aug 2026 06:24:22 +0000

The worst test suite I ever inherited had 400 tests, and I trusted about six of them. The rest were either testing implementation details nobody cared about, duplicating each other, or so tightly coupled to internal function names that a harmless refactor broke thirty tests for no real reason. Reading that codebase taught me more about what not to do than any greenfield project ever has.

So when you're starting from zero, the goal isn't "write a lot of tests fast." It's building a suite you'll still trust a year from now. If you're new to this, getting the software testing basics right early matters more than covering everything - learning how to build a first test suite from scratch teaches you what to prioritize in a way that inheriting someone else's bloated suite never will. Here's roughly how I'd approach it.

Start with what would actually hurt if it broke

Before writing a single test, list the handful of things that would be genuinely bad if they silently broke - checkout completing, auth working, the core thing your product does actually happening. Not every function, not every branch. Just the stuff where a silent failure costs you money, users, or trust.

This list is usually shorter than people expect. Five to ten flows for most early-stage products. That's your actual test suite's job in the first few months, not "100% coverage."

Unit tests for logic, not for plumbing

Unit tests are for things with actual decision-making in them - pricing calculations, validation rules, state transitions, anything where "given this input, is the output correct" is a real question with a wrong answer possible. They're fast, they're cheap, and they should make up the bulk of your suite.

Skip unit-testing pure plumbing: a function that just calls another function and returns its result doesn't need its own test. That's the kind of test that pads a coverage number without catching anything real, and it's exactly the kind of test that made that 400-test suite so hard to trust.

Integration tests for the seams

This is where most early suites are too thin. Unit tests tell you your pricing function is correct in isolation. They don't tell you the checkout flow actually calls it correctly, passes the right currency, or handles what happens when the payment provider returns an error you didn't expect.

The seams between your code and everything external - your database, third-party APIs, other internal services are where bugs actually like to hide, because that's where two sets of assumptions meet and don't always agree. This is also the hardest category to get right early on, because hand-writing every plausible external response (success, timeout, malformed payload, rate limit) is a lot of manual work, and most teams just don't do it, which is part of why integration bugs are so common in production even in codebases with decent unit coverage. Tools that can record real API traffic and replay it as test cases - Keploy is one - exist specifically to close that gap without hand-writing every scenario, which is worth knowing about before you spend a week writing mocks by hand.

Don't test the framework

A depressing amount of early test-writing time goes into testing things the framework already guarantees - does this React component render, does this route return a 200. If your framework is reasonably mature, trust it. Test your logic, not its plumbing.

Make flaky tests a same-week problem, not a someday problem

The first flaky test in a new suite is a fork in the road. Fix it now, while the codebase is small enough that the cause is easy to find, or let it slide and watch the team's trust in the whole suite erode as more join it. I've written about this before - flaky tests are almost never actually about the test itself, they're usually pointing at shared state, a race condition, or an environment assumption nobody wrote down.

Let the suite grow with real usage, not just imagination

Whatever you write on day one will cover the bugs you could imagine. It won't cover the malformed payload from a partner's outdated client, or the auth token format some old integration still sends. As the project gets real traffic, treat production incidents as test-suite input - every bug that reaches production and wasn't caught is a gap in the suite, and turning it into a regression test is the cheapest way that gap ever gets closed.

The suite you actually want in six months

Not the biggest one. The one where a red result still means something, where the team fixes failures instead of quietly muting them, and where adding a new feature doesn't mean an afternoon fighting tests that were testing the wrong thing to begin with. Start small, aim it at what actually matters, and let it grow from real usage rather than a coverage target.

What AI Actually Helps With in Test Automation

Marx Jenes — Mon, 10 Aug 2026 04:30:11 +0000

What AI Actually Helps With in Test Automation (and Where It Falls Apart)

A teammate spent an afternoon last month feeding our checkout flow into an AI coding assistant and asking it to generate a test suite. Twenty minutes later he had forty tests. Good coverage on paper, clean syntax, decent naming. Then we ran them against a known bug we'd fixed the week before, and every single test passed anyway. The AI had written tests that matched what the code currently did, not what the code was supposed to do. It had no way of knowing the difference.

That's the honest version of where AI sits in test automation right now, and it's a lot less dramatic than most of what gets written about it.

Where it's genuinely useful

Scaffolding, fast. Writing the boilerplate for a new test file, setting up fixtures, wiring up assertions for a straightforward CRUD endpoint - this is exactly the kind of repetitive, low-judgment work AI is good at. It saves real time, especially early in a project when you're writing a lot of similar tests.

Spotting edge cases you didn't think of. Describe a function's inputs and ask what could break it, and you'll usually get a decent list back: empty strings, null values, boundary numbers, unicode weirdness. Not exhaustive, and not always relevant to your actual domain, but a reasonable starting checklist that's faster than staring at a blank test file.

Explaining existing test failures. Pointing an AI assistant at a stack trace and a failing assertion and asking "what's likely going on here" is one of the more underrated uses. It's fast at pattern-matching common failure modes - off-by-one errors, type mismatches, obvious null pointer issues, even if it can't always diagnose the deeper cause.

Refactoring test code itself. Cleaning up duplicated setup logic, converting a pile of copy-pasted tests into a parameterized suite, modernizing old assertion syntax - mechanical work that AI handles well because it's about the shape of the code, not what the code is supposed to prove.

Where it falls apart

It doesn't know what "correct" means for your system. This is the checkout example above. AI can write a test that passes against your current code, but it has no independent way of knowing whether your current code is right. It's not testing against your business logic, it's testing against whatever's already there - which means it's very good at locking in bugs, not just catching them.

It doesn't know your actual traffic patterns. A generated test suite tends to cover the inputs a human (or an AI) can imagine, not the inputs your system actually receives. Real API traffic is messier than anyone's imagination: malformed payloads from a partner's outdated client, an auth token format from three integrations ago that somehow still works, timing patterns nobody would think to write by hand. Software test automation built entirely from imagined scenarios misses a category of bug that only shows up under real usage.

Maintenance is still a human problem. AI can generate forty tests in twenty minutes. It doesn't show up six months later when the API contract changes and half of them need updating. Automated testing has always had a maintenance cost, and generating tests faster just means you're generating maintenance debt faster too, unless something is tracking how the underlying system actually behaves over time and flagging what's drifted.

It can't tell you what to test, only how. The genuinely hard part of test automation was never syntax. It's deciding what matters: which flows are business-critical, which failure modes are acceptable, where the risk actually lives. That's a judgment call rooted in what your product does and who depends on it. No amount of prompting hands that over.

Where this is heading

The realistic near-term picture isn't "AI writes your test suite," it's AI doing more of the mechanical work - scaffolding, edge-case suggestions, failure triage - while the actual test strategy, the ground-truth of what correct behavior looks like, and the ongoing maintenance stay firmly a team's responsibility. The tools that will matter most in this space long-term probably aren't the ones that generate the most tests fastest, but the ones that keep tests grounded in what a system genuinely does in production, rather than what someone (human or AI) guessed it should do.

Worth remembering next time a demo makes AI-driven test automation look like a solved problem: the demo is testing against a scenario someone already knew the answer to. Production isn't that generous.

Why Flaky Tests Are Rarely About the Test

Marx Jenes — Thu, 06 Aug 2026 12:42:52 +0000

We had a checkout test at my last job that everyone called "the coin flip." Green for a week, red twice on a Tuesday, green again. Someone eventually wrapped it in a retry and it sat like that for eight months before anyone looked at it again. Turned out the real bug was a webhook that occasionally fired before the order record finished writing to the DB - a two-hundred-millisecond gap that only showed up under load. The test wasn't broken. It was the only thing in the entire pipeline that noticed.

That's usually the story. Someone blames the test - bad selector, missing wait, a sleep(2) some intern left in there three years ago, and half the time they're right. But when a test flakes repeatedly and nobody can explain why, the test is rarely the actual problem. It's just the part of the system rude enough to say something.

A few places I keep finding the real cause hiding.

Tests that quietly depend on each other

Test A writes a row, Test B reads it and never knew it needed to. Run B by itself, it passes. Run the suite in a different order, or in parallel, and B fails for no reason anyone can point to. I've lost a full afternoon to this exact thing more than once - a cache value from Test 12 leaking into Test 47.

The actual fix is annoying and unglamorous: every test gets its own fixtures, its own scoped data, no assumptions about what ran before it. If your suite only goes green in one specific order, you don't have a flaky test. You have an undocumented dependency graph, and it's going to bite someone eventually.

The app is racing, not the test

Click a button, immediately assert on the result - that's a bet that the UI update lands the instant the click handler returns. It usually does, on your machine, on a good day. Add a debounce, a background job, or just enough network latency and that bet stops paying off.

This one's frustrating because the test isn't being paranoid. The app genuinely has a race condition. The test just runs the interaction often enough, across enough machines, that it eventually catches the app mid-race - something a human clicking through the same flow once or twice would probably never notice.

Clocks, timezones, and CI boxes that aren't your laptop

Date.now() and anything DST-adjacent will happily pass for months and then fail on one specific day of the year, or the moment your CI runner's timezone doesn't match what the original author assumed (usually their own laptop, usually not documented anywhere). Same idea with tests that were written assuming reasonably fast hardware - throw them on a throttled CI box and timing assumptions that never mattered locally suddenly do.

Mocks that stopped matching reality

This is the sneaky one, because it doesn't look like flakiness at all - it looks like a test that's passing. Someone mocks a third-party API, gets it right on day one, and then the real API changes: a field gets renamed, an error shape changes, rate limiting gets added. The mock has no idea. It keeps returning exactly what it always returned, the test stays green, and production quietly starts failing in a way nothing in CI can see.

If anything this is worse than a red test, because it erodes trust in the wrong direction - teams end up more confident in a mock-backed test than they should be, right up until it fails somewhere that actually matters.

The twenty-minute question

It's easy to treat a flaky test as noise to manage - quarantine it, slap a retry on it, mute it and move on. But almost every one of them is pointing at something real: state nobody's tracking, a race condition in the app itself, an environment assumption that was never written down, or a mock that fell out of sync with the thing it's supposed to represent.

Muting it doesn't make any of that go away. It just makes the pipeline quieter while whatever's actually wrong keeps happening, unwatched, in production.

Next time a test earns a reputation, it's worth the twenty minutes to ask why before reaching for a retry decorator. Most of the time it wasn't lying to you.