DEV Community: Marx Jenes

What Is API Regression Testing, and Why Modern Teams Can’t Skip It

Marx Jenes — Thu, 29 Oct 2026 11:48:02 +0000

What Is API Regression Testing, and Why Modern Teams Can't Skip It

The release notes said "minor refactor of the user service." Twenty minutes after deploy, the iOS app started showing blank profile pages. The refactor had renamed phone_number to phoneNumber in one response, every backend test still passed, and the mobile team found out from their crash dashboard.

That's the whole case for API regression testing in one story. API regression testing is the practice of re-running checks against an API after every change to confirm that existing endpoints still behave the way their clients depend on: the same status codes, the same response shapes and fields, the same error formats and the same side effects. It doesn't ask whether the new feature works. It asks whether everything that already worked still does.

What API regression testing covers

It helps to separate it from regular functional API testing. Functional tests check that new behavior works as designed: the new endpoint returns the right data, the new filter actually filters. Regression tests check that a change didn't quietly alter behavior that already shipped, often in endpoints nobody touched on purpose.

In practice, an API regression testing suite tends to watch for:

Status codes that change, like a 404 becoming a 200 with an empty body
Response structure, including renamed, removed or newly nested fields
Value formats, like dates switching from ISO strings to timestamps, or numbers turning into strings
Null handling, where a field that used to be null starts disappearing entirely, or the reverse
Error responses, since clients often parse error bodies to show messages
Side effects, like the database queries or downstream calls an endpoint makes

That last one gets missed a lot. An endpoint can return an identical response while suddenly making three extra database queries per request, which is fine in a test environment and painful under production load.

The regressions that actually happen

Real API regressions are rarely dramatic. They're small, reasonable-looking changes that someone else was relying on.

A serializer upgrade starts omitting null fields to "clean up" responses, and a client that checked for the key's presence breaks. A validation rule gets stricter, so requests that worked yesterday now get a 400. Pagination moves from page numbers to cursors, and the old parameter silently gets ignored, returning page one forever. A query refactor drops an ORDER BY, and a list that always came back sorted now comes back in whatever order the database prefers.

None of these throw errors on the server. Unit-level checks pass. Code review approves them because each change makes sense on its own. They only show up when something calls the API and compares the answer to what it used to get.

Why API regression testing matters in modern software development

APIs have consumers you don't control

A mobile app version released six months ago is still on thousands of phones, and it can't be updated by redeploying. Partner integrations were built against how your API behaved the day they shipped. Every one of those clients depends on behavior, not on your intentions.

Microservices multiply the blast radius

In a microservices setup, one API's response is another service's input. A small change in an upstream service can break three downstream ones, owned by teams who didn't know anything changed. API regression tests at service boundaries are often the only thing standing between "small refactor" and a cross-team incident.

Deploy frequency leaves no room for manual checks

Teams shipping several times a day can't run a manual regression pass before each release. If regression checks aren't automated and running on every pull request, they effectively aren't running at all.

Code changes faster than people can review it

AI-assisted coding has made large diffs routine. A refactor that touches forty files can look perfectly clean in review while changing a response shape somewhere in the middle. An automated comparison of API behavior doesn't get tired at file thirty-two.

Dependency upgrades change behavior quietly

Framework, ORM and serializer upgrades routinely ship small behavior changes inside minor versions. A green unit test run after a dependency bump says your assertions still hold, not that your API responds the same way it did yesterday.

Building an API regression suite without months of work

The classic approach is writing regression tests by hand, endpoint by endpoint. It works, but it's slow, and it only covers the cases someone thought to write down. Schema-based checks help catch structural drift if the API has an accurate OpenAPI spec. The approach that's grown fastest, though, is generating regression tests from real traffic, because real requests already contain the edge cases nobody would think to script. Open-source API regression testing tools like Keploy work this way, recording the real calls a service handles without code changes and saving its database and third-party API calls as mocks, so replays in CI stay stable and a failing test points at an actual change in behavior.

Whichever route a team takes, the baseline matters most. A regression test is only as good as the "before" picture it compares against, and the closer that picture is to how clients really use the API, the more regressions it'll catch.

Running API regression tests in CI

Wherever the tests come from, a few practices decide whether the suite stays trusted:

Run on every pull request. Regression checks that only run before releases catch problems after they're already tangled up with other changes.
Make dependencies deterministic. Tests hitting live databases and third-party sandboxes will flake, and flaky suites train people to ignore red builds.
Ignore true noise, and only true noise. Timestamps, request IDs and generated tokens change every run. Exclude those specific fields, never whole responses.
Treat baseline updates as reviews. When behavior changes on purpose, the updated baseline should go through the same review as the code, so "intended" is a decision someone actually made.
Prioritize the endpoints clients lean on most. Auth, checkout, anything a mobile app or partner calls directly. Broad coverage can grow later.

Conclusion

API regression testing exists because APIs make promises that outlive the code that made them. Every response shape, status code and error format becomes something a client depends on, whether or not anyone documented it. Modern development, with its microservices, constant deploys, AI-assisted diffs and dependency churn, keeps putting those promises at risk in ways that never show up as server errors.

The teams that handle this well don't rely on remembering what used to work. They capture it, replay it on every change, and let the diff tell them when something moved. A good first target is the one endpoint your oldest mobile app version calls the most, since that client will never get the chance to adapt.

What White Box Testing Misses When Your Code Talks to External Services

Marx Jenes — Tue, 27 Oct 2026 11:35:31 +0000

White box testing gives you visibility into the code. You can see which branches execute, which paths get covered, which conditions trigger which outcomes. For logic that lives entirely within a service boundary, that visibility is genuinely useful.

The problem shows up the moment your code talks to something outside that boundary.

The coverage number that doesn't tell you anything useful

A payment processing service with 87% code coverage sounds well-tested. Walk through what that coverage actually validates and the picture gets less reassuring.

The tests exercise the request construction logic. They exercise the response parsing logic. They exercise the retry logic. All of that runs against a mock of the payment gateway - a mock that was written when the integration was first built, reflecting what the gateway returned at that point in time.

What the coverage number doesn't capture: whether the mock still reflects what the gateway actually returns. Whether the retry logic handles the specific error format the gateway uses today, not the format it used when the mock was written. Whether the fields the parsing logic depends on are still present in the real response.

The code is covered. The integration isn't tested.

What white box testing is actually measuring

White box testing measures whether the code does what the developer intended. That's a meaningful thing to measure for logic that's self-contained - algorithms, data transformations, business rules that don't depend on external state.

For code that integrates with external services, the developer's intention is only half the equation. The other half is whether the external service behaves the way the developer assumed it would. White box testing has no visibility into that half. It can tell you the request was constructed according to the developer's model of the API. It can't tell you whether that model matches how the API actually behaves today.

The gap between "the code does what I intended" and "the integration works correctly" is exactly where production failures accumulate in services with external dependencies.

How the mock creates a false floor

The mock is the problem, but it doesn't look like the problem.

When a test passes against a mock, the implicit claim is that the code handles the dependency correctly. The test doesn't make this claim explicitly, but that's what passing means to the person reading the CI results. Green means the integration works.

What the mock actually validates: the code handles the simulated dependency correctly. Whether the simulation is accurate is a separate question that white box testing doesn't ask.

External APIs change without coordinating with the teams consuming them. A field gets renamed. A previously optional key becomes required. An error code gets restructured. The pagination behavior changes. None of these changes break the mock - the mock doesn't know about them. The tests keep passing. The code that was validated against the old behavior gets deployed to production, where it encounters the new behavior for the first time.

This isn't a testing failure in the sense of bad test design. It's a structural limitation of what white box testing can see.

The boundary white box testing can't cross

Code coverage tools instrument the code under test. They can observe every branch and condition inside the service. What they can't observe is the behavior of the external service itself - what it actually returns under what conditions, how its response structure has evolved, what its failure modes look like under real load.

That boundary is the one that matters most for integration reliability. The bugs that white box testing catches well - logic errors, edge case handling, incorrect branch conditions - are the bugs that would have been caught in code review or local testing anyway. The bugs it misses - assumptions about external service behavior that are no longer accurate - are the ones that survive to production because nothing in the test suite was positioned to catch them.

Observation-based testing approaches close this gap differently. Instead of asserting that the code handles a mock correctly, they record what the real service actually returns and validate the code against those observations. When the external service changes, re-recording shows exactly what shifted. The team finds out before deployment rather than from a production failure.

What actually needs to be tested at the external boundary

White box testing remains the right tool for validating internal logic. The retry algorithm, the request construction, the response parsing - these are code paths that benefit from branch coverage and condition testing.

What needs complementary coverage is the contract between the service and its external dependencies: whether the requests the service sends match what the external service expects, and whether the responses the external service returns match what the service's parsing logic assumes.

That contract isn't visible inside the code. It exists between the service and the external world, and validating it requires something that can observe both sides - not just the internal logic, but the actual behavior of the dependency the code is written to handle.

White box testing tells you the code is internally consistent. It can't tell you whether the world the code was written to interact with still looks the way the code expects it to.

Dependabot Says the Upgrade Is Safe. Is It?

Marx Jenes — Thu, 22 Oct 2026 12:45:37 +0000

You know the drill. A Dependabot PR shows up. Some library went from 4.2.1 to 4.3.0. CI runs, everything’s green, maybe there’s even an auto-merge rule that just takes care of it. Nobody really looks. Why would they? It’s a minor version bump and the tests passed.

Then a week later someone notices the mobile app is showing blank dates on the order screen. Or an integration partner emails asking why a field disappeared from your API responses. You go digging, and eventually you find it: that harmless little dependency bump changed something. Your tests never noticed because, honestly, your tests were never looking there.

Green CI doesn’t mean nothing changed

Here’s the uncomfortable part. When a dependency PR passes CI, what it really proves is that your tests still pass. That’s it. It doesn’t prove your app behaves the same way it did yesterday.

Those are really different claims. Your test suite checks the things someone thought to check, usually the main logic, the important calculations, the obvious error cases. A dependency upgrade tends to change stuff in the cracks. How a value gets serialized. What a default timeout is. How an ambiguous date string gets parsed. Tests rarely assert on that stuff, so they stay green while behavior shifts underneath them.

Semver is a promise, not a guarantee

“But it’s only a minor version!” Yeah. Semantic versioning says minor and patch releases shouldn’t break things. And library maintainers genuinely try to honor that. But here’s the catch: one person’s bug fix is another person’s breaking change.

Say a JSON library used to include null fields in its output, and in a minor release they “fix” it to leave them out, because that’s cleaner. Totally reasonable change. Also completely breaking for the client that expected those fields to be there, even as null. Or an HTTP client tightens a default timeout. Or a validation library gets stricter about input it used to quietly accept. None of these feel like breaking changes to the people who made them. They just are, for somebody.

And that’s before we even get to transitive dependencies, the libraries your libraries pull in. A single bump can drag in a bunch of other changes you never see in the PR title.

Where upgrade regressions like to hide

From what comes up in teams over and over, a few spots are repeat offenders.

Serialization is a big one. Field names, null handling, number formatting, date formats. All of it is controlled by libraries, and all of it is visible to whoever consumes your API.

Database access is another. ORM upgrades can change the queries that actually get generated. Sometimes it’s harmless. Sometimes it’s an extra query per row, or a different sort order when nobody specified one.

Then there’s the network layer. HTTP clients, retry libraries and TLS settings can all change how your app talks to the services it depends on, in ways that only show up under specific conditions.

Comparing behavior, not just running tests

So what actually catches this stuff? The trick is changing the question. Instead of asking “do the tests still pass?”, ask “does the app still behave the same way?”

The most reliable way I know to answer that is a before-and-after comparison. Capture how your app really behaves today, ideally from real traffic, so you’ve got actual requests and the actual responses that went back. Then run those same requests against the version with the upgraded dependency and compare. Any difference in a response shows up right away, whether it’s a missing field, a changed status code or a date in a new format.

It’s worth comparing what your app sends out, too, not just what it returns. If an ORM upgrade suddenly makes your service run extra database queries, or an HTTP client starts calling a downstream API differently, the response might look identical while something important changed underneath. Some regression testing tools record those outgoing calls along with the incoming traffic, often down at the network level so no code has to change, which means a dependency upgrade gets checked from both directions at once.

Making it part of the dependency PR

The nice thing is that this fits right into the workflow you already have. Dependabot opens a PR, CI runs your normal tests, and then it also replays your recorded traffic against the upgraded build and reports any differences.

Now the PR tells you something genuinely useful. Instead of just “tests passed,” it says “tests passed, and here are the three responses that changed.” Most of the time that list is empty and you merge with real confidence. When it’s not, you’ve got a specific thing to look at before anything ships.

A couple of habits make this work even better. Grouping a pile of upgrades into one PR saves time, but it makes it much harder to tell which library caused a difference, so keep the risky ones separate. And be thoughtful about auto-merge. It’s great for upgrades that pass a behavior comparison, but a bit nerve-wracking for ones that only passed unit tests.

A few low-effort extras

None of these replace a behavior check, but they help.

Actually skim the changelog for minor bumps on libraries that touch serialization, databases or networking. It takes two minutes and sometimes the answer’s right there. Keep your lockfile committed, so the exact versions you tested are the exact versions you ship. And when an upgrade does cause a regression, add that case to your baseline, so the next upgrade gets checked against it automatically.

Conclusion

I really like Dependabot. Keeping dependencies current is genuinely important, for security if nothing else. But a green check on a dependency PR deserves a little skepticism, because it proves a lot less than it looks like it does. Tests tell you your assertions still hold. They don’t tell you nothing changed.

If you want to try this, pick the service where a dependency bump has bitten you before. Capture a baseline of how it behaves today, and the next time Dependabot opens a PR against it, compare before and after. The first time it flags a “harmless” upgrade that quietly changed a response, you’ll probably never look at those green checks the same way again.

From the DevTools Network Tab to an API Test Suite

Marx Jenes — Mon, 19 Oct 2026 13:31:49 +0000

If you build for the web, you’ve probably done this a hundred times. Something’s acting weird, so you pop open DevTools, click over to the Network tab, filter down to Fetch/XHR, and watch the requests roll in. Find the one that looks off, right-click it, Copy as cURL, paste it into a terminal, and start poking at it.

It’s one of the most useful little habits in web development. It’s also, if you think about it, the first step of writing an API test. You’ve got a real request, captured from real usage, that you can run again whenever you want. The only problem is that it usually ends up in a terminal history or a Slack message, never to be seen again. So what would it look like to take that habit seriously and turn it into an actual API test suite?

Why the Network tab is secretly a test generator

Think about what’s sitting in that Network tab after you click through a feature. Every request your frontend sent. The exact URLs, headers and payloads. Every response the backend sent back, status codes and bodies included. In the right order, too.

That’s basically everything an API test needs. Real inputs. Real expected outputs. Real sequence. Nobody had to sit down and guess what payload the checkout endpoint expects, because the frontend already sent it.

Compare that to writing API tests from scratch, where someone reads the docs, guesses at the request shape, and hopes it matches what the UI actually does. The Network tab skips all that guesswork.

Where copying single requests stops scaling

Copy as cURL is great for one request. It gets painful fast after that.

For one thing, auth. That cURL command has a session cookie or a bearer token baked right into it, and those expire. Run it tomorrow and you’ll probably get a 401. Then there’s sequence. A lot of real flows are chains. Create a cart, add an item, check out, each step using an ID from the step before. Copy them one at a time and you lose the thread connecting them.

And there’s no notion of “correct.” A cURL command just fires a request. It doesn’t know what response is supposed to come back, so it can’t tell you when something’s broken. You’re the assertion, squinting at JSON in a terminal.

Capturing whole flows instead of single requests

The upgrade is to stop copying requests one by one and start capturing entire user flows.

DevTools can take you part of the way. Turn on “Preserve log” so requests don’t vanish when the page navigates, click through the flow, then export everything as a HAR file. A HAR is just a JSON log of every request and response in the session. Recent versions of Chrome even offer a sanitized export that leaves out sensitive stuff like cookies, which is a nice safety net if you plan to share the file.

That gets you raw material. Turning it into tests is still manual work, though. Someone has to pull out the relevant calls, untangle the auth, stitch the sequence back together and add checks.

This is where dedicated recorders earn their keep. Instead of exporting and hand-converting, they let you record API calls from your browser and turn them into tests more or less directly. You click through your app like a normal user, the recorder captures each XHR and fetch call along with its headers, cookies and auth context, and the captured flow comes out the other side as runnable tests, often with assertions already in place. Same habit you already have, just without the copy-paste.

Turning captures into a real API test suite

Whichever way you capture, a few things separate a pile of recorded requests from a test suite you’d actually trust.

Filter down to what matters

A raw capture includes way more than your API. Analytics pings, fonts, third-party widgets, that one chat bubble script. Filter by your API’s domain or path before anything else. If you can set a URL filter while recording, even better, since the junk never gets captured in the first place.

Keep complete request and response pairs

Every test needs both halves: what was sent and what came back. Background requests sometimes get captured with a missing body or status, especially when the page navigates away mid-request. Toss incomplete pairs, or repair them, before they turn into confusing tests.

Sort out auth for replay

Captured tokens won’t live forever. The usual fix is having the suite log in fresh at the start of each run, then reuse that session for every test. That way recordings stay valid long after the original session expired.

Add assertions that won’t break for no reason

Comparing the entire response body sounds thorough, but timestamps and IDs change every run. Focus on the status code, the overall shape of the response and the handful of fields that actually matter, like an order total or a user’s role.

Keep each flow in one piece

If a flow depends on order, keep it together as a single test. A checkout broken into separate, unrelated requests will just fail in confusing ways.

Running it on every change

Once the suite exists, put it where it’ll actually get used. Running it in CI on every pull request means backend changes get checked against real user flows before they merge. And because these tests skip the browser entirely, they’re fast. A flow that takes a minute to click through can replay in a second or two.

Keeping it fresh is pleasantly low effort. When a feature changes, record that flow again and replace the old captures. Review the diff in a pull request, and any new field or changed status code jumps right out.

Conclusion

The way I see it, the DevTools Network tab has been a test generator hiding in plain sight this whole time. Every developer already uses it to grab real requests. The only leap is treating those captures as something worth keeping, then capturing whole flows instead of single calls and turning them into a suite that runs on its own.

Want to try it? Pick one flow you test by hand all the time, maybe login or checkout. Capture it once from your browser, turn it into a test with a couple of sensible assertions, and run it next to your normal suite. The next time that flow breaks, you’ll hear about it from CI instead of from a frustrated user.

Sandbox Testing Tools for Reproducing Production Bugs Locally

Marx Jenes — Fri, 16 Oct 2026 13:48:32 +0000

Every developer has a version of this story. A bug report comes in. Customers are definitely seeing it. You pull the latest code, run the service locally, try the exact same thing, and... nothing. Works perfectly. You try again. Still fine. Meanwhile the error dashboard keeps ticking up.

“Works on my machine” is a running joke for a reason. Production is a messy place, full of real data, real traffic patterns and real dependencies that behave in ways your laptop never sees. The good news is that this gap has gotten a lot easier to close. Modern sandbox testing tools make it possible to pull the conditions of a production bug into an isolated environment on your own machine, where you can poke at it as long as you need without touching anything live.

Why production bugs refuse to show up locally

When a bug only happens in production, the code is rarely the whole story. Usually something about the surroundings is different.

Data is the most common culprit. A customer record with an unusual field, an order with a currency nobody tested, a username with an emoji in it. Your local database simply doesn’t have that row, so the code path never runs.

Dependencies come next. A partner API might return a slightly different response shape for certain accounts. A downstream service might send back an empty list where you expected at least one item. Locally, you’re either hitting a stub that always returns the happy path or a sandbox that behaves nothing like production.

Then there’s sequence and timing. Some bugs only appear when calls happen in a particular order, or when one dependency is slow and another is fast. Reproducing that by hand, clicking around a local UI, is close to impossible.

What it actually takes to reproduce a bug

If you strip it down, reproducing a production bug means rebuilding a small slice of reality. You need the request that triggered the problem. You need to know what every dependency returned at that moment, including database queries, API responses and messages from queues. And you need the service running in an environment where nothing else can interfere.

That last point is where sandbox testing comes in. A sandbox gives the service its own isolated space, so the only things influencing its behavior are the inputs you deliberately feed it. Get those inputs right, and the bug has nowhere to hide.

Capturing the evidence

The first job is gathering enough detail about what happened. Logs and distributed traces are usually the starting point. A request ID or trace ID ties together everything the service did while handling the failing request, which tells you which dependencies were involved and roughly what they returned.

Traces can only take you so far, though. They often summarize payloads or skip them entirely. The richer option is recorded traffic: the actual requests and responses flowing between the service and everything it talks to. Some teams capture this continuously in a staging environment where the bug also appears. Others mirror a slice of production traffic into a safe environment specifically so they can record it. Either way, having the real bytes beats guessing at them from a log line.

Keeping customer data safe

Anything captured from real traffic needs careful handling. Personal data and credentials should be redacted before recordings leave a secure environment, and ideally replaced with consistent placeholder values. Consistency matters here. If the same customer ID gets replaced with a different random value in every recording, the relationships between calls break, and so does the reproduction.

Building the local sandbox

Once you have the recorded traffic, the local setup gets surprisingly straightforward. Your service runs on your machine as usual. The request that triggered the bug gets replayed against it. And instead of talking to real dependencies, the service gets answered from the recordings: the same API responses, the same database results, the same queue messages it saw when the bug happened.

The way recordings are captured makes a real difference here. Tools that capture traffic at the network level, rather than through a proxy configured inside the application, can record database and broker traffic alongside API calls, without any code changes. That matters for bug reproduction, because a surprising number of production issues trace back to what the database returned rather than what an API sent.

Because replay is deterministic, the bug shows up the same way every time. You can attach a debugger, add logging, step through line by line, change one thing and replay again. No waiting for the error to happen in production. No hoping the timing lines up.

Reproducing timing and ordering bugs

Some bugs need more than the right data. They need the right timing. A race condition between two calls, or a timeout that only triggers when a dependency is slow, won’t reproduce if every recorded response comes back instantly.

Good sandbox testing tools let you adjust replay for this. You can slow down one dependency, deliver responses in a different order, or simulate a timeout at a specific point in the flow. It turns a “happens once in a while” bug into one you can trigger on demand, which is usually most of the battle.

Turning the reproduction into a regression test

Here’s the part that pays off long after the fix ships. Once a bug is reproduced in a sandbox, you already have everything a regression test needs: the triggering request, the recorded dependency responses and the expected behavior.

Keep that scenario around. After the fix, replay it again and confirm the service now handles it correctly. Then add it to the test suite so it runs on every change. That particular bug, with that exact data and that exact sequence, can never quietly come back.

Over time, this builds a suite of tests shaped like real incidents rather than imagined ones. Those tend to be the most valuable tests a team has.

What to look for in sandbox testing tools for bug reproduction

Not every tool is built with this use case in mind. These questions help when evaluating options:

Can it record traffic from a real environment, not just generate responses by hand?
Does it capture every dependency type the service uses, including databases and queues?
How much does the application need to change to record or replay traffic?
Can sensitive fields be redacted with consistent placeholders before recordings are shared?
Is replay deterministic, so the same bug appears on every run?
Can timing and ordering be adjusted during replay to surface race conditions?
Can a reproduced scenario be saved and run as a regression test in CI?

Conclusion

My take is that reproducing production bugs shouldn’t depend on luck or heroics. When the real conditions of a failure can be captured and replayed in an isolated sandbox, debugging turns from guesswork into routine work. And every bug you reproduce becomes a test that guards against it coming back.

If you want to try this, start with the next bug that refuses to show up locally. Instead of hunting for it by hand, capture the traffic around it, replay it in a sandbox on your machine, and see how quickly it surfaces. Once you’ve watched a “can’t reproduce” ticket turn into a failing test you can step through, it’s hard to go back.

Open Source Service Virtualization Tools I’ve Actually Used (and When Each One Made Sense)

Marx Jenes — Tue, 13 Oct 2026 11:44:26 +0000

Open Source Service Virtualization Tools I've Actually Used (and When Each One Made Sense)

The first time I heard the phrase "service virtualization," I assumed it was some enterprise thing with a sales team attached. Turns out I'd been doing a scrappy version of it for years, mostly by hand-rolling fake endpoints in Express whenever a partner API was down, rate-limited or just painfully slow.

Over the last few years I've worked on a payments integration, a couple of microservice-heavy backends and one very stubborn legacy system. Along the way I've tried most of the open source service virtualization tools people usually recommend. Some stuck. A couple I dropped after a week. None of them is "the best one," which I know is an annoying thing to read in a roundup, but it's true. They solve different problems.

So instead of ranking them, I'll walk through the ones I've actually used, what each is like day to day, and the situations where it earned its place.

First, What I Mean by Service Virtualization

Quick definitions, because people use these words loosely. A mock or stub usually lives inside your test code and fakes one function or one call. Service virtualization is a bit bigger. You stand up something that behaves like a whole dependency (a third-party API, a database, a downstream microservice) and your app talks to it over the network the same way it would talk to the real thing.

Why bother? In my experience it's usually because the real service charges per call, or it's flaky, or it doesn't exist yet because another team is still building it. Any of those can stall a test suite for days. Virtualizing the dependency gets you unstuck.

What I Look For Before Picking One

Before getting into specific tools, here's what I've learned to check up front. It saves a lot of switching later.

Protocol support comes first. If everything you call is plain HTTP and JSON, almost any tool works. Throw in gRPC, Kafka or raw TCP and the list gets short fast.

Then there's how the fake behavior gets created. Some tools want every stub written by hand, others record real traffic and play it back. Hand-written stubs give you precise control. Recorded ones save time and tend to match reality more closely. I've wanted both at different points, honestly.

I also think about who'll maintain the stubs. A tool the whole team can read and update beats a clever one only I understand. And does it slot into CI without drama? If it needs a special VM or a ten-step setup, it won't survive on a busy team.

The Open Source Service Virtualization Tools I've Actually Used

Same format for each one: how it works, when it made sense for me, and a few things worth knowing before you start.

Keploy: Record-and-Replay From Real Traffic

How it works: Keploy records real API calls to your application, along with the calls your app makes to its own dependencies like databases and external APIs, and turns them into test cases and mocks. It uses eBPF to capture traffic, so there's no need to change application code or wrap your HTTP clients. On replay, it serves the recorded dependency responses as mocks and compares your app's output with what it recorded.

When it made sense: One of those microservice backends had decent unit tests and almost no integration coverage, and nobody had time to write hundreds of stubs. Recording a few days of staging traffic gave us a regression suite that matched how the service was really used, database calls included. That database part mattered more than I expected. Most tools here virtualize HTTP services, and this covered the Postgres side as well.

Good to know: It captures traffic at the kernel level, so it runs natively on Linux, and on macOS or Windows you'd run it through Docker or WSL. Recording works best when realistic traffic flows through the app first. A staging environment or a few scripted user flows is plenty to get started.

WireMock: Flexible HTTP Stubbing

How it works: WireMock started life as a Java library, and these days it also runs as a standalone server or a Docker container. You describe stubs as JSON mappings, basically "match this request, return that response." Request matching covers URLs, headers and bodies. It can record from a real API and save what it sees as stubs, and it can simulate faults like slow responses or dropped connections.

When it made sense: Our payments integration. The provider's sandbox went down at least once a week, usually right before a demo. We recorded the main flows into WireMock once, cleaned the stubs up by hand, and after that our integration tests stopped caring whether the sandbox was alive. Fault simulation turned out to be a nice bonus. We finally tested what happens when the provider takes eight seconds to answer, which, yes, happened in production more than once.

Good to know: It's at its best with REST and JSON. On a big API the mapping files pile up quickly, so agree on a folder structure early. Ask me how I know.

Hoverfly: Lightweight Proxy-Based Simulation

How it works: Hoverfly is written in Go and runs as a small proxy between your app and the services it calls. You put it in capture mode, run traffic through it, then switch to simulate mode and it replays what it saw. There are other modes too, like spy, which passes through anything it doesn't have a recording for. It comes with a CLI called hoverctl for managing everything.

When it made sense: A Go service that talked to four internal APIs owned by four different teams. Getting all four into one test environment at the same time was close to impossible. With the proxy approach we didn't have to change how the service made its calls at all. Point it at Hoverfly, record once, done.

Good to know: Since it works as a proxy, HTTPS means the client has to trust Hoverfly's certificate, which cost one of our devs most of an afternoon. It's centered on HTTP and HTTPS, so other protocols are out of scope.

Mountebank: Multi-Protocol Test Doubles

How it works: Mountebank runs on Node.js and uses what it calls imposters. Each imposter listens on a port and pretends to be a service. You create them through its REST API, and HTTP, HTTPS, TCP and SMTP are supported out of the box. It can proxy to a real service to record responses, and you can inject JavaScript when canned responses aren't enough.

When it made sense: That legacy system I mentioned. Part of it spoke a custom protocol over raw TCP, which ruled out most HTTP-only tools straight away. Mountebank's TCP imposters let us fake that piece well enough to test the newer services around it.

Good to know: The configuration style takes a little getting used to. And injection is powerful enough that it's easy to write test logic nobody else wants to debug, so I kept ours as small as I could.

Microcks: Mocks Straight From API Specs

How it works: Microcks takes your API artifacts (OpenAPI files, AsyncAPI specs, gRPC protobufs, GraphQL schemas, even Postman collections) and turns the examples inside them into live mock endpoints. It can also run contract conformance tests against your real implementation. It's built with Kubernetes in mind, though you can run it locally with Docker Compose.

When it made sense: A newer microservices project where the team was already spec-first. The OpenAPI files existed before the code did, so mocks generated from them let frontend and backend work in parallel from day one. It handled our Kafka event mocks too, which was a big plus.

Good to know: It's a bigger piece of infrastructure than a single mock server, so it pays off most when you have lots of APIs and specs to manage. Your mocks are only as good as the examples in your specs. Thin examples, thin mocks.

So Which One, When?

If you skipped straight to this part, no judgment. Here's roughly how I'd decide now.

Want regression tests and mocks generated from real usage, database calls included, without writing stubs by hand? Keploy. Mostly REST, and you want precise control over every response? WireMock. Can't touch the app's code and just need to capture and replay HTTP traffic? Hoverfly's proxy model is hard to beat. Stuck with TCP or SMTP alongside HTTP? Mountebank. Spec-first team with lots of APIs and event streams? Microcks.

Plenty of teams end up running two of these side by side, by the way. We had WireMock and Microcks together for a while and it was fine.

My Take After All of This

The tool matters less than people think. What matters more is picking one dependency that's genuinely hurting you and virtualizing that first. Teams that try to fake their entire architecture in week one usually give up by week three. I did, once.

So find the flakiest or most expensive service your tests lean on, try one of these against it for a sprint, and see whether your test runs get calmer. That'll tell you more than any roundup, this one included.

Parasoft Virtualize vs Modern Alternatives: How Service Virtualization Has Changed

Marx Jenes — Mon, 05 Oct 2026 13:02:00 +0000

A friend who's maintained a Parasoft Virtualize setup for the better part of a decade put it well: it was built for a world where a release happened every quarter, a dependency map fit on one slide, and the person configuring a virtual service was often a dedicated specialist with the time to do it properly. That world still exists in plenty of large enterprises, and Parasoft still serves it well. But a lot of teams evaluating service virtualization today aren't in that world anymore, and the gap between what they need and what the original generation of virtualization tooling assumed is exactly why parasoft alternatives for service virtualization has become such a common query.

Worth being specific about what actually changed, rather than treating this as a simple old-vs-new story.

What Parasoft Virtualize was built to solve, and still solves

Parasoft Virtualize is a mature, enterprise-grade platform for simulating dependent systems - APIs, databases, mainframes, message queues - in environments where those dependencies are expensive, unavailable, or risky to call directly. It supports a wide range of protocols beyond plain REST, which matters a lot to organizations running legacy systems alongside modern services, and it gives teams fine-grained manual control over exactly how a virtual service behaves, which some regulated or highly specific testing scenarios genuinely require.

Its strengths are real: broad protocol support, mature tooling for complex stateful simulations, and a track record in large, compliance-heavy environments where that maturity matters more than setup speed. None of what follows is an argument that it's a bad tool. It's an argument that it was designed for a specific shape of team and workflow, and a lot of teams today don't have that shape anymore.

What changed underneath it

Three shifts explain most of why teams go looking for alternatives:

Release cadence. Quarterly releases gave teams time to hand-configure a virtual service carefully and let it live for months before anything needed to change. Teams shipping multiple times a week don't have that slack - a virtual service that takes a day to update becomes a bottleneck, not a convenience.

Service count. A dependency map that used to fit on a slide now often spans dozens of microservices, each with its own API surface. Manually defining and maintaining that many virtual services, by hand, in a GUI-driven workflow, becomes its own full-time job at a certain scale.

Who's doing the configuring. The earlier generation of tooling assumed a dedicated specialist, often in a central QA or environments team. Modern teams increasingly expect individual engineers to own their own service's test environment, without needing to learn a separate platform's configuration language to do it.

None of this makes the older model wrong. It makes it a worse fit for teams whose release cadence, service count, and ownership model look different from what it was designed around.

The alternatives, and what each one changes

WireMock is a common lightweight starting point - open source, HTTP-focused, and simple enough that an individual engineer can stand up a stub server without needing a platform. It trades Parasoft's broad protocol coverage and enterprise tooling for speed and developer-level control, which suits teams whose dependencies are mostly REST APIs and who want virtualization to live close to the code rather than in a separate system.

Keploy addresses the maintenance side of the problem directly. Instead of manually defining virtual service behavior, it captures real API and database traffic at the network layer using eBPF and generates mocks and test cases from that captured traffic automatically. Because capture happens at the network layer, it works across different languages and frameworks without instrumenting application code, which matters for teams running a genuinely mixed stack. The practical effect is that updating a virtual service becomes a matter of recording traffic again rather than someone revisiting a configuration by hand, which is the specific pain point teams with frequent releases and many services tend to hit hardest with older tooling.

Hoverfly is another open-source option focused on HTTP service simulation, with support for capturing and replaying real traffic as well as manually defined scenarios. It sits between WireMock's simplicity and a fuller platform's feature set, and is often chosen by teams that want capture-based behavior without committing to a broader tool.

Testcontainers, while not a virtualization tool in the strict sense, is worth mentioning because it solves a related problem differently - running real instances of infrastructure dependencies (databases, queues) in containers rather than simulating them at all. For dependencies a team actually controls, this sidesteps the simulation-fidelity question entirely by using the real thing, scoped to a test run.

How to actually choose

The honest answer depends more on your situation than on which tool is "better" in the abstract:

Heavy legacy/mainframe protocol needs, regulated environment, dedicated environments team - Parasoft's maturity and protocol breadth are hard to replace with newer tooling.
Mostly REST APIs, small number of stable dependencies, want fast manual control - WireMock or Hoverfly fit well without much overhead.
Many services, frequent releases, a recurring problem with virtual services going stale between updates - a capture-based approach like Keploy addresses the maintenance burden specifically, rather than just making manual configuration faster.
Dependencies you actually own and can run in CI - Testcontainers sidesteps virtualization by using the real thing.

The actual shift

The real story isn't that newer tools are simply better than Parasoft Virtualize. It's that service virtualization has split into different tools optimized for different constraints - protocol breadth and regulatory maturity on one end, developer-level speed and low maintenance overhead on the other, with traffic-capture approaches specifically targeting the staleness problem that hand-configured virtual services have always had. Which one fits depends on how often your dependencies change, how many of them there are, and who's actually responsible for keeping the simulation honest.

What Makes an API Mock Trustworthy

Marx Jenes — Wed, 30 Sep 2026 08:49:47 +0000

A mock earns its place in a test suite the same way any other piece of infrastructure does: by being right often enough that nobody thinks to question it. That's also exactly what makes a bad mock dangerous. Nobody double-checks the thing they've stopped noticing. A mock that quietly stopped matching reality six months ago doesn't announce itself. It just keeps returning green, right up until the gap it's been hiding shows up somewhere much more expensive than a test run.

Worth being specific about what separates a mock that's actually doing its job from one that's just occupying the space where a real dependency test should be.

It reflects what the dependency actually does, not what someone assumed it does

The most common origin story for a mock is someone reading documentation, or remembering roughly how an API behaves, and writing a response that seems reasonable. That's a fine starting point, and it's also the exact spot where trustworthiness starts to erode, because documentation and memory both drift from reality in ways nobody notices until something breaks.

A trustworthy mock is built from observed behavior: an actual response the dependency actually returned, at some point, under real conditions. This matters more than it sounds, because real APIs are full of small inconsistencies that no one writes into a spec — a field that's sometimes null and sometimes just absent, a timestamp format that varies by endpoint, an error response shaped differently than the success response's documentation implies. A mock built from what actually happened carries all of that. A mock built from what someone assumed happened carries none of it.

It includes failure, not just success

Most hand-built mocks are optimized for the case that's easiest to imagine: the request that works. That's understandable, since the success case is usually what the person writing the mock was actually trying to test in the first place. It's also the least useful part of a mock to get right, because the success case is rarely where production bugs live.

The valuable part of a mock is what happens when the dependency times out, returns a malformed payload, hits a rate limit, or fails in some specific way particular to that service. These are the cases most likely to be missing from a mock, precisely because they require someone to think of them in advance and deliberately write them in. A mock's trustworthiness is really a question of how much of the dependency's actual behavior it captures, and failure modes are usually the majority of that behavior that goes uncaptured.

It doesn't quietly go stale

A mock is a snapshot. The moment it's written, it starts drifting away from whatever the real dependency is doing, at a pace nobody's tracking. This is the failure mode that's hardest to notice, because a stale mock doesn't look broken. It looks exactly like it did the day it was written. Tests built on it keep passing, confidence keeps building, and the actual dependency keeps changing underneath, unmonitored.

The trustworthy version of this isn't "write a good mock once." It's having a clear answer to who refreshes it, how often, and what would actually trigger that refresh - a scheduled re-check, a contract version bump, a captured-traffic pipeline that updates automatically rather than depending on someone remembering. Mocks that get treated as permanent fixtures are the ones most likely to be silently wrong by the time anyone looks at them again.

It's scoped to what it's actually simulating

A trustworthy mock does one job clearly: standing in for a specific dependency's specific behavior. Mocks lose credibility when they start absorbing logic that belongs somewhere else - business rules baked into a mock's response, conditional behavior that exists only to make a particular test pass, special cases added over time by different people for different reasons until nobody's sure what the mock is actually simulating anymore.

The cleanest mocks are the ones that map directly to something a real dependency did, with nothing added and nothing simplified away. The moment a mock needs its own internal comments explaining why it behaves the way it does, that's usually a sign it's drifted from representing the dependency into representing whatever made a specific test pass at some point.

It's reproducible

A mock that behaves differently between runs, whether from race conditions, uncontrolled randomness, or hidden state carried over from a previous test, undermines the entire reason mocks exist. The whole premise of replacing a real dependency with a mock is predictability: the same input should produce the same output, every time, regardless of what ran before it or what environment it's running in. A flaky mock is arguably worse than no mock at all, because it produces the appearance of determinism while quietly not providing it, and debugging a flaky test caused by a flaky mock is a particularly frustrating way to lose an afternoon.

Where this actually comes from in practice

Most of these properties point in the same direction: a mock's trustworthiness comes from how closely it's tied to real, observed behavior, and how deliberately that connection is maintained over time. Hand-written mocks can achieve all of this, but only with real ongoing discipline, since every property above requires someone to keep paying attention after the mock is first written.

This is the specific gap that dependency mocking as a tooling category exists to close - generating and maintaining mocks from real recorded behavior instead of leaving that entirely to manual upkeep, so freshness and failure-case coverage come from the process itself rather than from whether someone remembered to go back and update a stub file.

The actual test

A simple way to check whether a mock in your suite still deserves trust: could you explain, right now, where its behavior came from and how recently that source was checked. If the honest answer is "someone wrote it a while ago based on the docs," that's not automatically wrong, but it's worth treating as a question rather than an assumption. The mocks worth trusting are the ones where that question has a specific, recent, confident answer.

Testing Against a Dependency You Can't Call: When Service Virtualization Is the Only Option

Marx Jenes — Mon, 28 Sep 2026 05:46:06 +0000

A team I know integrated with a partner's inventory system that had exactly one environment: production. No staging, no sandbox, no test credentials. Their only way to "test" the integration was to send real requests to a live system that held real stock counts for real customers. For the first few months they did what most teams do in that spot. They wrote a mock by reading the partner's PDF documentation, and they trusted it. Then the partner returned a response with a field the PDF never mentioned, and an order flow that had been green in CI for months failed on its first live day.

Most service virtualization content treats it as an optimization, a faster or tidier way to run tests you could technically run against the real thing. This piece is about the other situations, the ones where calling the real dependency isn't merely inconvenient. It's off the table.

The situations where you genuinely can't call it

The dependency charges per call. Some third-party APIs bill for every request: credit checks, identity verification, geolocation lookups, SMS delivery. A regression suite that runs a few hundred times a day against one of these turns testing into a line item on the budget.

There is no test environment. Partner systems, government services, and older internal platforms often exist in production only. The other side has no interest in maintaining a sandbox for your integration, and you can't create one yourself.

The dependency is rate limited or quota bound. Running a full suite in parallel against a service that allows a handful of requests per minute gives you failures caused by throttling, not by your code. Those failures are noise, and noise trains teams to ignore red builds.

Another team owns it and it's unstable. If the service you depend on is mid-rewrite, down every other afternoon, or deployed on a schedule you don't control, your pipeline inherits its instability. Your tests fail for reasons that have nothing to do with your change.

The real thing has side effects you can't undo. Sending an actual email, triggering an actual payment, or creating an actual shipment during a test run is a problem no cleanup script fully solves.

In all five cases the choice isn't "virtualize or call the real service." It's "virtualize or don't test this integration at all," and the second option is how integration bugs reach production.

What a virtual service has to get right

Once service virtualization is the only route, the quality of the virtual service decides whether your tests mean anything. A few properties matter more than people expect.

Fidelity to real responses. A virtual service built from documentation reproduces what the documentation says. Real systems return things documentation leaves out: extra fields, inconsistent casing, empty strings where the spec promises nulls. The inventory story above is a textbook case. The closer the virtual service is to observed behavior, the fewer surprises reach production.

Error behavior, not just success. Happy-path responses are easy to simulate and rarely where integrations break. Timeouts, malformed payloads, partial failures, and odd status codes are what your error handling gets tested against, and they're also what a hand-built virtual service is least likely to include, because nobody thought to write them down.

State across a sequence. Many real interactions span several calls: create a record, fetch it, update it, fetch it again. A virtual service that returns the same canned response regardless of history will pass tests that a real system would fail.

Staying current. A virtual service is a snapshot. When the real dependency changes and the snapshot doesn't, tests keep passing against a system that no longer exists. This is the quiet failure mode of the whole category, and it's worth deciding up front who is responsible for refreshing it and how often.

Build it by hand, or derive it from real behavior

There are two broad ways to produce a virtual service. The first is to define behavior manually: read the documentation, write out the responses, and script whatever logic the interaction needs. This is quick to start and gives full control, and it inherits every weakness above. It reflects what the author believed the dependency does on the day it was written.

The second is to derive the virtual service from observed traffic: capture real requests and responses, then replay them. This addresses fidelity and error coverage directly, since the recorded behavior includes whatever the dependency actually did, including the odd cases nobody documented. Refreshing it becomes a matter of recording again instead of editing definitions by hand.

Keploy is one tool that works this way. It captures API and database traffic at the network layer using eBPF, without changes to application code, and turns that traffic into replayable mocks and test cases, so the simulated dependency reflects real recorded interactions rather than an author's assumptions.

The practical catch with any capture-based approach is that you need traffic to record. For a dependency you truly can't call from a test environment, that usually means recording from an environment that already talks to it, such as a development setup with limited access, or production traffic captured carefully and scrubbed of sensitive data before it becomes a test fixture. That scrubbing step deserves real attention. Recorded traffic can contain tokens, personal data, and account identifiers that must not end up committed to a repository.

A short checklist before you rely on one

Before trusting a virtual service in CI, it's worth asking a few plain questions:

Where did its behavior come from: documentation, someone's memory, or observed traffic?
Does it include failure cases, or only successes?
Can it handle a multi-step sequence, or does it return the same answer every time?
Who refreshes it when the real dependency changes, and how would anyone know it had drifted?
Is there any sensitive data baked into its recorded responses?

If most of those answers are vague, the virtual service is closer to a guess than a test double.

The point of all this

When you can't call a dependency, a virtual service stops being a convenience and becomes the only evidence you have that your integration works. That raises the bar on how it's built. The inventory integration at the start of this piece didn't fail because the team skipped testing. It failed because the thing standing in for the partner was a reasonable guess, and a reasonable guess is a fragile thing to build a release on.

Regression Test That Would Have Caught It, If Anyone Had Written It

Marx Jenes — Wed, 23 Sep 2026 13:45:27 +0000

We fixed a discount-stacking bug once - a promo code and a loyalty discount were combining in a way finance definitely hadn't approved - shipped the fix, closed the ticket, moved on to the next sprint. Eleven months later, a refactor of the pricing service reintroduced the exact same bug. Same root cause, same wrong output, same customers getting a discount they shouldn't have. Nobody on the team remembered the first incident. The person who fixed it originally had left the company. The only record that it had ever happened was a closed Jira ticket nobody thought to search before merging the refactor.

That's the entire argument for regression testing in one story, and it's a much narrower argument than "run the tests again before release." A regression test isn't a general-purpose safety net. It's a specific, permanent record that says: this exact thing broke once, here's proof it's not allowed to break the same way again.

Why the fix alone was never enough

Fixing a bug closes the incident. It doesn't prevent the next person from reintroducing it, because a fix by itself leaves no trace in the codebase that says why the old behavior was wrong. Six months later, someone refactoring that code has no way of knowing a previous version of this exact logic caused a real problem - they just see code that looks reasonable, changes it in a way that looks reasonable, and ships it.

A regression test is what turns "we fixed this" into "this can't happen again without someone finding out immediately." It's not really about testing in the abstract sense. It's closer to institutional memory that runs in CI instead of living in a person's head, which is exactly why it survives someone leaving the team and a ticket getting buried in a backlog nobody searches.

Why it gets skipped anyway

Nobody skips this step because they don't understand its value. They skip it because writing the regression test is extra work at the exact moment the incident feels resolved - the fix is deployed, the customer's been refunded, everyone wants to move on to whatever got deprioritized while the fire was being put out. Writing a test that specifically encodes "promo codes plus loyalty discounts shouldn't stack past X%" takes real thought: what's the minimal case that reproduces it, what's the assertion, does it need its own fixture or does it fit into an existing suite.

Under deadline pressure, that thirty minutes loses to whatever's next on the sprint board almost every time. The bug gets fixed. The proof that it was ever a bug doesn't get written down anywhere durable. This is an extremely normal way for teams to operate, and it's also exactly how the same bug comes back a year later wearing a different diff.

What actually closes the gap

The teams that do this well tend to treat "write the regression test" as part of the incident's definition of done, not an optional follow-up. Not every bug fix needs one - a typo in an error message doesn't need a permanent regression test guarding it forever - but anything involving money, data correctness, security, or a customer-facing failure earns one as a rule, no exceptions for how busy the sprint is.

The other thing that helps, less obvious but just as important: the regression test should be built from what actually happened, not a cleaned-up, idealized version of it. If the real bug involved a specific combination of a promo code, a loyalty tier, and a checkout retry after a timeout, the test should reproduce that specific combination, not a simplified "discounts shouldn't exceed X%" check that misses the exact conditions that caused the real failure. A regression test that tests a simplified version of the bug can pass while the actual bug is still very much alive.

This is where a lot of regression suites quietly lose their value over time - not because nobody's writing tests, but because the tests that do get written are approximations of the incident rather than reconstructions of it, written from memory a few days later instead of from what actually happened on the wire when it broke.

The suite that actually earns trust

A regression suite built this way - one entry per real incident, grounded in what genuinely happened, added as a non-negotiable step rather than a someday task - becomes something a team actually trusts, because every test in it has a story behind it. Not "we thought this might be a problem," but "this exact thing happened, and here's the proof it can't happen silently again."

The discount bug that came back eleven months later wasn't a failure of testing philosophy. It was a gap between "we fixed it" and "we made sure it can't happen again the same way," and that gap is exactly where regression testing is supposed to live. Closing it takes thirty minutes at the time it's most annoying to spend them. Not closing it costs a lot more than thirty minutes, just later, and usually to someone who wasn't even there the first time.

Playwright API Testing Plus E2E: Testing the Same User Flow at Two Layers, on Purpose

Marx Jenes — Mon, 21 Sep 2026 12:48:39 +0000

We had a signup flow that passed its E2E suite every single time a backend bug shipped that made new accounts default to the wrong pricing tier. The UI never noticed, because the UI didn't care what tier got assigned, it just cared that signup completed and redirected somewhere that looked like a dashboard. It took a support ticket from an annoyed customer to catch what an API-level assertion would have flagged in about four seconds.

That's the case for testing the same flow twice, on purpose, at two different layers. Not out of paranoia, and not because either layer is unreliable on its own, but because each one is structurally blind to a different category of bug.

Two layers, two different questions

An E2E test asks: does this work the way a person experiences it? Click, type, submit, does the right thing show up on screen, in the right state, without anything visibly breaking. It's the closest thing to ground truth for "does this feature actually work," and nothing else really substitutes for it.

An API test asks a narrower, sharper question: given this exact request, is the response correct? Right status code, right shape, right data, right side effects in whatever it touched. It doesn't care what any of that looks like rendered. It just cares whether the backend did the right thing.

The pricing-tier bug above is a clean example of the gap. The UI layer genuinely couldn't have caught it, because the UI never displayed the tier during signup at all — that information only showed up three screens later, in a part of the flow the E2E test didn't walk through. The API response, on the other hand, included the tier on every signup call. An assertion there would have caught it same-day.

Why one flow, tested twice, isn't redundant

The instinct to avoid this is understandable — testing the same user journey twice feels like doubling maintenance for no reason. In practice it's closer to doubling your blind-spot coverage than doubling your effort, because the two layers rarely fail for the same reason.

A flow can be API-correct and UI-broken: the backend does exactly the right thing, and a frontend bug displays it wrong, loses it in a bad render state, or fails to trigger the next step. Only the E2E test catches that.

A flow can be UI-correct and API-wrong: the frontend renders a perfectly reasonable-looking success state built on a response that's subtly incorrect underneath — wrong tier, wrong currency, a field that's silently null where it shouldn't be. Only the API test catches that, and usually catches it faster, because it's checking data directly instead of waiting for a person (or an E2E assertion) to notice something looks off on screen.

What actually changes when you build both, deliberately

This isn't "run every test twice." It's picking the handful of flows where the consequences of being wrong are actually expensive — signup, checkout, anything involving money or account state — and being intentional about covering each one from both directions instead of assuming one layer implies the other.

In practice this usually means the API-level version of the test goes deeper into the specific values and edge cases (what happens with an invalid promo code, what happens if a webhook fires twice, what the response looks like for every plan tier) while the E2E version stays focused on the handful of states that actually matter visually — does the confirmation show up, does the redirect happen, is there any state where the user gets stuck or sees something clearly broken.

Playwright api testing makes this pairing easier to actually maintain than it would be with two separate tools, since request-level API calls and full browser interaction live in the same test runner, the same fixtures, sometimes even the same file — which matters, because the version of this that never gets written is the one that requires switching tools and mental models every time.

The part that's easy to skip

The temptation, once both layers exist, is to let the API test quietly become the E2E test's setup step — using API calls to fast-forward through login or account creation so the E2E test can jump straight to the interesting part. That's a legitimate and common pattern for speed, but it's worth being honest that it's a different use of the API layer than testing the API layer as its own subject. A signup API call used to skip ahead in an E2E test isn't verifying the signup response is correct. It's assuming it is, so the UI test can move faster.

Both are useful. They're just not interchangeable, and conflating them is how a team ends up with heavy API usage across their suite and still no actual API-level assertions checking that the backend does the right thing on its own terms.

Where this earns its keep

The signup bug at the start of this piece is the shape of thing this pattern catches early instead of catching from a support ticket: something correct enough to pass the layer someone happened to test, and wrong enough to matter in the layer nobody checked. Testing a flow once tells you it works from one angle. Testing it deliberately from two tells you which angle, if either, is lying to you when something eventually breaks.

Sandbox Testing Tools for Teams Testing Against Real Dependencies

Marx Jenes — Wed, 16 Sep 2026 11:06:29 +0000

A team I worked near a while back had a "sandbox" environment that was really just a second production database with fewer permissions. Tests ran against it, passed, and nobody noticed the sandbox had drifted out of sync with what the real payment provider actually returned until a release broke checkout for a subset of users whose cards triggered an error path the sandbox had never simulated. The sandbox wasn't fake in the sense of being obviously unrealistic. It was fake in a quieter way — it just hadn't kept up.

That's usually the actual failure mode with sandbox testing, not "the sandbox is missing" but "the sandbox stopped matching reality months ago and nobody was checking." Worth going through what a sandbox needs to actually do its job, then a rundown of the tools teams commonly reach for.

What a sandbox testing tool actually needs to provide

Before comparing any sandbox testing tool, it's worth being specific about what "good" means here, because a lot of tools that call themselves sandboxes only solve part of this:

Isolation — tests shouldn't touch real production systems, real user data, or real third-party services with real consequences (real charges, real emails sent, real rate limits consumed).
Realism — the simulated environment needs to behave like the real one, including error cases, latency, and edge cases, not just the happy path someone remembered to configure.
Freshness — this is where most sandboxes quietly fail. A sandbox configured once and never updated drifts from the real dependency's actual behavior, the same way a hand-written mock does.
Reproducibility — the same test run against the sandbox should produce the same result every time, which is harder than it sounds once real network calls or shared state are involved.
Low setup cost — a sandbox that takes a day to configure per service doesn't get maintained, and an unmaintained sandbox is worse than no sandbox, because it gives false confidence.

With that as the bar, here's how the commonly used tools stack up.

Container-based sandboxes: Docker Compose, Testcontainers

Testcontainers and plain Docker Compose setups are probably the most widely used approach for sandboxing infrastructure dependencies — spinning up real instances of a database, message queue, or cache in a container, scoped to a test run, then tearing it down. This gets isolation and reproducibility right, since you're running the actual software, not a simulation of it.

Where this approach is weaker: it sandboxes infrastructure you control, but doesn't help much with third-party APIs or internal services you can't just spin up in a container — a payment provider, an external auth service, another team's API you don't own.

API mocking/simulation platforms: WireMock, Mockoon

For external APIs, WireMock and Mockoon are common choices — they let you define stub responses for HTTP endpoints and run them as a lightweight standalone sandbox server. Strong on setup simplicity and control; you can hand-craft exactly the responses and error cases you want to test against.

The weakness is the same one hand-written mocks always have: someone has to define every scenario manually, and nothing keeps those definitions in sync as the real API changes. A WireMock sandbox is only as realistic as the person who last updated it remembered to make it.

Contract-based sandboxes: Pact

Pact takes a different angle — instead of manually defining sandbox responses, it generates them from contracts agreed between a service and its consumers. This solves part of the drift problem, since a contract change is at least visible and versioned, but it depends on both sides maintaining accurate contracts, which is its own discipline that not every team keeps up consistently.

Traffic-capture-based sandboxes: Keploy

Keploy takes an approach closer to the "freshness" problem directly: instead of hand-defining sandbox behavior or relying on maintained contracts, it captures real API and database traffic at the network layer using eBPF and turns that captured traffic into a sandbox environment, along with test cases and mocks generated from it.

The practical difference this makes: the sandbox reflects what a dependency actually did, including real error responses, real latency characteristics, and edge cases that occurred in practice rather than ones someone thought to configure. Because capture happens at the network layer, it also works across different languages and frameworks without requiring code instrumentation in the service being tested, which matters for teams with a mixed-language stack where writing per-language stub definitions gets expensive to maintain consistently.

This doesn't make it strictly better in every situation — a Testcontainers-based sandbox is still the right call for infrastructure you control and want to run for real rather than simulate, and a hand-crafted WireMock stub is still faster to set up for a single, simple, rarely-changing endpoint. Where the traffic-capture approach earns its place is specifically the freshness and coverage problem: sandboxes that would otherwise need constant manual upkeep to stay accurate, especially across a distributed system with a lot of interdependent services.

Choosing between them

A rough way to think about it: use container-based sandboxes for infrastructure you own and can run directly. Use hand-defined mocking tools for external dependencies with a small, stable set of scenarios you're comfortable maintaining by hand. Use contract-based tools where you have real cross-team discipline around API contracts already. And where the actual problem is a sandbox drifting out of sync with a dependency's real behavior over time, especially across several services, a traffic-capture-based approach is worth prioritizing over one that depends on someone remembering to keep definitions updated.

None of these are strictly better than the others across every situation. The mistake worth avoiding is the one from that payment sandbox story: picking an approach once, without revisiting whether it's actually still matching reality six months later.