I want to talk about a specific kind of green build. The kind where 612 tests pass in 38 seconds, the coverage badge says 87%, and the thing is broken in production.
We had that build for most of a quarter.
The setup
We run a small logistics product. One of our TypeScript services talks to a third-party carrier API to fetch rates, book shipments, and pull tracking updates. When we built that integration, a colleague recorded a bunch of real responses, saved them as JSON fixtures, and wired them into our tests with nock. Fast, deterministic, easy to read. Everyone liked it.
That was in January.
How drift actually happens
Nobody decides to let mocks go stale. It happens in tiny steps.
The carrier added an optional surcharges array to rate responses. Our fixtures didn't have it, so our code never learned about it. Nothing broke. Yet.
Then they started returning estimated_delivery as a date-only string (2026-03-14) for some ground services instead of a full timestamp. Our code did new Date(value) and formatted it in the user's timezone. For anyone west of UTC, the delivery date showed up a day early. That one sat in prod for weeks, because who double checks whether "Thursday" should say "Friday"?
Then the big one. In April they moved tracking events behind pagination. The first call returned 20 events and a next_cursor. Our fixture had 6 events and no cursor, because the shipment we'd recorded back in January was a short one. In production, long-haul shipments stopped showing their latest status. Customers saw "In transit, Memphis" for packages that had been delivered two days earlier.
Every one of our tests passed through all of this. They were testing our code against January's API.
The afternoon I diffed everything
After the tracking complaint landed in our support channel ("customer says delivered, app says Memphis, again"), I blocked out an afternoon and did something slightly tedious. I wrote a throwaway script that sent the same requests behind each of our 34 fixtures to the carrier's sandbox, then diffed the real response shape against the saved one. Keys and types only, not values.
11 of 34 had drifted. Most trivially, a new optional field here and there. Three in ways that changed behavior.
I pasted the diff output in our team channel with no comment. Someone replied with the skull emoji, which was fair.
Where unit tests stop being useful
I don't think the lesson is "mocks bad". Our mocked unit tests were doing their actual job: given input X, does this function produce Y. The problem was that we were treating them as proof that our service worked with the carrier. They can't be that. A mock is a claim about someone else's system, and nobody was checking the claim.
That's the boring core of the whole unit testing vs integration testing argument, at least as I understand it now. A unit test isolates your code so you can trust its logic. An integration test puts your code next to the real dependency so you can trust the connection. You want both, and the trap is letting the fast, pleasant one stand in for the slow, annoying one.
We'd let the fast one stand in for everything.
What we do now
Nothing dramatic. Three changes.
A nightly drift check
That throwaway diff script became a nightly job. It hits the sandbox, compares response shapes to our fixtures, and posts to Slack if anything moved. It doesn't fail the build, because the sandbox is flaky enough that we'd be crying wolf twice a week. But we now see drift within a day instead of within a quarter.
Re-recording fixtures instead of hand-editing them
When drift showed up, we used to open the JSON and patch it by hand. That's how the January fixtures had turned into a weird hybrid of real and imagined responses. Now we re-record from the sandbox and commit the new file, so the PR diff shows exactly what changed upstream. Reviewers actually read those diffs, which surprised me.
Some teams push this further and record production traffic directly, then replay it as tests with dependencies mocked from the same recording. Keploy works roughly that way. We're not there yet, but fixtures that can't drift because they're re-captured from reality is a nice idea.
A thin layer of real calls
We added a handful of integration tests that run against the carrier sandbox on every PR that touches the carrier client. Seven of them. They book a shipment, fetch rates for three service levels, and pull tracking for a sandbox shipment with enough events to force pagination. About 40 seconds total, mostly because the sandbox is slow. We gate them on file paths so they don't run on unrelated changes.
Things I got wrong along the way
My first instinct was to replace all the nock mocks with sandbox calls. I got about halfway before the suite went from 38 seconds to just over six minutes and started failing randomly whenever the sandbox hiccuped. A teammate asked, very politely, whether I was trying to get people to stop running tests locally. I reverted most of it.
The other mistake: I assumed the carrier would loudly announce breaking changes. They did, kind of. Two of the three were on a changelog page I didn't know existed. The pagination change went out by email to an address belonging to someone who left the company last year. Go check where your vendors' notices actually land. I'll wait.
Where we ended up
612 unit tests, still mocked, still fast. Seven integration tests against the sandbox. One nightly drift check. Extra CI time on a relevant PR is under a minute.
The pagination test would have caught the tracking bug on its first run. The nightly diff would have flagged the date format the morning after it changed. Not perfect. Still a lot better than finding out from a customer whose package is apparently stuck in Memphis forever.
Top comments (0)