Last spring I spent a Thursday afternoon explaining to our support lead why customers in Ontario couldn't finish a purchase while our CI dashboard showed nothing but green. It wasn't a fun conversation, and it changed how I think about what a test is supposed to prove.
What we were actually testing
Our checkout service had decent coverage. Around 80% line coverage, a few hundred unit tests, and a pipeline that ran in under four minutes. The tax calculation module alone had 40 tests. Every one of them passed.
The problem was that those tests checked functions, not behavior. calculateTax() returned the right number for every province we fed it. What nobody had checked was whether the checkout endpoint ever called calculateTax() with the province the customer actually selected. After a refactor of the address form, the frontend started sending the province as a two-letter code in a field called region instead of province. The backend quietly fell back to a default, the default had no tax rule for Ontario, and order validation rejected the total.
Every unit test was correct. The feature was not.
The question I started asking
After that week I started asking one question before writing any test: if this passes, what does a user get to do?
For a unit test the answer is usually "nothing directly." It proves a piece works in isolation. That's useful, but it isn't the thing the customer cares about. A functional test answers a different question: given this input at the edge of the system, does the system produce the output the requirement describes? If you want a longer breakdown of how that differs from other layers, this write-up on functional testing covers the main types and where each one fits. The short version that stuck with me is that a functional test doesn't care how the code is organized. It cares whether the behavior matches the spec.
So for checkout, the functional test isn't "does calculateTax return 13% for ON." It's "when a customer with an Ontario address submits a cart, do they get an order confirmation with the correct total."
How we rebuilt the suite
We didn't throw away the unit tests. We added a thin layer above them, and I want to be specific about what it looked like, because "add functional tests" is advice that sounds obvious and gets ignored.
Start from the requirement doc, not the code
I pulled up the original ticket for checkout and listed every sentence with the word "should" in it. There were 23. Things like "the customer should see shipping options based on their postal code" and "a cart with a discontinued item should fail with a clear message." Each of those became a test case. Some needed three or four variations (different provinces, empty cart, expired coupon). We ended up with about 60 cases.
Hit the API, not the functions
Our checkout is a set of REST endpoints, so most of the new tests sent real HTTP requests to a running instance of the service with a seeded database. This is where API functional testing earns its keep. You're validating the contract the frontend actually depends on: field names, status codes, error shapes, and the business result. The region vs province bug would have failed on the very first run because the response total would have been wrong.
Assert on outcomes a person would notice
I made a rule that every functional test had to assert on at least one thing a person would see. Order total. Error message text. Whether an email job got queued. Asserting only on a 200 status code isn't a functional test, it's a ping.
What it cost us
More than I expected at first. The functional suite took about 9 minutes to run, which more than doubled our pipeline time. We had flaky failures for the first two weeks because our seed data had timestamps that drifted. We fixed that by freezing time in the test environment.
Maintenance was the bigger cost. Writing request payloads and expected responses by hand for 60 cases is tedious, and every time the API changed someone had to update a pile of JSON. We tried a few approaches: hand-written pytest suites using requests, Pact for the contract between frontend and backend, and a record-and-replay tool (Keploy) that captures real traffic and turns it into test cases. I'm not going to tell you one of them is right, because it depended on the endpoint. Pact was great for the frontend contract. Recorded traffic was great for the gnarly endpoints with big nested payloads that nobody wanted to write by hand. Plain pytest stayed best for edge cases we had to construct on purpose.
What changed afterward
In the eight months since, the functional suite has caught 11 bugs before they shipped. I counted. Seven of them had the same shape as the Ontario bug: every unit passed and the wiring between them was wrong. Two were config issues that only showed up when the service read real environment variables. Two were me.
It also changed code review. When someone opens a PR now, the reviewer asks "which functional case covers this?" If the answer is none, either we add one or we write down why it doesn't need one.
What I'd tell my past self
Coverage percentage told me how much code ran during tests. It never told me whether a customer could buy something. Those are different numbers, and I spent a long time treating them as the same one.
If your suite is mostly unit tests and you've never had an incident like ours, you might just be lucky. Pick your most important user flow, write down what it should do in plain sentences, and turn each sentence into a test that goes through the front door. You'll find out quickly whether your green dashboard means what you think it means.
Top comments (0)