A payment-confirmation check fails on a Tuesday. The check is written as a UI journey: log in, add an item, pay with a test card, wait for the order status to read Paid. The report says expected "Paid", got "Pending" after 30s. That single line is consistent with the provider sandbox being slow, the webhook consumer not acknowledging the message, a dead-lettered event, an expired session, a feature flag, or someone renaming the status element. The report cannot tell you which, because the assertion never touched any of them. It touched a DOM node six hops downstream of the thing it was supposed to prove.
The rule I use to assign a check to a layer has two parts: which component can produce the failure, and which layer can observe that failure directly.
Cost and speed are consequences of getting that right, not the criterion for deciding. A check that asserts a backend rule through the UI is not "more thorough" — it is an assertion pointed at the wrong object, and you pay for the mistake every time it goes red.
The shape of the problem is visible in the before-state of BookThatApp, a Shopify appointment-scheduling app: UI tests only, fewer than 200 automated tests, one browser, one device, a single thread, no CI, and a regression cycle that took 4 days. Nothing there was lazy. It was one layer being asked to answer every question.
What actually breaks in an integration
"Integration test" is not a category. It is six different failure classes wearing the same label, and each one has a layer where it is cheapest to observe and hardest to misattribute.
- Schema breaks. A field is renamed, a type changes, an enum gains a member, null becomes possible. Observable in one request/response pair.
- Semantic breaks. The shape is intact and the meaning moved: an amount arrives in cents instead of units, a timestamp loses its timezone, status: "complete" now means captured rather than settled. No schema check catches this. It needs an assertion about business meaning.
- Timing and async breaks. The message is published but not consumed, consumed twice, consumed out of order, or retried into a duplicate side effect.
- Auth breaks. Token expiry, scope changes, clock skew, a rotated key on the provider side.
- Third-party behavior. The dependency starts returning a new error code, rate-limits you, or changes its own retry semantics.
- Data drift. The fixture account got closed, the sandbox reset, the seeded record now has a state nobody expected.
Through the UI, a schema break and a semantic break produce the same artifact: a wrong number on a screen. At the API layer they are two obviously different failures — one is a missing key, the other is a wrong value with a right key.
Tipalti, a cloud AP-automation platform handling high-volume international payments, is a useful illustration of how many transports one product can carry at once. The QA toolchain spans REST (Postman, Swagger), GraphQL (Insomnia), SOAP (SoapUI), RabbitMQ, SQL databases and MongoDB — alongside Selenium WebDriver, Capybara and Cucumber for the UI. That is at least four protocol surfaces where a schema or semantic break can originate. The case study lists tools and scope, not which class of failure each one caught, so read it as evidence of surface area, not as a measurement.
The question that assigns a layer
Before writing a check, ask: if this fails, what will it have proven?
If the answer names a backend rule — "a refund above the daily limit is rejected", "a duplicate webhook does not create a second ledger entry", "a cancelled subscription stops accruing charges" — then the check belongs at the API or service layer. An assertion has to touch the thing it asserts about. When it does not, every failure carries a diagnostic tax: someone re-runs it, someone watches a video of a browser, someone eventually opens a log and finds a 422 that a two-line API test would have printed in the failure message.
Concretely, this is the difference between:
UI assertion pointed at a backend rule
visit "/account/refunds"
fill_in "amount", with: "5000.00"
click_button "Request refund"
expect(page).to have_content("Refund request submitted")
and:
same rule, asserted where it lives
res = api.post("/v1/refunds", amount_cents: 500_000, order_id: order.id)
expect(res.status).to eq(422)
expect(res.body["error"]["code"]).to eq("daily_refund_limit_exceeded")
expect(res.body["error"]["limit_cents"]).to eq(250_000)
expect(db[:ledger_entries].where(order_id: order.id).count).to eq(0)
The second version fails with the reason attached, and it also asserts the thing the first one silently skipped: that nothing was written. UI success messages are famously willing to appear over a backend that did nothing.
BookThatApp moved from UI-only to a keyword-driven suite of 1,000+ scripts with API tests running alongside UI, on GitHub Actions, across 5 device configurations, 4 browsers and 4 parallel threads, at >80% coverage. Regression went from 4 days to roughly 2 hours. Attribute that honestly: parallelisation, CI and the rewrite all landed together, so the duration is not a measurement of layering. What layering changed is what a red build tells you.
The counterargument holds in a narrow band. Some rules genuinely have no API surface — rendering, layout, client-side permission logic expressed only in the front end, anything where the browser is the system under test. Those stay up top, and the next-to-last section is about exactly that remainder.
Queues and asynchronous flows
This is where UI-shaped checks do the most damage. With a broker in the path, the screen is the last thing to know, and "wait until the page updates" quietly converts every broker problem into a timeout.
The flakiness data lines up with that. In a 34-month study of 559 flaky-test issue reports at SAP HANA covering 2023–2025, timeouts accounted for 18% and async wait for 10% of system-test flakiness — roughly a quarter of it timing-related — while async wait was the root cause of only 2% of native unit-test reports. An earlier analysis of 452 flakiness-related commits across 40 top-starred JavaScript projects put concurrency and async wait together at 40.3% of categorised causes. Timing flakiness is a property of how far the assertion sits from the event.
So assert on the message and on the consumer's resulting state, not on a screen that eventually updates:
- Publish and inspect. Assert the message hit the exchange with the right routing key, headers and payload.
- Consume and inspect state. Assert the consumer's write — a row, a document, a projection — not a rendering of it.
- Treat ack, retry and dead-letter paths as first-class cases, not error handling. Publish a poison message; assert it lands in the DLQ with the failure reason and that the main queue drained.
- Assert idempotency explicitly. Redeliver the same message with the same ID and assert one side effect, not two. Brokers redeliver; that is normal operation, not an edge case.
- Bound your polling. Never sleep.
def await_state(fetch, predicate, timeout=15.0, interval=0.25):
deadline = time.monotonic() + timeout
last = None
while time.monotonic() < deadline:
last = fetch()
if predicate(last):
return last
time.sleep(interval)
raise AssertionError(f"state never reached after {timeout}s; last seen: {last!r}")
publish(exchange="payments", key="payment.captured", body=evt, headers={"x-idempotency-key": k})
row = await_state(lambda: db.fetch_payment(evt["id"]), lambda r: r and r["state"] == "captured")
assert row["amount_cents"] == 500_00
assert queue_depth("payments.dlq") == 0
The failure message names the state that never arrived. That is the whole point.
Logs and metrics are legitimate assertion surfaces here too — if a consumer emits a payment.duplicate_suppressedcounter, asserting that counter incremented is a cleaner proof of idempotency than inferring it from a ledger. On Tipalti, the toolchain includes RabbitMQ alongside Grafana, Kibana, Elasticsearch, Prometheus and Coralogix, plus custom microservice-level tooling ("Microtools", "QAtools") and an admin dashboard. The case study does not describe assertion strategy, so treat the method above as method — the stack only shows that queue-level and observability tooling were part of the QA surface, not decoration.
Dependencies you don't control
The stubbing decision is not "mock everything" or "mock nothing". Split it by ownership: stub for the logic you own, hit the real sandbox for the contract you don't — and run the second one on a separate cadence, because it fails for reasons unrelated to your commit.
A sandbox failing at 3 a.m. is information about the provider. Mixed into your PR pipeline, it is just a red build someone re-runs. Two schedules, two meanings:
.github/workflows/api.yml — every PR, stubs only, deterministic
on: [pull_request]
env: { STRIPE_BASE_URL: http://localhost:12111 } # local mock
.github/workflows/provider-contract.yml — nightly, real sandbox, own alert route
on:
schedule: [{ cron: "0 3 * * *" }]
env: { STRIPE_BASE_URL: https://api.stripe.com }
Reasons to keep the sandbox run at all: sandboxes diverge from production, rate limits only appear against the real endpoint, and sandbox state is non-deterministic in ways your stub will never reproduce.
Across these engagements the tooling reflects that split, though no case study states a stubbing policy — read the following as tools present, not as documented decisions. Dootrix, a UK EV-charging network app, integrates the Stripe API and runs 50+ backend automated integration scenarios using REST Assured and RestSharp, with Charles Proxy in the toolchain. A cryptocurrency wallet platform supporting swaps and dApp interaction across 20+ blockchains uses Hardhat — local simulation of a protocol-level dependency — plus ~20 load tests built against RPC proxies for roughly 20 networks. Favro, a project-management platform, supports 5 integrations with Braintree, Okta and OneLogin in the stack, and reports ~90% of customer-reported issues identified, reported and resolved over an engagement running since 2019 with 2 manual QA engineers.
Contract testing: what it does and does not replace
Contract checks are cheap and narrow. They catch shape and compatibility breaks — a removed field, a changed type, a provider response that no longer satisfies a consumer's expectations. They prove nothing about behavior, ordering, or business rules. A provider can satisfy every contract in your repository and still capture the wrong amount.
Adoption reflects the narrowness. In Postman's 2025 State of the API report, contract testing sits at 17% against 67% for both functional and integration testing and 57% for performance testing — the weakest area on the list. The same report puts REST at 93%, webhooks at 50%, GraphQL at 33%, and finds 60% of teams version their APIs while only 26% use semantic versioning. That last pair matters: consumer-driven contracts pay off when you have many independently deployed consumers and the versioning discipline to enforce a break. Without the second half, a contract suite is a shape linter.
Worth noting as an observation rather than a recommendation: none of these engagements report a contract-testing tool. What they built was API automation. At SoftNAS — a storage product tested across NFS, CIFS/SMB, iSCSI and AFP on Windows and Linux, with VMware and Azure CLI in the loop — there was no API testing at all before the engagement, and "a low-level API automation test suite was developed from scratch", reaching 80% coverage with 1,000+ automated scripts and 200+ bugs reported, 30% of them critical. The case study also reports 45% faster delivery of automation results and 25% fewer issues, both without a stated baseline, so treat those two as directional only.
What is left for the UI
A small, expensive remainder: things no lower layer can observe. Rendering, layout across breakpoints, localisation, client-only permission logic, and the handful of journeys where the composition itself is the risk.
They carry the highest maintenance rent, and that rent is measurable. Grover, a technology-rental platform operating across 5 markets, runs 600+ automated scripts at 90% coverage with 300+ test cases and 4 localizations; the suite was migrated from Ruby to Java with all logic rewritten, and has been revised twice in response to changes in the primary UI. Regression there still takes 1–2 days, with smoke at 3–4 hours. Two rewrites is what a UI-weighted suite costs when the product's front end moves.
Dootrix runs 100+ mobile automated scenarios against those 50+ backend scenarios, with 400+ test cases across 20+ devices. That 100:50 split reflects a mobile-first EV-charging product, not a target proportion — do not copy the ratio.
The cryptocurrency wallet engagement makes the sharper point about sequencing. Before the work started there were no backend automated tests, no backend test cases, no CI/CD integration, no scheduled runs, no parallel threads and no load tests. What exists now is ~600 backend automated tests and ~1.2k backend test cases running in ~40 minutes across 10 parallel threads on GitHub Actions — while the mobile automation framework is still in development. Substantial integration confidence was accumulated with the UI layer unfinished. UI coverage is not a precondition for it. (The 40-minute figure belongs to parallelism, suite size and scope as much as to layering — it is not a backend-first benchmark.)
What splitting layers actually costs
Layering buys diagnosis and speed. It spends them on:
- Setup complexity. Per-layer environments, seed data and teardown. An API suite needs its own fixtures; a queue suite needs a broker and a way to reset it between runs.
- Suite count. More suites, more owners, more CI configuration, more places for a flaky test to hide.
- Drift. A stub that stops matching the live contract is worse than no stub, because it is confidently wrong. That is what the nightly sandbox run is for.
- Gaps. The failure mode of layering is a green API suite over a broken user path, because the seam nobody assigned to a layer is the seam nobody asserts.
And it does not remove human testing. Tipalti has run for 12 years with roughly 20 manual QA engineers alongside 1 automation QA engineer, producing 7,500+ test cases, 200+ test plans, 100+ regression tests, 50+ performance automated tests and 12,000+ reported bugs — of which 5,000+ were critical, blocker or major — across 250+ releases. Regression still takes 2 days at a cadence of 25 releases a year. A layered strategy at scale looks like that, not like a pyramid diagram.
Apply it to three checks you already have
A queue consumer. Origin: the consumer. Observable: the message and the resulting row. Publish directly, poll the consumer's state with a bounded wait, assert the DLQ is empty, then redeliver the same message and assert one side effect. No browser involved.
A payment provider call. Origin: split. Your amount, currency and idempotency handling are yours — stub them and assert on every PR. The provider's response contract is not — hit the sandbox nightly, on its own alert route, so its outages stop reading as your regressions.
A rendered total. Origin: the client. Observable only in the client. Assert the computation at the API layer, and keep exactly one UI check that the correct value reaches the screen in the correct format for each locale you support.
The reflex worth dropping is "it isn't really tested until it's tested end to end." End-to-end is one observation point among several, and it is the one furthest from every cause. Use it where it is the only place a failure becomes visible, and stop asking it to explain failures it can only report.
Top comments (0)