How to Build a Resilient Integration Layer for Dozens of External APIs

A carrier's label API times out at ten seconds. Your HTTP client throws, your retry wrapper fires, the second request lands — and the vendor processed both. The customer now has two shipping labels on one parcel. Swap the carrier for a payment provider and it's two charges. Nothing in your dashboards looks wrong, because from your side every request eventually "succeeded."

That incident is what a uniform retry policy buys you once third-party API integration becomes the product.

Because integrations don't fail the same way:

• Some gate a flow and can't degrade — KYC checks, custody, payments.
• Some can serve stale data for an hour and nobody notices — prices, metadata, enrichment.
• Some cause things to happen in the physical world — labels print, locks open, drivers dispatch.

My thesis for this whole article: resilience in an integration layer comes from classifying each vendor by what breaks when it fails, then isolating it — its own adapter, its own data model, its own timeout budget, its own cached state — so the failure stays local. The thing that scales worst is the thing most teams build first: one canonical interface with one retry-and-circuit-breaker policy applied to everything.

Count integrations by blast radius, not by vendor

When someone says "we have 30 integrations," that number tells me nothing. The useful inventory is by blast radius, and in my experience it always sorts into three classes.

1. Critical-path, can't degrade. On Mode, a UK digital finance app dealing in GBP, EUR and BTC, Onfido gates onboarding and Chainalysis and Sardine.ai gate compliance checks — if those calls fail, the flow stops, full stop. Coinbase and Modulr sit in the money movement path. There is no "serve stale" option for any of them.
2. Degradable. Mode's Twilio integration sends notifications; a delayed SMS is an annoyance, not an incident. Token prices, asset metadata, enrichment data — all of this can be served from cache while the vendor is down.
3. Side-effecting on the physical world. A shipping label, a door lock, a dispatch call. These are the scariest, because a duplicated call isn't a wasted request — it's a duplicated event you can't un-happen.

Each class gets different rules. In practice that looks like config, not philosophy:

const policies = {

critical:    { timeoutMs: 8000, retries: 2,  fallback: 'fail-closed' },

degradable:  { timeoutMs: 1500, retries: 0,  fallback: 'serve-cached' },

sideEffect:  { timeoutMs: 15000, retries: 0, fallback: 'park-and-reconcile' },

} satisfies Record;

Note the retries on the last two: zero. More on that below.

The honest limitation: classes shift under your feet. A price feed is degradable while it renders a chart and critical the moment it prices a trade. The classification is per call site, not per vendor — and it needs re-checking every time a new feature reuses an old integration.

One adapter per provider, one owner per data class

The structural rule I hold to: every provider — or every data class, where that's the better seam — gets its own adapter that owns:

• the translation to and from vendor types,
• the auth and credential handling,
• the rate limits,
• the error taxonomy.

Nothing downstream ever sees a vendor's types. This is the anti-corruption layer pattern, and Microsoft's own writeup is upfront that it costs you latency and an extra service to maintain — it's still the cheapest line item in this article.

DEFI Wallet, a multichain Web3 wallet a 9-person Devico team has been building since 2021, is split exactly along data-class seams:

• a Python service for token metadata,
• a Node.js/Nest.js service for asset data,
• a separate service for NFT balances,
• a GraphQL router on top, combining the service schemas.

The vendor list underneath is long — Infura, RPC nodes, DefiLlama, THORNode, Moralis, Covalent, CoinGecko, TrustWallet API — and no consumer of the graph touches any of them directly.

GoodShape, a UK wellness platform, supports 16 different fitness-tracker brands — an integration surface that became tractable after a migration out of a roughly 15-year-old monolith into microservices, where tracker data collection lives as its own service on RabbitMQ and AWS SQS.

Per-vendor adapters also give you bulkheads for free: one slow vendor exhausts its own connection pool and its own worker budget, not the shared one your payments path is standing in.

The cost is real, though. Dozens of adapters means dozens of deploy units, credential sets and dashboards. Don't pretend otherwise when you pitch this to your team.

Where a gateway helps — and where it hides failure

A router or gateway in front of the adapters earns its place by deciding what the caller sees: which fields compose into one response, what happens when a sub-service is down.

That's also its failure mode. A GraphQL router combining schemas is a coupling point where a partial outage flattens into a null field — the NFT service is down, the wallet renders an empty tab, and nobody pages, because every layer "handled" the error.

A gateway decides what the caller sees. It does not make the underlying call resilient. If your resilience story is "we have a gateway," you have a single point of failure with good manners.

Reads: get vendors off the request path

The highest-leverage change I know for read traffic: stop proxying. Every synchronous pass-through call donates your vendor's latency and downtime directly to your user.

The alternative is to ingest — pull vendor data into your own store on your own schedule, serve every user request from that store, refresh asynchronously. Cache, materialized view, indexer, full pipeline: the mechanism varies, the principle doesn't. The vendor's availability stops being your availability.

DEFI Wallet takes this to the logical end: rather than reading chains per request, the team built blockchain indexers for real-time transaction processing, so the wallet reads its own indexed state.

Sway, a US reverse-logistics company (ex-Returnmates), does the same for reporting: a pipeline from Node.js through Google Cloud Storage into Snowflake, surfaced in Sigma — not live queries against operational systems. On the operational side, PostgreSQL indexing and materialized views were part of a rework that Sway reports cut scan times from 3–5 seconds to 1–2 seconds and improved query speed by at least 40%. Those figures come from the case study, and they cover a bundle of work — the mobile app, DB indexing, refactored parcel creation, queues — so I won't pretend they isolate the read-path change. The architecture, though, is exactly the shape I'm describing.

What you inherit in exchange:

• staleness windows you now have to define per data class,
• backfills for whenever ingestion falls behind,
• a reconciliation job that never quite finishes.

For prices and metadata, stale is fine. For balances, compliance states and anything money-shaped, stale isn't "old data" — it's wrong data, and this pattern doesn't apply. That's the classification from section one doing its job.

Writes: idempotency before retries

Here's the assumption I most want to break: that retry logic is where write-path resilience starts. It's where it ends.

Order of construction on any side-effecting integration:

1. An idempotency key on every outbound write.
2. A persisted intent record.
3. A reconciliation job.
4. Then retries.
5. Only then a circuit breaker.

Without the first three, a retry converts a lost response into a duplicate action.

Stripe's API is the reference design worth copying: the client sends an idempotency key, the server saves the status code and body of the first request for that key — success or failure — and returns the same result on every replay, pruning keys after at least 24 hours. That contract is what makes a retry safe: the vendor guarantees the second request can't act twice.

Your side of the contract is the intent record — persist what you're about to do before you do it:

INSERT INTO outbound_intents (idempotency_key, vendor, action, payload, status)

VALUES ('parcel-19f3-label', 'carrier-x', 'create_label', $1, 'pending');

• - call the vendor with the same key; mark 'confirmed' only on response.
• - A reconciliation job sweeps 'pending' rows and ASKS the vendor
• - what happened before anything retries.

The row that stays pending after a timeout is precisely the case from this article's first paragraph: you don't know what happened. Reconciliation — query the vendor, then decide — is the only honest answer. Blind resend is a coin flip.

Webhooks force the same discipline from the other direction. Stripe's docs state plainly that:

• endpoints might receive the same event more than once,
• event order isn't guaranteed,
• delivery retries run for up to three days with exponential backoff.

Their recommended defense is tracking processed event IDs — idempotency again, inbound this time. And with a three-day retry window, "we'll have processed the earlier event by then" is not a safe assumption at any point.

The stakes scale with how physical the side effect is. Sway generates carrier labels and dynamic ZPL for Zebra printers across FedEx, ClearJet, Million, T-Force, SmartKargo and UDS — six carriers, each a place where a duplicated call is a real label on a real box. Space Plus, a staffless self-storage operator, issues digital keys through Sensorberg's IoT platform — a retried unlock is a door opening. To be clear about attribution: those case studies document the integrations, not the idempotency machinery; the machinery is my recommendation for anyone operating that class of integration.

The gap you'll hit: plenty of vendors offer no idempotency key at all. Then your dedupe needs a natural key — order ID plus carrier plus day, say — and natural keys are ambiguous in exactly the edge cases that matter, like a legitimate second label for a split shipment. Budget design time for that key; it's harder than the plumbing.

When the retry is the bug

Retries also fail at the aggregate level, and there's good data on how. Google's SRE book works through the arithmetic: a backend at capacity starts rejecting requests, clients retry, and the retries themselves push the backend further past capacity — in their worked example a 10,000 QPS service ends up receiving 10,200 QPS precisely because it's failing. Their guidance:

1. Randomized exponential backoff, always.
2. A hard cap on retries per request.
3. A server-wide retry budget — their example is 60 retries per minute, after which requests just fail.

The multiplier that catches microservices teams: retries stack per layer. Three attempts at each of four layers is up to 64 attempts hitting the bottom, per the same chapter. If your adapter retries, your gateway must not.

That's why retry policy is a per-write-path decision, not a framework default. A blanket "3 retries with backoff" in your HTTP client is the single config line most likely to charge a customer twice.

Aggregators vs. direct integrations

At some vendor count, someone proposes an aggregator: one contract, one API, many providers behind it. It's usually decided by reflex — either "obviously yes, less code" or "obviously no, we control our stack." Both reflexes skip the actual trade.

What an aggregator buys you:

• one contract and one integration surface,
• someone else maintaining the long tail of providers.

What it costs you:

• one failure domain where you used to have several,
• one shared rate limit,
• someone else's semantics for everything behind it — their error taxonomy, their timeout behavior, their idea of what a "declined" means.

Space Plus is a reasonable picture of the pragmatic middle: payments route through Adyen, alongside PayPal, AmazonPay and Sofort, plus a Store365 CRM integration — a setup the case study describes as a hybrid of bespoke build and off-the-shelf integration. Aggregate where the aggregator's domain is mature; go direct where the vendor is load-bearing enough to deserve its own adapter.

The overlapping-provider version of this problem is subtler. DEFI Wallet pulls asset and market data from several providers whose coverage overlaps — Moralis, Covalent, CoinGecko, DefiLlama and others. I'll flag what the source does and doesn't say: it lists the integrations; it doesn't describe a failover strategy, and I won't invent one. But anyone who's run overlapping providers knows the operational truth: two providers of "the same" data return different numbers, and deciding which one your product displays is a product decision wearing an engineering costume.

Vendor drift is an operational problem

Everything so far is architecture. The failures that actually page you a year in are operational: integrations rot silently.

• A vendor tightens a rate limit without an announcement.
• A deprecated field starts returning null.
• A schema gains a field your parser chokes on.
• Semantics move under a stable version number.

None of this throws a 5xx, so none of it trips a circuit breaker — your integration is "up" and wrong.

What catches it:

• per-vendor contract tests running against recorded responses,
• alerts on response shape and volume anomalies, not only on error rates,
• a habit of treating a vendor changelog like a dependency upgrade.

A breaker knows the vendor is down. It cannot know the vendor is lying.

Sandboxes, UAT and release cadence at hundreds of integrations

Past a certain integration count, keeping validation running is a team-topology problem, not a tooling one. MINT, an adtech resource-management platform, operates 16 channels and 350+ vendor integrations; the working setup there was organizational:

• 14 independent teams, each with its own sandboxes,
• QA embedded in the teams rather than centralized,
• a dedicated UAT environment.

That replaced release-approval bottlenecks with multiple releases a day. The relevant lesson is about process: at that vendor count, a centralized release gate is itself the outage.

One warning from hard experience: sandboxes lie. Plenty of vendor test environments differ from production in latency, rate limits, and occasionally behavior. A green sandbox suite tells you your code matches the vendor's documentation — production tells you whether the documentation was true.

What this costs

None of this is free, and I'd rather you hear it from me than from your own on-call rotation.

• Per-vendor isolation multiplies code, deploy units, credentials, dashboards and alert routes.
• Caching reads buys you a reconciliation problem in perpetuity.
• Idempotency adds a storage table, a dedupe path and a sweeper job to everything that writes.
• Contract tests are a suite someone has to own forever.

Below roughly five integrations, most of this is overhead — a well-tested direct integration with sane timeouts beats a prematurely general integration layer. The flip point is easy to name because you'll feel it: the first time one vendor's outage takes down a feature that had nothing to do with that vendor, you've outgrown the shared plumbing.

So the order of effort, when you get there:

1. Classify every integration by blast radius.
2. Isolate each vendor behind its own adapter.
3. Move reads off the request path.
4. Make writes idempotent.
5. Only then add the retries and circuit breakers everyone starts with.

Each step makes the next one safe; run in reverse order, the list is a duplicate-payment generator.

And keep the last dollar for operations. The integration that breaks in production is rarely the one that went down — it's the one that changed on the vendor's side six weeks ago, while nobody was watching the response shape.