DEV Community

xanderblack5716
xanderblack5716

Posted on

Warm Fallback Provider Tests and Billing Attribution During a Live API Key Rotation

#api #billing #architecture #observability

Pinning every request to one vendor produces the cleanest cost attribution you will ever have, and it is the same decision that quietly turns that vendor into a single point of failure. Spreading traffic across a default route with a second provider behind it inverts the pair: concentration risk falls, and your per-customer numbers go soft the moment a request lands on the fallback without anyone noticing. In short, keep the default route and buy the attribution back — make every response say which vendor served it and which credential generation signed it, then write both into your own ledger before the request closes.

The system worth holding in mind here is ordinary. A property management platform, 400 management companies on it, somewhere north of 60,000 doors, and a third-party API doing applicant screening, owner-statement rendering and maintenance dispatch. The platform rebills that API spend per management company. Attribution is therefore not a dashboard nicety; it is a line on an invoice that a regional operator with 90 doors will eventually dispute, in writing, eleven months later.

And the production key has to rotate without a maintenance window.

Why a credential rotation and a warm fallback are one problem

Both events change the identity of whatever served your request, in flight, without a single line of caller code changing with them. A rotation swaps the credential; a fallback swaps the vendor. In both cases the request still succeeds, the customer still sees a screening report, and the only artifact that records what actually happened is whatever you chose to persist at the moment of the response.

Month-end reconciliation is where the gap shows. You receive one invoice per vendor, each aggregated their way, and your ledger has to sum to those invoices per vendor before you can rebill anything. If the ledger stores only tenant and timestamp, every request the fallback absorbed is attributed to the primary vendor's rate card — a rate card that doesn't apply to it. The error is not random, either, which is the part that hurts: fallback traffic clusters around incidents, incidents cluster around a handful of loud customers, and the mis-attribution lands disproportionately on exactly the accounts most likely to audit their bill.

A credential generation label does the same work for rotations. During an overlap window two credentials are valid at once, and if the provider prices or meters them separately — many meter per key, because that is how they scope quotas — then a bill split across two keys reconciles against a ledger that thinks there was only ever one.

So the attribution tuple I'd defend in a design review is four fields wide: tenant, vendor, credential generation, and a synthetic flag. Everything else in this article follows from keeping those four fields honest.

Two valid credentials and the overlap window that makes rotation boring

Zero-downtime rotation has one mechanical requirement: a period during which both the outgoing and incoming credentials authenticate. Create the new secret, distribute it, flip the default, verify, then revoke the old one — and do the revocation as a separate, scheduled, reversible step rather than as the last line of the deploy script.

The pattern is well trodden. AWS Secrets Manager models it with staging labels, where AWSCURRENT, AWSPENDING and AWSPREVIOUS point at different versions of the same secret so consumers can move across the boundary at their own pace. HashiCorp Vault's KV v2 engine reaches the same property from a different angle by keeping prior versions addressable by number. The OWASP secrets management guidance argues for automated, frequent rotation on the grounds that a manual rotation is one nobody performs; NIST SP 800-57 Part 1 frames the same question as a cryptoperiod, which is a more useful way to think about cadence than picking a round number of days.

What the platform side of this looks like is a probe against the new credential before anything depends on it:

curl -sS -X POST https://api.vendor-a.example/screening/checks \
  -H "Authorization: Bearer ${API_KEY_NEXT}" \
  -H "X-Key-Generation: 2026-09-a" \
  -H "Idempotency-Key: 7f2b9c41-0a3d-4e55-9c2f-1b8d6e40aa12" \
  -d '{"applicant_id":"ap_4412","unit":"MAPLE-207","mode":"dry_run"}' \
  -D headers.txt -o body.json
Enter fullscreen mode Exit fullscreen mode

The idempotency header is doing real work during a rotation, not decoration. A retry that crosses the credential flip must not produce a second billable screening report, and an IETF HTTP API working group draft has been standardising the Idempotency-Key header for precisely this class of retry. Providers that support it generally echo enough metadata for you to tell a replayed response from a fresh one, and that distinction is the difference between a customer being charged once or twice.

Read the response headers, not the body, for attribution:

HTTP/1.1 200 OK
X-Served-By: vendor-a
X-Key-Generation: 2026-09-a
X-Billed-Units: 1
X-Request-Id: 01J9Q2K7YV3M8X
Enter fullscreen mode Exit fullscreen mode

Those four values are the ledger row. If a provider doesn't expose them, you are inferring attribution from your own routing intent rather than observing it, and intent is exactly the thing that diverges during a failover.

How should a fallback provider be tested when default routing hides which vendor answered?

Default routing is the right posture, and it is also an information problem: under normal conditions the second provider may serve almost nothing, so its health is unmeasured and its credential ages in the dark. The failure I would plan against is not the fallback returning errors. It is the fallback's own credential having expired months earlier, unnoticed, because nothing ever authenticated with it.

Test it on a schedule tied to the credential lifetime rather than a round calendar number. If your cryptoperiod is 90 days, a probe every 30 days gives you three observations per generation, which is enough to catch an expiry before it matters and cheap enough that nobody argues about it in the cost review.

A forced-route probe pins the alternate vendor explicitly and marks itself synthetic:

curl -sS -X POST https://gateway.internal.example/screening/checks \
  -H "Authorization: Bearer ${GATEWAY_TOKEN}" \
  -H "X-Route-Preference: vendor-b" \
  -H "X-Synthetic: true" \
  -d '{"applicant_id":"ap_seed_001","unit":"TEST-000","mode":"dry_run"}'
Enter fullscreen mode Exit fullscreen mode

That synthetic flag is not hygiene, it is accounting. Canary traffic is billable traffic, and a fallback probe that lands unmarked in the ledger gets rebilled to whichever tenant owns the seed record — a small number, wrong in a way that destroys trust in every other number on the same invoice.

Comparing status codes alone is not a test. Two providers returning 200 can disagree about the shape of the payload, the enum values in a screening decision, or the units on the bill — one charging per check and the other per applicant, which is a two-to-one difference on a multi-unit application. A useful probe asserts on the decoded response, records the billed units the provider reported, and stores both alongside the primary's answer for the same seed input.

The catch is that this only proves the path works at probe volume. A fallback that handles four requests a month tells you nothing about rate limits, concurrency ceilings or per-minute quotas at full production load, and I don't think there is an honest way to close that gap short of periodically shifting a real share of traffic — say five percent for an hour — and watching the error budget. Whether that is worth its own operational risk depends on how catastrophic a failover would be, and reasonable architects land on different sides of it.

What per-request attribution costs in cardinality and retention

This is where the instinct to label everything meets the bill. Put the tenant on your metrics and the arithmetic goes bad fast: 400 management companies × 3 vendors × 2 live credential generations × 6 endpoints × 5 status classes is 72,000 active series. At a 15-second scrape that is roughly 415 million samples per day, and Prometheus documents an average storage cost of about one to two bytes per sample once compressed — call it 700 MB per day, or something near 280 GB across a 13-month retention window.

Drop the tenant label and the same cut is 180 series. Under 2 MB a day.

The tenant dimension has to live somewhere, though, and the honest answer is that it belongs in a billing ledger rather than a time series. One row per request at roughly 180 bytes, 2.4 million requests a month, is about 430 MB a month — an order of magnitude cheaper than the labelled metric, queryable by exactly the dimension finance cares about, and retainable for the 25 months that a dispute window actually requires.

Signal Volume Sampling Retention Question it answers
Metrics: vendor × generation × status ~180 series none needed 13 months Is the fallback healthy right now?
Billing ledger: one row per request ~2.4M rows/month never sample 25 months Who pays for this request?
Traces 1% head-sampled aggressive 7–30 days Why was this call slow?
Raw access log all requests none 14 days What happened last Tuesday?

Sampling is the one place where the rules genuinely differ by signal. Traces at one percent are fine, because a trace answers a question about a class of requests. A sampled billing ledger is an estimate, and estimates lose disputes — if a management company can show you charged for 47 screenings and your evidence is a 1% sample extrapolated to 4,700, the conversation is over and you are issuing a credit. Keep the ledger complete, keep it narrow, and resist every request to add a column that would be more comfortable as a metric label. Cardinality arrives one well-intentioned label at a time.

Rolling it out without a maintenance window

The sequence that has the least drama in it: add the four attribution fields to the ledger schema first and dual-write them for a week while the routing and credentials stay exactly as they are. Nothing is verified until the ledger sums to the current invoice, per vendor, on data you collected before you changed anything. Then issue the second credential and let it overlap. Then move the default route. Revoke the old credential last, on its own day, when someone is awake.

Reconciliation is the acceptance test, and I'd set the tolerance tight: more than about one percent drift per vendor per month means a label is wrong, not that the invoice is.

Stick with a single pinned vendor when the constraints genuinely demand it. Data residency rules can force a pin outright, and committed-spend pricing tiers can make split traffic cost more than the redundancy is worth — in both cases the right move is to write the reduced redundancy down as an accepted risk with a named owner and a review date, rather than pretending the fallback exists. A warm second provider isn't a good fit either when the two vendors' outputs are not substitutable; a screening decision from a different data source may be a different product, whatever the status code says. Two providers you can't swap without a contract review are one provider with extra integration cost.

Sources

Top comments (0)