Short answer: keep a second provider warm, but make the default route boring and measurable; test the fallback with synthetic media-tenant key operations on a schedule, and record every decision in an append-only audit log.
For a media platform, the dangerous moment is rarely a dramatic outage. It is a quiet concentration problem: every tenant's upload worker, captioning job, and partner integration depends on one API account. When that account is throttled, changes policy, or becomes unreachable in one region, the team discovers that its “fallback” has never issued or revoked a real scoped key.
I build RAG and agent features in Python, so I care about an eval harness before I trust a routing change. The same habit applies here. A fallback is a hypothesis until a test proves that it can issue, validate, and revoke a key without widening tenant access.
What should default routing and fallback tests prove?
Start with an explicit contract. For tenant studio-17, an issued key may call only the media operations that tenant bought, and its audit record must identify the tenant, provider, scope, actor, and correlation ID. The provider is an implementation detail; those invariants belong to your platform.
The test should exercise the whole lifecycle, not just a health endpoint:
- issue a short-lived key for a disposable tenant;
- call one allowed operation and one denied operation;
- revoke the key;
- confirm that a second call is denied after revocation;
- export the provider response metadata and your audit events.
Here is a deliberately small Python harness. Provider is an adapter around any API supplier, and the fake implementation makes the assertions executable in CI without contacting a live service.
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Protocol
from uuid import uuid4
class Provider(Protocol):
def issue_key(self, tenant_id: str, scopes: list[str], ttl_seconds: int) -> str: ...
def call(self, key: str, operation: str) -> int: ...
def revoke_key(self, key: str) -> None: ...
@dataclass
class AuditEvent:
action: str
tenant_id: str
provider: str
scope: tuple[str, ...]
correlation_id: str
at: str
def run_fallback_eval(provider: Provider, provider_name: str) -> list[AuditEvent]:
tenant_id = "studio-17"
scopes = ["media:read", "media:caption"]
correlation_id = str(uuid4())
events: list[AuditEvent] = []
key = provider.issue_key(tenant_id, scopes, ttl_seconds=300)
events.append(AuditEvent(
"issue", tenant_id, provider_name, tuple(scopes), correlation_id,
datetime.now(timezone.utc).isoformat(),
))
assert provider.call(key, "media:read") == 200
assert provider.call(key, "billing:write") in {401, 403}
provider.revoke_key(key)
events.append(AuditEvent(
"revoke", tenant_id, provider_name, tuple(scopes), correlation_id,
datetime.now(timezone.utc).isoformat(),
))
assert provider.call(key, "media:read") in {401, 403}
return events
My first version only called issue_key and treated a 2xx response as success. That passed every unit test and still left a hole: the revoke path was untested, and a provider could silently map media:caption to a broader permission. The corrected eval checks both positive and negative authorization. Three hundred seconds is a test TTL, not a production policy; choose a lifetime that matches your threat model.
How do you keep a second provider warm without splitting traffic?
Warm does not mean “send half of production traffic there.” It means the adapter is deployed, credentials are valid, quotas are known, and a synthetic tenant can complete the lifecycle above. Keep the provider in a standby state with a small, predictable test budget. Store no real customer media in that test.
The router should make one decision per request and persist it with the request ID. A simple policy is easier to audit than a clever weighted balancer:
- Route to the default provider while its dependency and policy checks are healthy.
- Trip to standby only for classified transport failures or an explicit provider-level rejection, never for a tenant authorization denial.
- Apply a short deadline, bounded retry count, and idempotency key to issue operations.
- Pin the chosen provider for the key's lifetime so a revoke follows the same control plane.
That last rule matters. If issuance goes to provider B after provider A times out, you need a durable record saying which provider owns the key. Blindly retrying an issuance can create two active credentials. For media workloads, duplicate captions are annoying; duplicate write access is an incident.
Use a circuit breaker with a half-open probe, but keep its state outside individual workers. Otherwise each autoscaled process makes a different routing decision. Emit counters for fallback_attempt, fallback_success, fallback_denied, and fallback_exhausted, plus latency histograms split by provider and operation. The dashboard should answer “which tenant, which scope, and which provider?” without joining opaque logs by hand.
The audit record is the product boundary
Provider dashboards are useful evidence, but they are not your tenant ledger. Write an event before returning a credential, then append the provider's request ID when it arrives. Protect the log from edits and restrict who can read secret-bearing fields. OWASP's Secrets Management Cheat Sheet recommends limiting secret exposure, rotating credentials, and auditing access; those controls still apply when a second supplier is involved.
Keep the key value out of ordinary logs. Hash a stable key identifier, retain scope and expiry, and link issue and revoke events with one correlation ID. Clock skew can make a revoke appear earlier than an issue, so store UTC timestamps and a monotonic sequence from your event store as well.
For incident review, I want a query that can reconstruct this sequence:
def decision_record(event: AuditEvent, outcome: str, request_id: str) -> dict:
return {
"request_id": request_id,
"correlation_id": event.correlation_id,
"tenant_id": event.tenant_id,
"provider": event.provider,
"action": event.action,
"scope": list(event.scope),
"outcome": outcome,
"recorded_at": event.at,
}
Do not put the raw credential in this structure. A redaction test should fail the build if fields such as key, token, or secret contain values that look like credentials.
Measuring the fallback before trusting it
Run the eval at least daily and before changing the default route. Measure completion rate, p95 issue and revoke latency, authorization-test accuracy, and the age of the last successful probe. Track spend as a guardrail, not as the success metric: a cheap provider that cannot preserve scope semantics is a security regression.
Inject controlled failures in a staging account: DNS failure, connection timeout, a rate-limit response, and a provider policy denial. Your expected result differs for each. A timeout may open the circuit; a 403 for an invalid scope should remain a 403 and must not trigger fallback. Record the decision and verify that retries do not create duplicate keys.
I’m not sure any single synthetic schedule catches regional policy changes, so your mileage may vary. Add a small canary from the regions where tenants actually run, and review the evidence rather than assuming green probes equal readiness.
The catch is operational cost and complexity. Two providers mean two credential rotation procedures, two retention policies, and two sets of contractual limits. This design is not suitable when your team cannot staff 24/7 incident ownership or when the second provider lacks the scope and revoke semantics your tenants require. Stick with one provider, with a documented recovery export and a tested manual rotation, when those prerequisites are missing. A warm standby is a liability if nobody can operate it.
The decision rule I use is narrow: keep the second provider warm when concentration risk could stop tenant access and both adapters pass the same authorization and audit eval. Otherwise, invest in portability at the boundary and revisit the standby after the missing control is measurable.
Top comments (0)