The operating problem
Third-party failure is a design input, not an incident. Your payment agent just timed out against a wallet API. The next 500 ms determine whether the system degrades or cascades.
Most teams discover the answer in production. A wallet service returns 503s, a payment provider stalls on TLS handshake, an identity API accepts connections but never sends a response. The agent retries with no cap. Other agents in the workflow retry too. Within minutes, a recovering provider gets hit by hundreds of queued retries, connection pools exhaust, and a customer stares at a spinner.
The outage isn't the problem. The problem is that most agent architectures treat it as an exception to catch, not a state to design for. Agents that survive provider outages share one trait: they assume external services will fail, degrade, and recover on their own schedule, and they encode that assumption in timeouts, state, and fallback paths.
The architecture that holds up
Start with a dependency map, not a circuit breaker. For each agent, list every external call and classify it as critical, degradable, or optional. Critical dependencies block the agent's core function; degradable dependencies reduce quality or throughput but allow progress; optional dependencies can be skipped. A payment agent calling a wallet API has a critical dependency. A support agent calling identity verification has a degradable dependency if it can fall back to cached KYC attributes with a TTL and an audit flag. A procurement agent calling supplier inventory has a degradable dependency if it can checkpoint the workflow and resume later.
Agent Dependency Map: Critical vs Degradable Services
See how a payment agent classifies external APIs by criticality and what happens when each fails.
Then implement per-dependency circuit breakers. Each breaker needs three parameters: a timeout, a retry budget, and a half-open probe interval. The timeout bounds thread and connection pool occupancy; set it to the provider's p99 latency plus headroom, not a guess. The retry budget caps total attempts before the breaker opens; use exponential backoff with jitter to avoid synchronized retry storms. The half-open probe interval controls how quickly you test recovery; too short and you re-overload a recovering provider, too long and you extend degraded mode.
payment_wallet_api:
timeout_ms: 3000
retry_budget: 3
half_open_probe_interval_s: 30
fallback: queue_transaction
Circuit Breaker State Machine with Fallback
Trace how a circuit breaker moves between closed, open, and half-open states and when fallback logic engages.
Circuit breakers alone aren't enough. You need local fallback policies that preserve user intent without violating security or compliance. A payment agent that can't reach a wallet service should offer an alternative payment method or queue the transaction with an idempotency key, not silently retry. A support agent with a down identity API should continue with cached KYC attributes only if the cache is fresh, the risk score is below threshold, and the interaction is flagged for re-verification. A procurement agent facing a 503 from supplier inventory should checkpoint the workflow and resume from the last consistent state, using a saga or outbox pattern to avoid partial commits.
State persistence is the difference between retry and duplicate. Persist agent state and idempotency keys so retries after provider recovery never duplicate payments or orders. A payment that partially wrote before timing out must not be re-submitted as a new charge; use the provider's idempotency key if available, like Stripe's Idempotency-Key header, or a local Postgres dedupe table keyed by (agent_id, intent_id, provider_tx_id). An order that reached the supplier but lost the response must not be placed twice; store the provider's idempotency key in the same transaction as the order state.
Agent Degraded Mode Transitions
See how an agent moves between normal, degraded, queued, and recovered states during a provider outage.
Finally, expose degraded-mode UX. Tell users what is unavailable, what is queued, and when retry will occur. A customer who knows their payment is queued for retry in 30 seconds is less likely to double-submit or abandon the flow.
Where teams usually fail
Teams fail in the same five places.
Unbounded retries. Multiple agents retry against a recovering provider with no coordination, no backoff, and no cap. The provider comes back online and immediately gets hammered by hundreds of queued retries. It goes down again. The fix is a per-agent retry budget plus a shared circuit breaker state, not a global retry queue.
Missing or single timeouts. An agent calls a provider that is slow but not down. No timeout is set, or only a read timeout is set while the connect timeout is infinite. Threads block, connection pools exhaust, and the agent runtime stalls. Set separate connect and read timeouts; a connect timeout of 500 ms and a read timeout of 3 s is a reasonable starting point for HTTP APIs, whether you're calling Plaid or a wallet service.
Non-idempotent retries. An agent submits a payment, the provider processes it, but the response never arrives. The agent retries. The customer pays twice. This is the most expensive failure mode and the most common. Use provider idempotency keys when available; otherwise maintain a local dedupe table with a TTL longer than the provider's maximum retry window.
Silent fallback to a less secure path. An identity verification API is down, so the agent falls back to a weaker check without logging the change or getting approval. Compliance finds out months later during an audit. Every fallback path must be pre-approved, logged with the reason and the degraded attributes, and surfaced in the agent's audit trail.
Observability that only tracks agent uptime. The agent is up, so everything looks fine. But provider latency is climbing, error rates are rising, and the agent is slow. Teams misdiagnose provider degradation as agent slowness and waste days debugging the wrong layer. Instrument per-provider latency percentiles, error rates by status code, and circuit breaker state transitions per agent. Use OpenTelemetry metrics or Datadog custom metrics.
We cover the governance angle of these failures in our piece on Red Cards in Agentic AI: How to Handle Agent Misbehavior and Policy Violations. The resilience patterns are complementary.
How to measure progress
Agent uptime tells you almost nothing about resilience. What you need is per-agent, per-provider instrumentation.
Track provider health, latency, and error rates per agent, not just service-level uptime. A payment agent should report wallet API latency percentiles (p50, p95, p99), error rates by status code, and circuit breaker state transitions. A support agent should report identity API availability, fallback activation counts, and the age of cached KYC attributes. A procurement agent should report checkpoint frequency, resume success rate, and the number of in-flight sagas.
Three signals matter most:
- Mean time to degrade (MTTD): how quickly an agent detects provider failure and switches to fallback mode. Target under 2 seconds for synchronous calls; this is driven by timeout settings and breaker thresholds.
- Fallback success rate: what percentage of degraded interactions complete without data loss or duplication. Target above 99.9% for payment agents; lower for read-only fallbacks.
- Recovery time: how long it takes the agent to return to normal operation after the provider recovers. Target under 60 seconds after the half-open probe succeeds.
Run chaos experiments that simulate third-party outages, including partial responses and slow timeouts. Don't just kill the provider. Return 503s for 10% of requests, delay 20% of responses by 5 seconds, and drop 1% of responses after the provider has processed the request. These three failure modes expose different bugs: retry storms, timeout misconfiguration, and non-idempotent retries.
Top comments (0)