Testing Circuit Breaker Patterns with Browser-Based Health Checks

When building resilient API gateways, circuit breakers are essential—but verifying they actually work under real-world conditions is harder than it looks. Traditional synthetic monitoring misses edge cases like partial DOM rendering failures, authentication redirect loops, and client-side error boundaries that mask upstream failures.

I recently integrated browser-use into our failover testing pipeline to validate circuit breaker behavior end-to-end, including browser-rendered status pages and client-side retry logic.

The Problem: False Positives in API Health Checks

Our multi-region API gateway sits behind Cloudflare with automatic failover. Standard HTTP health checks (curl -f https://api.example.com/health) reported 200 OK, but users saw blank screens because:

1. The health endpoint returned cached responses during database connection pool exhaustion
2. JavaScript bundle errors broke the error page itself
3. Authentication middleware redirected /health to login when Redis sessions expired

We needed browser-context validation that exercises the full request path—TLS handshake, CDN routing, server-side rendering, and client hydration.

Solution: Automated Browser Health Probes

browser-use provides a clean Python API for browser automation with built-in stealth and cloud execution. Here's our circuit breaker validation harness:

from browser_use import BrowserSession
import asyncio
import time

class CircuitBreakerProbe:
    def __init__(self, primary_url, fallback_url, failure_threshold=3):
        self.primary = primary_url
        self.fallback = fallback_url
        self.failures = 0
        self.threshold = failure_threshold
        self.state = "CLOSED"  # CLOSED -> OPEN -> HALF_OPEN

    async def health_check(self, url, session):
        """Returns (success: bool, latency_ms: float, screenshot_path: str)"""
        start = time.time()
        try:
            page = await session.new_page()
            response = await page.goto(url, timeout=5000)

            # Validate both HTTP status AND rendered content
            if response.status != 200:
                return False, (time.time() - start) * 1000, None

            # Check for error indicators in DOM
            error_selectors = [
                "[data-error='true']",
                ".error-boundary",
                "#maintenance-mode"
            ]
            for selector in error_selectors:
                if await page.query_selector(selector):
                    screenshot = await page.screenshot()
                    return False, (time.time() - start) * 1000, screenshot

            # Verify critical interactive element loaded
            api_status = await page.query_selector("[data-api-status='healthy']")
            latency = (time.time() - start) * 1000
            return api_status is not None, latency, None

        except Exception as e:
            print(f"Health check failed: {e}")
            return False, (time.time() - start) * 1000, None
        finally:
            await page.close()

    async def run_check(self):
        async with BrowserSession() as session:
            # Try primary endpoint
            success, latency, screenshot = await self.health_check(self.primary, session)

            if not success:
                self.failures += 1
                print(f"Primary failure {self.failures}/{self.threshold} (latency: {latency:.0f}ms)")

                if self.failures >= self.threshold:
                    self.state = "OPEN"
                    print(f"Circuit breaker OPEN - testing fallback")

                    # Validate fallback immediately
                    fb_success, fb_latency, _ = await self.health_check(self.fallback, session)
                    if fb_success:
                        print(f"Fallback healthy ({fb_latency:.0f}ms) - traffic should route there")
                        return {"status": "FAILOVER_OK", "fallback_latency": fb_latency}
                    else:
                        return {"status": "TOTAL_FAILURE", "alert": "ALL_REGIONS_DOWN"}
            else:
                # Reset on success
                if self.failures > 0:
                    print(f"Primary recovered (latency: {latency:.0f}ms)")
                self.failures = max(0, self.failures - 1)
                self.state = "CLOSED"
                return {"status": "HEALTHY", "latency": latency}

# Usage in monitoring loop
async def main():
    probe = CircuitBreakerProbe(
        primary_url="https://api-us-east.example.com/dashboard",
        fallback_url="https://api-eu-west.example.com/dashboard",
        failure_threshold=3
    )

    while True:
        result = await probe.run_check()
        # Push to Prometheus/Datadog
        await asyncio.sleep(30)

if __name__ == "__main__":
    asyncio.run(main())

Key Implementation Details

Distributed Rate Limiting Integration

Our API gateway uses Redis-backed sliding window rate limits. To prevent the health check itself from triggering rate limits during incident recovery:

# Redis Lua script for atomic rate limit check + increment
RATE_LIMIT_SCRIPT = """
local key = KEYS[1]
local limit = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local now = tonumber(ARGV[3])

redis.call('ZREMRANGEBYSCORE', key, 0, now - window)
local current = redis.call('ZCARD', key)

if current < limit then
    redis.call('ZADD', key, now, now)
    redis.call('EXPIRE', key, window)
    return 1
else
    return 0
end
"""

# Exempt health check user-agent from rate limits
async def setup_browser_session():
    session = BrowserSession(
        headless=True,
        user_agent="HealthProbe/1.0 (exempt-from-rate-limit)"
    )
    return session

Exponential Backoff for Retry Storm Prevention

When the circuit opens, naive retry logic creates thundering herd problems. Add jittered backoff:

import random

def backoff_delay(attempt, base=2, max_delay=60):
    """Exponential backoff with full jitter"""
    delay = min(base ** attempt, max_delay)
    return delay * random.uniform(0.5, 1.0)

# In the monitoring loop:
for attempt in range(5):
    result = await probe.run_check()
    if result["status"] == "HEALTHY":
        break
    await asyncio.sleep(backoff_delay(attempt))

Production Results

After deploying this to our staging cluster:

• Caught 4 false-positive scenarios standard HTTP checks missed (auth redirect loops, stale CDN cache serving 200 with error DOM)
• Reduced MTTR by 40%: Automated failover validation eliminated manual browser testing during incidents
• Zero false circuit trips: Jittered backoff prevented cascading failures during transient network blips

Integration with Existing Monitoring

We push circuit breaker state changes to our observability stack:

from prometheus_client import Gauge, Counter

circuit_state = Gauge('circuit_breaker_state', 'Circuit state', ['region'])
failover_total = Counter('api_failover_total', 'Failover events')

def update_metrics(region, state):
    state_map = {"CLOSED": 0, "HALF_OPEN": 1, "OPEN": 2}
    circuit_state.labels(region=region).set(state_map[state])
    if state == "OPEN":
        failover_total.inc()

Lessons Learned

1. Browser context matters: 30% of our "healthy" API responses had broken frontends
2. Rate limit your health checks: Probes can DDoS your failover region during recovery
3. Screenshot evidence is gold: Captured DOM states make postmortems 10x faster
4. Test the fallback path cold: Our EU region's cold database pool caused 8-second first-request latency

browser-use turned circuit breaker testing from a manual QA checklist into a continuous validation loop. The Python API is clean, the stealth browser avoids bot detection on our CDN, and the cloud execution option removes local Chromium dependency hell.

For teams running multi-region APIs with automatic failover, I'd recommend adding browser-based health validation to your circuit breaker test suite—it catches an entire class of failures that HTTP-only checks miss.