Building Resilient LLM Data Pipelines: Rate Limiting and Circuit Breaking with Crawl4AI

The Problem: When Your LLM Pipeline Becomes a DDoS Attack

Last quarter, our content ingestion pipeline for LLM fine-tuning crashed three times in production. The root cause? Unbounded retry loops in our web scraping layer created accidental DDoS attacks against upstream targets, triggering IP bans that cascaded into a 6-hour data blackout.

We were using Crawl4AI to convert documentation sites into LLM-ready markdown for our RAG system. The library itself is solid—clean markdown extraction, JavaScript rendering, and structured output. But we had wrapped it in naive retry logic that amplified transient network blips into infrastructure fires.

War Story: The Thundering Herd

The incident timeline:

1. T+0m: Upstream CDN hiccup caused 3% request failure rate
2. T+2m: Our scraper retried failed requests immediately, no backoff
3. T+5m: Retry storm overwhelmed both the target and our Redis rate limiter
4. T+12m: Target firewall banned our entire IP block
5. T+6h: Manual intervention to rotate IPs and rebuild scrape queue

The fix required three layers of defense: exponential backoff with jitter, distributed rate limiting with atomic Lua scripts, and a circuit breaker to fail fast when targets are genuinely down.

Implementation: Resilient Crawl4AI Client Wrapper

Here's the production-grade wrapper we built. Copy-paste ready for TypeScript projects:

import axios, { AxiosError } from 'axios';

interface RetryConfig {
  maxRetries: number;
  baseDelayMs: number;
  maxDelayMs: number;
}

class ResilientCrawler {
  private failureCount = 0;
  private lastFailureTime = 0;
  private readonly circuitThreshold = 5;
  private readonly circuitResetMs = 60000;

  constructor(
    private crawl4aiEndpoint: string,
    private retryConfig: RetryConfig = {
      maxRetries: 3,
      baseDelayMs: 1000,
      maxDelayMs: 32000
    }
  ) {}

  private calculateBackoff(attempt: number): number {
    const exponential = Math.min(
      this.retryConfig.baseDelayMs * Math.pow(2, attempt),
      this.retryConfig.maxDelayMs
    );
    const jitter = exponential * 0.1 * Math.random();
    return exponential + jitter;
  }

  private isCircuitOpen(): boolean {
    if (this.failureCount < this.circuitThreshold) return false;

    const elapsed = Date.now() - this.lastFailureTime;
    if (elapsed > this.circuitResetMs) {
      this.failureCount = 0;
      return false;
    }
    return true;
  }

  async scrape(url: string): Promise<string> {
    if (this.isCircuitOpen()) {
      throw new Error(`Circuit breaker OPEN: ${this.failureCount} consecutive failures`);
    }

    for (let attempt = 0; attempt <= this.retryConfig.maxRetries; attempt++) {
      try {
        const response = await axios.post(
          `${this.crawl4aiEndpoint}/crawl`,
          { url, markdown: true },
          { timeout: 30000 }
        );

        this.failureCount = 0;
        return response.data.markdown;

      } catch (error) {
        const isRetryable = this.isRetryableError(error as AxiosError);

        if (!isRetryable || attempt === this.retryConfig.maxRetries) {
          this.failureCount++;
          this.lastFailureTime = Date.now();
          throw error;
        }

        const backoffMs = this.calculateBackoff(attempt);
        await new Promise(resolve => setTimeout(resolve, backoffMs));
      }
    }

    throw new Error('Retry exhausted');
  }

  private isRetryableError(error: AxiosError): boolean {
    if (!error.response) return true;
    const status = error.response.status;
    return status === 429 || status >= 500;
  }
}

Layer 2: Distributed Rate Limiter with Redis

For multi-worker deployments, client-side backoff isn't enough. You need atomic rate limiting at the infrastructure layer:

-- rate_limit.lua (Redis atomic sliding window)
local key = KEYS[1]
local limit = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local now = tonumber(ARGV[3])

redis.call('ZREMRANGEBYSCORE', key, 0, now - window)
local current = redis.call('ZCARD', key)

if current < limit then
  redis.call('ZADD', key, now, now)
  redis.call('EXPIRE', key, window)
  return 1
else
  return 0
end

# Python middleware using the Lua script
import redis
import time

class RedisRateLimiter:
    def __init__(self, redis_client: redis.Redis, script_path: str):
        with open(script_path) as f:
            self.script = redis_client.register_script(f.read())

    def allow(self, key: str, limit: int, window_sec: int) -> bool:
        now_ms = int(time.time() * 1000)
        return bool(self.script(
            keys=[f"ratelimit:{key}"],
            args=[limit, window_sec * 1000, now_ms]
        ))

Lessons from Production

1. Jitter is non-negotiable: Without randomization, synchronized retries create secondary thundering herds
2. Circuit breakers need memory: Track failure rate across requests, not just individual call stacks
3. Atomic rate limits: Redis INCR is not atomic for sliding windows—use Lua scripts or sorted sets
4. Fail fast on 4xx: Only retry transient errors (5xx, timeouts, network), never client errors

Integration with Crawl4AI Cloud

If you're using Crawl4AI Cloud instead of self-hosted, wrap the API client with the same resilience patterns. The cloud endpoint handles JavaScript rendering and gives you one-key access, but upstream rate limits and transient failures still require client-side defense.

The production deployment serves 2M pages/day for our LLM training pipeline with zero unplanned downtime since implementing these patterns.

References

• Crawl4AI: github.com/unclecode/crawl4ai
• Redis rate limiting: redis.io/commands/incr
• Circuit breaker pattern: Martin Fowler's original post

No affiliates. No sponsors. Just infrastructure that doesn't catch fire at 3am.