The LLM Waterfall Pattern: Never Let a Rate Limit Kill Your Workflow

The LLM Waterfall Pattern: Never Let a Rate Limit Kill Your

Workflow

## The Problem

You're in the zone. Your AI agent is cranking out code,
reviewing PRs, and generating documentation. Then it happens:

 Error: 429 Too Many Requests

Your workflow stops. You wait. You retry. You lose momentum.

Sound familiar?

## The Solution: LLM Waterfall

Instead of relying on a single API, build a cascading fallback

system:

 Primary API (OpenAI, Anthropic, Gemini)
     ↓ rate limited
 Secondary API (OpenRouter)
     ↓ rate limited
 Local Model (Ollama, LM Studio)
     ↓ still failing
 Queue and retry

## How It Works

The waterfall pattern catches 429 (Rate Limit) and 5xx (Server

Error) responses automatically. When the primary API fails, it

instantly cascades to the next option without losing context.

### Key Features

1. Transparent failover — Your agent doesn't see the failures
2. Context preservation — The exact same prompt and tools go to the fallback
3. Automatic recovery — When the primary comes back, it switches back
4. Zero configuration — Works out of the box

## Implementation

Here's a simplified version:

   type WaterfallProvider struct {
       providers []LLMProvider
   }

   func (w *WaterfallProvider) Generate(ctx context.Context, prompt 
 string) (string, error) {
       for _, provider := range w.providers {
           response, err := provider.Generate(ctx, prompt)
           if err == nil {
               return response, nil
           }

           // Check if it's a rate limit or server error            
           if !isRetryableError(err) {                              
               return "", err
           }

           // Log and try next provider
           log.Printf("Provider %s failed: %v, trying next...",     
 provider.Name(), err)
       }

       return "", fmt.Errorf("all providers failed")
   }

   func isRetryableError(err error) bool {
       // Check for 429, 5xx, timeout errors
       return strings.Contains(err.Error(), "429") ||
              strings.Contains(err.Error(), "500") ||
              strings.Contains(err.Error(), "502") ||
              strings.Contains(err.Error(), "503")                  
   }

Real-World Results

After implementing this pattern:

• Zero downtime from rate limits
• 30% faster development (no waiting for retries)
• Local models as backup = offline capability
• Cost savings — Route to cheaper models when possible

The Provider Chain

My current setup:

1. Primary: Claude API (best quality)
2. Secondary: OpenRouter (multiple models)
3. Tertiary: Local Ollama (free, offline)

When Claude hits rate limits, OpenRouter catches it. When
OpenRouter is slow, Ollama takes over. The agent never stops

working.

Key Insights

### 1. Not All Errors Are Equal

• 429 (Rate Limit): Retry with backoff, then cascade
• 5xx (Server Error): Cascade immediately
• 4xx (Client Error): Don't cascade, fix the request
• Timeout: Cascade to faster provider

### 2. Context Matters

When cascading, you need to preserve:

• The full prompt
• Tool configurations
• Memory context
• System instructions

### 3. Cost Optimization

Use the waterfall for cost savings:

• Route simple tasks to cheaper models
• Use local models for development
• Reserve premium APIs for production

Try It Yourself

If you want to see this in action, check out HyperNexus — it

implements the LLM Waterfall out of the box:

• GitHub: https://github.com/HyperNexusLLC/hypernexus
• Website: https://hypernexus.site

What's Your Setup?

How do you handle rate limits in your AI workflows? Do you use a

single API or have a fallback system?

Let me know in the comments — I'd love to hear what's working for

others.

──────────────────────────────────────────────────────────────────

If you found this useful, star the repo on GitHub:

https://github.com/HyperNexusLLC/hypernexus