Best Practices for Configuring Fallback Settings with LiteLLM for Multi-Provider LLM Usage

📝 Originally published (in Japanese) at forge.workstyle.tech.

Introduction

To increase the reliability of AI applications, it is crucial to adopt an architecture that does not depend on a single LLM provider. LiteLLM acts as an AI Gateway, allowing you to handle multiple providers—such as OpenAI, Anthropic, Azure, and Vertex AI—through a single unified interface. In this article, we will introduce best practices for multi-provider operations using LiteLLM's fallback functionality.

What is LiteLLM's Fallback Feature?

Basic Fallback Mechanism

LiteLLM's fallback feature is a mechanism that automatically routes requests to a different provider when a specific provider or model fails. The main features are as follows:

• Automatic Provider Failover: If a request fails after a specified number of attempts (num_retries), it automatically switches to a different model group.
• Context-Aware Fallback: Introduced in LiteLLM v1.44 and later, this feature automatically removes incompatible parameters (e.g., response_format) to prevent silent failures.
• Context Window Fallbacks: When input is too long, it automatically routes the request to an appropriate model (e.g., glm47-flash).

Configuration Example

Fallback settings are configured in the LiteLLM configuration file (config.yaml). Here is a basic example:

model_list:
  - model_name: gpt-4
    litellm_params:
      model: openai/gpt-4
      api_key: os.environ/OPENAI_API_KEY
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: openai/gpt-3.5-turbo
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude-3-opus
    litellm_params:
      model: anthropic/claude-3-opus-20240229
      api_key: os.environ/ANTHROPIC_API_KEY

litellm_settings:
  num_retries: 3
  fallbacks: [{"gpt-4": ["gpt-3.5-turbo", "claude-3-opus"]}]

In this configuration, if gpt-4 fails, it will fall back to gpt-3.5-turbo and then to claude-3-opus. There are three key points:

1. Fallbacks should be defined under litellm_settings (or router_settings) in the fallbacks section, not inside the litellm_params of each model.
2. The fallback destinations must be specified using the model_name (alias) already registered in the model_list.
3. The fallback models themselves must also be defined within the model_list. Note that num_retries is also placed under litellm_settings.

Practical Settings for Operational Optimization

Balancing Cost and Performance

When utilizing fallback features, it is important to consider the balance between cost and performance. Keep these points in mind:

• Cost Tracking: LiteLLM supports token-based cost calculation, allowing you to track costs in detail per provider and model.
• Load Balancing: By leveraging the Router function to distribute the load across multiple providers, you can prevent heavy concentration of traffic on a single provider.
• Prompt Caching: By caching frequently used prompts, you can reduce latency and optimize costs.

Metrics and Monitoring

To operate fallback features effectively, monitoring request metrics is essential. Key metrics to watch include:

• Latency: Monitor the response time of each provider.
• Error Rate: Track the error rate when a fallback is triggered.
• Fallback Rate: Monitor how often fallbacks occur to evaluate provider stability.

By reviewing these metrics, you can optimize your operations by refining fallback thresholds and routing rules.

Real-world Operational Scenarios

Scenario 1: Fallback during High Load

If a specific provider becomes slow due to high load, LiteLLM automatically routes the request to a different provider. This maintains the user experience while distributing the load across providers.

Scenario 2: Model Compatibility Issues

Some providers may not support certain parameters (e.g., response_format). LiteLLM's Context-Aware Fallback feature can automatically strip these parameters to ensure the request succeeds.

Scenario 3: Automatic Routing for Long Inputs

When inputs are long and exceed the context window, LiteLLM automatically routes the request to an appropriate model (e.g., glm47-flash). This maintains response quality even with long-form inputs.

Best Practices and Precautions

Best Practices

1. Define Clear Fallback Priorities: Clearly define the priority of the providers or models you intend to use and reflect this in your configuration file.
2. Leverage Metrics: Regularly check metrics like latency, error rates, and fallback rates to optimize your operations.
3. Utilize Caching: Use Prompt Caching to shorten response times for frequently used prompts.
4. Validate in a Test Environment: Before applying changes to production, verify the fallback functionality in a test environment to ensure configuration accuracy.

Precautions

1. Cost Estimation: When a request is routed to a different provider via fallback, costs may increase. Perform cost estimations in advance.
2. Maintaining Response Quality: Since fallbacks can potentially lead to a decrease in response quality, regular quality checks are necessary.
3. Security and Privacy: When using multiple providers, ensure you comply with all security and privacy policies.

Conclusion

By utilizing LiteLLM's fallback features, you can significantly improve the reliability and availability of your AI applications. Use the configuration methods and operational know-how introduced in this article to implement this in your own systems. By properly utilizing fallback functions, you can flexibly respond to provider failures or load fluctuations, achieving stable service delivery.