Running Claude Code for Free Through Multiple AI Providers with LiteLLM

Ever wish you could route Claude Code through multiple AI providers—NVIDIA NIM, OpenCode Zen, Agnes AI—without changing a single line of your workflow? That's exactly what the litellm-proxy project does. It sits between Claude Code and your AI backends, handling load balancing, automatic fallbacks, and parameter translation so everything just works.

What Problem Does This Solve?

Claude Code speaks Anthropic's API. But what if you want to route requests through NVIDIA's Nemotron models, OpenCode's free tier, or Agnes AI's flash models? Normally you'd need separate integrations for each provider's API quirks.

LiteLLM Proxy solves this by speaking Anthropic's API on one side and everyone else's API on the other. Your code stays the same; the proxy handles the translation.

Core Concepts Made Simple

The Proxy Pattern

Think of LiteLLM Proxy as a universal translator. You configure it once with your API keys and model mappings, then point Claude Code at http://localhost:4000 instead of Anthropic's API. The proxy figures out which backend to call, handles retries, and normalizes parameters so nothing breaks.

Model Aliasing

The proxy maps friendly names (like claude-opus-4-8) to actual backend models (like nvidia_nim/nvidia/nemotron-3-ultra-550b-a55b). You ask for claude-opus-4-8; the proxy routes it to Nemotron behind the scenes.

Load Balancing Strategies

The proxy can distribute requests across multiple API keys for the same provider. The litellm-proxy repo uses latency-based routing—it learns which API key responds fastest and sends more traffic there.

router_settings:
  routing_strategy: latency-based-routing

Automatic Fallbacks

When a model fails (rate limit, timeout, error), the proxy automatically retries on the next available model. The config defines a fallback chain:

router_settings:
  fallbacks:
    - model: claude-sonnet-5

If the primary model fails three times (default num_retries: 3), traffic shifts to claude-sonnet-5 automatically.

Parameter Normalization (drop_params: true)

Different providers support different parameters. Anthropic might accept thinking: true while OpenAI doesn't. With drop_params: true, the proxy silently drops unsupported parameters instead of crashing:

litellm_settings:
  drop_params: true
  use_chat_completions_url_for_anthropic_messages: true

The second setting ensures all providers receive requests via the OpenAI-compatible /v1/chat/completions endpoint, which is what Claude Code expects.

How It Works: A Request Flow

Here's what happens when you send a request through the proxy:

1. You call Claude Code → It sends a request to http://localhost:4000/v1/chat/completions with model: "claude-opus-4-8"
2. Proxy receives request → Looks up claude-opus-4-8 in its model list
3. Routing decision → Picks the fastest healthy NVIDIA API key (latency-based)
4. Parameter cleanup → Drops thinking parameter since Nemotron doesn't support it (drop_params: true)
5. Forward request → Calls NVIDIA NIM API with the cleaned payload
6. Handle response → Translates response back to Anthropic format
7. Return to Claude Code → Your code gets a response it understands

If step 5 fails, the proxy retries up to 3 times, then falls back to claude-sonnet-5 (which routes to OpenCode's free models).

Practical Example: The Config

The heart of the proxy is litellm/config.yaml. Here's what the repo's config does:

model_list:
  # Two NVIDIA API keys for the same model = load balancing
  - model_name: claude-opus-4-8
    litellm_params:
      model: nvidia_nim/nvidia/nemotron-3-ultra-550b-a55b
      api_key: os.environ/NVIDIA_API_KEY_1
      extra_body:
        chat_template_kwargs:
          thinking: false

  - model_name: claude-opus-4-8
    litellm_params:
      model: nvidia_nim/nvidia/nemotron-3-ultra-550b-a55b
      api_key: os.environ/NVIDIA_API_KEY_2
      extra_body:
        chat_template_kwargs:
          thinking: false

  # OpenCode free models as fallback
  - model_name: claude-sonnet-5
    litellm_params:
      model: openai/mimo-v2.5-free
      api_base: https://opencode.ai/zen/v1
      api_key: os.environ/OPENCODE_API_KEY

  - model_name: claude-sonnet-5
    litellm_params:
      model: openai/hy3-free
      api_base: https://opencode.ai/zen/v1
      api_key: os.environ/OPENCODE_API_KEY

  # Haiku via NVIDIA's GPT-OSS
  - model_name: claude-haiku-4-5-20251001
    litellm_params:
      model: nvidia_nim/openai/gpt-oss-120b
      api_key: os.environ/NVIDIA_API_KEY_1

  - model_name: claude-haiku-4-5-20251001
    litellm_params:
      model: nvidia_nim/openai/gpt-oss-120b
      api_key: os.environ/NVIDIA_API_KEY_2

  # Agnes 2.0 Flash
  - model_name: agnes-2.0-flash
    litellm_params:
      model: openai/agnes-2.0-flash
      api_base: https://apihub.agnes-ai.com/v1
      api_key: os.environ/AGNES_API_KEY

litellm_settings:
  drop_params: true
  use_chat_completions_url_for_anthropic_messages: true

router_settings:
  routing_strategy: latency-based-routing
  num_retries: 3
  fallbacks:
    - model: claude-sonnet-5

Notice how claude-opus-4-8 appears twice with different API keys. The proxy treats these as two deployments of the same model and load-balances between them.

Getting Started

Prerequisites

• Docker and Docker Compose installed
• API keys for the providers you want:
  • NVIDIA API keys (from build.nvidia.com) — Free tier: ~40 RPM on 100+ models including Nemotron 3 Ultra, Llama, Gemma via NVIDIA-hosted endpoints. Requires NVIDIA Developer Program membership (free). Self-hosted NIM removes rate limits.
  • OpenCode API key (from opencode.ai) — Free models on Zen: MiMo-V2.5, Nemotron 3 Ultra, DeepSeek V4 Flash, North Mini Code, Big Pickle (all $0/token). Free period is limited-time; rate limits not publicly documented per model. Pro tier $9.99/mo for unlimited.
  • Agnes AI API key (from agnes-ai.com) — Free tier: ~20 RPM on Agnes-2.0-Flash (multimodal text/image/video). No credit card required. Check current limits on their site as they may change.

1. Clone and Configure

git clone https://github.com/sydasif/litellm-proxy
cd litellm-proxy

cp .env.example .env
# Edit .env with your actual API keys:
# NVIDIA_API_KEY_1, NVIDIA_API_KEY_2, OPENCODE_API_KEY, AGNES_API_KEY

2. Start the Proxy

docker compose up -d

3. Verify It's Running

curl http://localhost:4000/v1/models

You should see a JSON list of available models including claude-opus-4-8, claude-sonnet-5, claude-haiku-4-5-20251001, and agnes-2.0-flash.

4. Configure Claude Code

Add these to your shell profile (~/.zshrc, ~/.bashrc, etc.):

export ANTHROPIC_AUTH_TOKEN=sk-12345  # Any value works; proxy doesn't validate
export ANTHROPIC_BASE_URL=http://localhost:4000

Restart your terminal, then run claude in your project folder. When prompted, select the model you want (e.g., claude-opus-4-8) and start prompting. All requests route through the proxy.

Test Directly with curl

curl -X POST http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-opus-4-8",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 100
  }'

Check the response headers for x-litellm-model-id to see which backend actually handled your request.

Using with Claude Code

Run claude in your project folder. When the model selector appears, pick the alias you configured (e.g., claude-opus-4-8). Your prompts now route through the proxy to whatever backend you've mapped.

The proxy exposes an Anthropic-compatible endpoint at http://localhost:4000/v1/chat/completions. Claude Code reads ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN, sends requests there, and the proxy translates them to the actual provider APIs.

Model selection in Claude Code:

• claude-opus-4-8 → NVIDIA Nemotron 3 Ultra (load-balanced across two API keys)
• claude-sonnet-5 → OpenCode Zen free models (MiMo-V2.5, HY3)
• claude-haiku-4-5-20251001 → NVIDIA GPT-OSS 120B
• agnes-2.0-flash → Agnes AI Flash model

Switch models in the CLI picker any time. The proxy handles the translation.

Day-to-Day Operations

Task	Command
Update API keys	Edit .env and docker compose restart
Change routing/providers	Edit litellm/config.yaml and docker compose down && docker compose up -d
Upgrade LiteLLM	docker compose pull && docker compose up -d
View logs	docker compose logs -f

Config changes require a full container restart because LiteLLM loads the YAML at startup.

Security Notes

• API keys live only in .env — The file is gitignored
• No secrets in config.yaml — All keys reference os.environ/
• Rotate keys regularly — Follow each provider's best practices
• Network exposure — The proxy binds to localhost:4000 by default. Don't expose it publicly without authentication.

Best Practices

1. Use multiple API keys per provider — The config shows two NVIDIA keys for claude-opus-4-8. This doubles your rate limits and provides redundancy.

2. Mix free and paid tiers — claude-sonnet-5 maps to OpenCode's free models. Great for fallback when paid quotas exhaust.

3. Monitor latency — The latency-based-routing strategy adapts automatically, but check logs occasionally to see which providers are winning.

4. Test fallbacks — Trigger a fallback with mock_testing_fallbacks=true in your request body to verify the chain works.

5. Keep .env out of git — The .gitignore already handles this, but double-check before pushing.

A Lab to Try

1. Start the proxy with only NVIDIA_API_KEY_1 set (leave _2 blank)
2. Send a burst of requests to claude-opus-4-8
3. Watch the logs—after rate limits hit, it should fail over to claude-sonnet-5
4. Add the second key and see load balancing resume

What's Next

• Add Redis for multi-instance proxy deployments (share rate-limit state)
• Configure enforce_model_rate_limits for hard quota enforcement
• Explore encrypted_content_affinity for OpenAI Responses API compatibility
• Set up team-scoped models with model_info.team_id for multi-user environments

---

Acknowledgment: This blog is inspired by publicly available materials, standards, and community research around LiteLLM, Anthropic's Claude Code, and multi-provider LLM gateway patterns.