Slash LLM Costs: open source LLM API gateway for 14+ Providers

This article was originally published on BuildZn.

Everyone's chasing AI features, then they get hit with the bill. My FarahGPT users spiked, so did the OpenAI API costs. Tried scaling free tiers manually, that was a nightmare. Turns out an open source LLM API gateway is the only sane way to keep recurring AI costs from bleeding your project dry.

Why Your LLM Bill is Too High (and What an open source LLM API gateway Fixes)

Look, paying $2000 for OpenAI or Claude every month stings. Especially when there are dozens of decent, free LLMs out there. The problem? Managing them. Different APIs, different rate limits, different uptime. One goes down, your app breaks. That's why I started looking into LLM cost optimization beyond just picking a cheaper model.

We needed something that:

1. Unified APIs: Speak OpenAI, but route to anything.
2. Automated Fallback: If one free provider chokes, try another.
3. Rate Limiting: Don't hammer a free API to death and get blocked.
4. Cost Reduction: Obviously, slash that recurring AI spend.

FarahGPT, my AI gold trading system, saw its inference costs explode. I built it for a niche, not for thousands of daily active users chatting constantly. Migrating to an open source LLM API gateway wasn't just an option; it was mandatory to keep the lights on without raising subscription prices. This isn't just theory; we dropped our primary LLM API costs by about 75-80% for FarahGPT's core agent communication by moving off a single paid provider.

The Solution: A Unified LLM API Gateway to Rule Them All

After digging around, the free-llm-gateway project clicked. It's essentially a proxy that exposes an OpenAI-compatible API endpoint. You hit your gateway, and it intelligently routes your request to one of over 14 supported free or low-cost providers: HuggingFace, Perplexity, You.com, Poe, even OpenRouter (which aggregates its own free tiers).

Here's the thing — this isn't just about "free." It's about resilience. If Perplexity AI’s free tier is busy, it can try You.com. If that fails, maybe HuggingFace. This multiple LLM provider routing strategy is key to stability and cost savings. It turns what would be an integration headache into a single endpoint.

• OpenAI API Compatibility: Your existing code that talks to api.openai.com needs minimal changes. Just point it to your gateway.
• Automatic Fallback: Configure a priority list of providers. The gateway tries them in order.
• Built-in Rate Limiting: Protects upstream providers from being overwhelmed by your requests.
• Self-Hosted: You control it. Run it on a cheap VPS or even a Raspberry Pi if your traffic is low. This makes it a true self hosted LLM gateway.

Setting Up Your Free LLM Backend (Step-by-Step)

Getting this gateway up and running isn't rocket science, but there are a few gotchas. I'll walk you through setting it up with Docker. For a free LLM backend, Docker Compose is usually the quickest way.

First, you need a docker-compose.yml file. Create a directory, drop this in:

version: '3.8'
services:
  free-llm-gateway:
    image: ghcr.io/ramonvc/free-llm-gateway:latest
    container_name: free-llm-gateway
    restart: unless-stopped
    ports:
      - "8000:8000" # Expose the gateway on port 8000
    environment:
      # --- General Settings ---
      - API_PORT=8000
      - OPENAI_COMPATIBLE=true # Important for seamless integration
      - DEFAULT_MODEL=gpt-3.5-turbo # Or any model you prefer the gateway to map to

      # --- Provider Configuration (Pick what you need) ---
      # Poe.com - requires token (grab from browser cookies)
      - POE_TOKEN=your_poe_token_here
      - POE_ENABLED=true
      - POE_MODEL=ChatGPT # Example model mapping

      # HuggingFace Inference API - requires token
      - HF_TOKEN=hf_your_huggingface_token_here
      - HF_ENABLED=true
      - HF_MODEL=meta-llama/Llama-2-7b-chat-hf # Example model

      # Perplexity AI (free tier, limited)
      - PPLEX_ENABLED=true
      - PPLEX_API_KEY=your_perplexity_api_key # Get from Perplexity Labs
      - PPLEX_MODEL=llama-2-70b-chat # Example model

      # You.com - no token needed for free tier, but rate limited
      - YOU_ENABLED=true
      - YOU_MODEL=you_chat_model # Example model

      # OpenRouter (aggregates free tiers, sometimes requires token for higher limits)
      - OPENROUTER_ENABLED=true
      - OPENROUTER_API_KEY=your_openrouter_key # Optional, but recommended for stability
      - OPENROUTER_MODEL=mistralai/mistral-7b-instruct-v0.2 # Example model

      # --- Rate Limiting (Crucial for free providers) ---
      - RATE_LIMIT_ENABLED=true
      - RATE_LIMIT_PER_PROVIDER_MINUTE=60 # Max requests per minute per unique provider
      - RATE_LIMIT_TOTAL_MINUTE=100 # Overall total requests per minute to the gateway

      # --- Fallback Strategy ---
      # This is the order the gateway will try providers
      - FALLBACK_PROVIDERS=PPLEX,OPENROUTER,POE,YOU,HF

Setup Steps:

1. Get Your Tokens/Keys: For providers like Poe, HuggingFace, Perplexity, and OpenRouter, you'll need API keys or tokens. For Poe, this is usually grabbed from your browser's cookies after logging in. For others, register on their respective sites to get an API key.
2. Configure Environment Variables: Replace your_poe_token_here, hf_your_huggingface_token_here, etc., with your actual values. Enable (_ENABLED=true) only the providers you want to use.
3. Define FALLBACK_PROVIDERS: This is your lifeline. Arrange providers in your preferred order. The gateway tries them one by one until a successful response or all fail. This is critical for uptime.
4. Set Rate Limits: RATE_LIMIT_PER_PROVIDER_MINUTE and RATE_LIMIT_TOTAL_MINUTE are non-negotiable for AI API rate limiting. Free tiers will block you if you don't respect their unspoken limits. I usually start conservative and increase if I see 200s.

5. Deploy:
   

   docker-compose up -d
   

   Your gateway should now be running on http://localhost:8000.

Integrating with Flutter and Node.js

Once your gateway is humming, your Flutter app can talk to it like it's OpenAI. If you're using a backend, like Node.js for security or additional logic (which you should for production), you'd route requests through that.

Flutter (via a Node.js Backend Proxy):
Your Flutter app should not directly hit the gateway from the client side. That exposes your backend gateway URL and potentially exhausts rate limits too quickly from distinct client IPs. Instead, your Flutter app talks to your Node.js backend, which then talks to the free-llm-gateway.

Here's a simplified Flutter example using http (assuming you have a backend proxy):

import 'dart:convert';
import 'package:http/http.dart' as http;

Future<String> getLLMResponse(String prompt) async {
  final url = Uri.parse('https://your-backend.com/api/chat'); // Your Node.js proxy endpoint
  final headers = {'Content-Type': 'application/json'};
  final body = jsonEncode({
    'messages': [
      {'role': 'system', 'content': 'You are a helpful assistant.'},
      {'role': 'user', 'content': prompt},
    ],
    'model': 'gpt-3.5-turbo', // The model name your gateway maps to
    'stream': false, // For simple non-streaming responses
  });

  try {
    final response = await http.post(url, headers: headers, body: body);

    if (response.statusCode == 200) {
      final data = jsonDecode(response.body);
      return data['choices'][0]['message']['content'];
    } else {
      print('Failed to get LLM response: ${response.statusCode}, ${response.body}');
      throw Exception('LLM API call failed');
    }
  } catch (e) {
    print('Error making LLM request: $e');
    throw Exception('Network or API error');
  }
}

// How you'd call it in your Flutter app:
// String response = await getLLMResponse("What's the capital of France?");
// print(response);

Node.js Backend Proxy (Express Example):
This is where your free-llm-gateway URL lives.

const express = require('express');
const axios = require('axios');
const app = express();
app.use(express.json());

const LLM_GATEWAY_URL = process.env.LLM_GATEWAY_URL || 'http://localhost:8000'; // Point to your gateway

app.post('/api/chat', async (req, res) => {
  try {
    const { messages, model, stream } = req.body;

    // Forward the request to your free-llm-gateway
    const gatewayResponse = await axios.post(
      `${LLM_GATEWAY_URL}/v1/chat/completions`,
      {
        messages,
        model: model || 'gpt-3.5-turbo', // Ensure this maps to a gateway-configured model
        stream: stream || false,
      },
      {
        headers: {
          'Content-Type': 'application/json',
          // No API key needed here as the gateway handles provider-specific keys
        },
        responseType: stream ? 'stream' : 'json',
      }
    );

    if (stream) {
      res.setHeader('Content-Type', 'text/event-stream');
      res.setHeader('Cache-Control', 'no-cache');
      res.setHeader('Connection', 'keep-alive');
      gatewayResponse.data.pipe(res); // Stream directly to the client
    } else {
      res.json(gatewayResponse.data);
    }
  } catch (error) {
    console.error('Error proxying LLM request:', error.response?.data || error.message);
    res.status(error.response?.status || 500).json({
      error: error.response?.data?.error || 'Failed to get response from LLM gateway',
    });
  }
});

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
  console.log(`Node.js proxy listening on port ${PORT}`);
});

This Node.js setup ensures that your Flutter app doesn't need to know anything about the underlying providers or their keys. It just calls your /api/chat endpoint, and your backend handles the rest, talking to your open source LLM API gateway.

What I Got Wrong First

Honestly, thinking any "free" LLM provider offers production-grade stability without significant fallback planning is naive. You'll hit 429 Too Many Requests more often than you think, especially with services like Poe or You.com after a few thousand requests. We saw a consistent 429 with free-llm-gateway routing to Poe on versions up to v0.2.1 when hitting more than 10 requests per minute from a single IP. It's not the gateway's fault; it's the upstream provider's free tier policy.

My initial mistake was assuming RATE_LIMIT_ENABLED=true alone would magically handle all upstream provider limits. Turns out, you need to be realistic about free tiers. They exist to lure you in, not to power your next unicorn. The gateway helps, but it can't invent capacity.

The Fix:

1. Aggressive FALLBACK_PROVIDERS list: Don't just list one or two. List all the free providers you've configured. The more options the gateway has, the higher your success rate.
2. Lower RATE_LIMIT_PER_PROVIDER_MINUTE: I started with 100, assuming 60 RPM was fine for most. For truly free tiers, sometimes you need to drop it to 10-20 to avoid blocks. Experiment.
3. Consider a "Semi-Free" Fallback: For critical paths, I added OpenRouter with a small credit balance. It aggregates its own free tiers (like Mistral-7B) but also offers cheap paid access to others. If all free options fail, OpenRouter's paid tier is still orders of magnitude cheaper than direct OpenAI access for non-GPT-4 models. This is a crucial LLM cost optimization strategy. It balances true free with low-cost reliability.

Optimization & Gotchas

• Provider Model Mapping: The gateway tries to map generic models (gpt-3.5-turbo) to specific provider models. Sometimes you need to be explicit. If you want Llama-2-70B from Perplexity, pass model: "llama-2-70b-chat" directly in your request. The gateway will try to route it to the PPLEX provider.
• Persistent Configuration: If you're running this on a server, use a .env file for your Docker Compose setup to manage your API keys, instead of hardcoding them.
• Monitoring: Keep an eye on your gateway's logs. If you're seeing a lot of 429 or 500 errors, it's a sign your rate limits are too high, or a specific free provider is having issues. This visibility is why a self hosted LLM gateway is so powerful.
• Streaming: The free-llm-gateway supports streaming responses. Make sure your Node.js proxy also pipes the stream correctly to your Flutter client for a better user experience. Check the axios configuration in the Node.js example above.

FAQs

How much can an LLM gateway actually save?

Significant amounts. For FarahGPT, we're talking about an 80% reduction in direct LLM API costs for the bulk of our inference. This comes from shifting requests from expensive paid models to free or low-cost alternatives, managed by the gateway's fallback and routing.

Is free-llm-gateway truly production-ready?

It's a solid foundation. For low-to-medium traffic apps like FarahGPT, yes, it’s stable enough. For high-volume, mission-critical systems, you need to augment it with robust monitoring, dedicated infrastructure, and possibly a low-cost paid provider as a final fallback, as discussed earlier. It handles multiple LLM provider routing well, which is half the battle.

How do I add new LLM providers to the gateway?

You generally can't just "add" a new provider yourself without modifying the free-llm-gateway source code. The project needs to be updated by its maintainers to integrate new provider APIs. Keep an eye on their GitHub for updates and new integrations.

Stop burning cash on LLM APIs when free alternatives exist. Setting up an open source LLM API gateway like free-llm-gateway isn't just about saving money; it's about building resilient AI infrastructure. You gain control, reduce vendor lock-in, and ensure your app keeps working even when a single provider chokes. It’s the smart play for any dev shipping AI features.