How I Cut My LLM API Costs by 80% With a Simple Relay Architecture

leechen — Fri, 29 May 2026 05:33:29 +0000

When you're building SaaS products that rely on LLM inference, the API bills can get painful fast. I was juggling multiple provider accounts, different API formats, and unpredictable pricing. So I built something simple: a unified relay layer.

The Problem

Direct API access to each LLM provider means:

Multiple accounts and billing cycles
Different API formats to integrate
No easy way to compare costs or switch models on the fly
Full retail pricing, every time

The Solution: A Relay Architecture

Instead of hitting each provider directly, I route all requests through a single relay endpoint. The relay handles:

Unified API format: One OpenAI-compatible endpoint for all models
Cost aggregation: Bulk access to DeepSeek, GPT-4o-mini, Gemini 2.5 Flash and more
Automatic fallback: If one provider is down, the relay routes to another

# Example: One endpoint, multiple models
import openai

client = openai.OpenAI(
    base_url="your-relay-url",
    api_key="your-key"
)

# DeepSeek
response = client.chat.completions.create(
    model="deepseek-chat",
    messages=[{"role": "user", "content": "Hello"}]
)

# GPT-4o-mini
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello"}]
)

# Gemini 2.5 Flash
response = client.chat.completions.create(
    model="gemini-2.5-flash",
    messages=[{"role": "user", "content": "Hello"}]
)

The Results

After running this setup for a few months:

Model	Direct Cost	Relay Cost	Savings
GPT-4o-mini	$0.15/M tokens	$0.10/M tokens	33%
DeepSeek Chat	$0.27/M tokens	$0.15/M tokens	44%
Gemini 2.5 Flash	$0.15/M tokens	$0.08/M tokens	47%

For a SaaS doing 50M tokens/day, that's thousands saved per month.

How It Works Under The Hood

The relay is built on top of One API with some custom routing logic. Key design decisions:

Token bucket rate limiting per user to prevent abuse
Model-specific pools to isolate capacity
Latency-based routing across multiple upstream providers
Usage tracking per API key with real-time dashboards

Getting Started

The relay is open for beta users. If you're a developer or SaaS founder looking to cut inference costs:

TG: @lee_heige
WhatsApp: +63 9603999881

I'll set you up with a test key so you can benchmark the speeds yourself.