Ashraf

Posted on Oct 19

If You Don't Run Your Own LLM Proxy, You're Flying Blind (Here's a 10-Minute Setup)

#llm #ai #opensource #devops

A post titled "If you don't run your own LLM proxy, how the f** do you even know what's going on?"* is sitting on the Hacker News front page right now with 160+ points and a 100-comment argument underneath it. The title is rude. The point is correct.

Here's the version with the profanity filed off.

The problem: your provider's dashboard is a toy

You're calling Claude, GPT, Gemini, or DeepSeek from your editor, your agents, a few scripts, and maybe production. Every one of those has its own key, its own client, its own failure mode.

And the "analytics" the labs give you? Usage graphs designed for a billing page, not for engineers. You can't answer basic questions:

Which key called which model, from which client, at what time, and did it succeed?
Why did my coding session suddenly get 10x more expensive? (Hello, cache miss.)
Which agent is burning tokens in a loop at 3am?
If I swap providers tomorrow, how many places do I have to edit?

The original author's line that stuck with me: when they see a cache miss mid-session, they can pinpoint why in 20 seconds. Without a proxy, you're guessing.

What an LLM proxy (gateway) actually gives you

Strip it down and it's a reverse proxy that speaks the OpenAI-compatible API in front of every provider. In return you get:

One endpoint you control. Multi-provider, multi-model, one base URL.
Virtual API keys. Your real provider key never touches an agent, a sandbox, or a teammate's laptop. Revoke a virtual key, nothing else breaks.
Real logs. Key, model, client, latency, status, tokens, cost. Time-series data you can query.
Fast provider pivots. Change one config line instead of re-wiring every tool for the third time.
Budgets, rate limits, fallbacks, guardrails. The boring stuff that saves you from a $4,000 surprise.

One HN commenter put the sandbox case well: you can't safely orchestrate agents in sandboxes without it, because the sandbox only ever sees a virtual key.

10-minute setup with LiteLLM

LiteLLM is the self-hosted open-source default: 100+ providers behind one OpenAI-format endpoint, with virtual keys, budgets, and logging. It isn't the only option (more on that below), but it's the fastest path to "I can see everything."

config.yaml:

model_list:
  - model_name: fast
    litellm_params:
      model: anthropic/claude-haiku-4-5-20251001
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: smart
    litellm_params:
      model: anthropic/claude-sonnet-5-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: cheap
    litellm_params:
      model: deepseek/deepseek-chat
      api_key: os.environ/DEEPSEEK_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL   # Postgres, for keys + spend logs

Run it:

docker run -d --name litellm -p 4000:4000 \
  -v $(pwd)/config.yaml:/app/config.yaml \
  --env-file .env \
  ghcr.io/berriai/litellm:main-stable \
  --config /app/config.yaml

Mint a virtual key with a hard budget, one per tool or agent:

curl -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"key_alias": "nightly-agent", "models": ["fast", "cheap"], "max_budget": 20}'

Point everything at it:

export OPENAI_BASE_URL=http://localhost:4000
export OPENAI_API_KEY=sk-your-virtual-key

That's it. Your agent now talks to fast, and you decide what fast means this week.

The honest counterarguments (HN had them)

The thread wasn't a love-in, and the pushback is worth knowing:

"Proxies are only useful if you pay API prices." Fair. If you're on a flat subscription, a proxy sees less than the local session logs do. One commenter suggests just collecting the JSONL files from ~/.claude/projects/ instead. Better organized, subagents included.
"A proxy is another point of failure." True. Put it behind health checks, keep a direct-to-provider fallback in your back pocket, and don't put it in the hot path of anything you can't afford to lose until you trust it.
"LiteLLM is a buggy, fast-moving mess." A real sentiment in the thread. Pin your image version, read the changelog, and don't expose the admin UI to the internet. Alternatives people named: Bifrost, OpenRouter (hosted, zero ops), Portkey (observability-first), and Tailscale's Aperture if you want it inside your tailnet.
"Just build your own harness." Some did. Valid if you enjoy it. Most of us have better things to ship.
"API pricing vs subscription." One commenter hit $200 of API spend in 2-3 days on usage that fit inside a $200/month plan. A proxy won't fix that math, but it will show you exactly where it goes.

Who should bother

Solo dev with one editor and one subscription: you can skip it. Honestly.
Anyone running agents, sandboxes, or CI that calls models: do it today. Virtual keys alone justify it.
Teams: you're already doing this badly with a shared key in a .env file. Do it properly.

Takeaway

The question isn't whether a gateway is cool. It's whether you can answer "what did my LLM calls do yesterday, and what did they cost?" in under a minute. If not, you don't have observability, you have vibes and an invoice.

Pick a proxy, self-host it, pin the version, mint virtual keys. The tool matters less than owning the chokepoint.

Source discussion: "If you don't run your own LLM proxy..." on Hacker News and the original post.

DEV Community