DEV Community

Cover image for Replacing LiteLLM in Docker (500 MB RAM) with an 11 MB Go binary: fast LLM fallbacks and OS keyring
Ariel Alejandro
Ariel Alejandro

Posted on AI-assisted

Replacing LiteLLM in Docker (500 MB RAM) with an 11 MB Go binary: fast LLM fallbacks and OS keyring

#showdev #go #llm #opensource

When I started building against the OpenAI API, I iterated a lot. Run the test, change something, run it again. Then it clicked that every word coming back from the model was money.

So I went looking at the free tiers of other providers. I went deep enough that I built a page about them: trymodels.aralde.dev. They helped, but each one was limited in its own way, and keeping a stable service on top of them was hard.

This post is about what I built to solve this once and for all. The short version: you don't need a 500 MB Docker container running Python just to route local LLM requests. A single 11 MB Go binary can configure multi-provider fallbacks instantly, store your API keys securely in your OS keyring instead of scattered .env files, and dramatically simplify your developer life.

Two problems with existing tools

The first problem was cost, stability, and resource waste. Because these providers speak the OpenAI-compatible API, in theory you swap a base URL and keep going. In practice you want something smarter: one name for the model, several providers behind it, and when one hits a rate limit or drops, the same call seamlessly falls back to the next.

LiteLLM does this, but running it in Docker took about 500 MB of RAM on my machine just to route local HTTP traffic. It felt overkill to keep a full container and Python runtime alive just for proxying requests. I wanted an instant, lightweight tool that does fast multi-provider fallbacks without the bloat.

The second problem was keys. Every new project started the same way:

# project-a/.env
OPENAI_API_KEY=sk-...
OPENROUTER_API_KEY=sk-or-...
GROQ_API_KEY=gsk_...
Enter fullscreen mode Exit fullscreen mode

Then project-b/.env, with the same three lines. Then project-c. Each copy is one more file with a live key in it, sitting in one more folder.

I decided to solve both in one place. The result is OperatorLM, a local OpenAI-compatible proxy. It's one Go binary of about 11 MB. No Docker, no Python, no node_modules.

What it looks like

Demo of OperatorLM: one model name mapped to three accounts, a test request returning 200 with the circuit-breaker settings beside it, a config file that holds only a keyring reference, the ~11 MB binary, and local models in the admin UI

The client side is a base URL. Anything that speaks the OpenAI API can point at it:

curl http://127.0.0.1:11434/v1/chat/completions \
  -H "Authorization: Bearer anything" \
  -H "Content-Type: application/json" \
  -d '{"model": "fast-llama", "messages": [{"role": "user", "content": "Hi"}]}'
Enter fullscreen mode Exit fullscreen mode

The API key can be any non-empty string. OperatorLM injects the real one.

fast-llama is not a model. It's an alias. In the README example it points at Groq first (30 requests per minute), then OpenRouter as the paid fallback. The client asks for one name. The alias walks its targets in order until one answers.

Aliases tab in the admin UI: one alias with strategy "order" and five targets across OpenRouter, Google, NVIDIA and Groq, each with an order and an RPM limit

Each alias decides which provider takes the first request. OpenAI, OpenRouter, Gemini, whatever you put at the top of the list.

It listens on 127.0.0.1:11434, the same port Ollama uses. So it works as a drop-in for tools that already talk to that address: Cursor, Continue, Cline, Zed and similar.

What I learned about failover

Retrying the same target is not failover

If a provider says you're rate limited, retrying it 500 ms later mostly gets you the same answer. The quota didn't reset because you waited half a second.

So an alias has a strategy, order, that walks the target list until one succeeds. Retries still exist (2 by default, backoff starting at 500 ms with jitter, capped at 10 s).

Not every failure is the same failure

This is the part that surprised me. You have to tell apart network failures, streaming timeouts and quota limits (RPM). If you treat them the same, you block an alias that didn't need blocking.

The circuit breaker has three states and opens after 3 failures. The cooldown depends on what failed:

  • 429 (rate limited): 15 s
  • 5xx (the provider is broken): 60 s
  • network error: 90 s

It also honors Retry-After when the provider sends it.

Streams fail quietly

A stream that stops sending tokens doesn't return an error code. The connection is open and nothing arrives. So there are two separate clocks: a stream idle timeout (30 s) and a total request timeout (180 s).

Skip a target you already know is full

If a free account allows 30 requests per minute, sending the 31st is a wasted round trip. The RPM limiter keeps a sliding 60 s window per target. When a target is full, it gets skipped instead of making the request wait.

All of this is tunable live from the Reliability tab, without a restart.

Reliability tab: retry policy, circuit breaker thresholds and cooldowns per failure type, and timeouts

Keys belong in the OS keyring

The fix for the .env problem was boring, which is why I like it. Keys go into the keyring your OS already has: Windows Credential Manager, macOS Keychain, or Linux Secret Service.

The config file at ~/.operatorlm/config.toml only holds references, like operatorlm:openai_work. There's no secret in it. Your projects don't hold keys either. They hold a base URL.

A few more things in the same spirit. The audit log is JSONL at ~/.operatorlm/audit.log, with Authorization headers redacted. The proxy only listens on loopback, with host-header validation on the admin API to block DNS rebinding. And v0.2.1 has zero telemetry.

Architecture diagram: editors and SDKs send requests to OperatorLM on 127.0.0.1:11434, which runs alias resolution, retry with jitter, a circuit breaker, an RPM limiter, keyring key injection and an audit log, then forwards to OpenAI, OpenRouter, Groq, Gemini, Azure OpenAI and other providers

  1. Receive an OpenAI-compatible request.
  2. Resolve the model name, by prefix match or alias.
  3. Inject the key from the OS keyring.
  4. Try, retry or break, following the rules above.
  5. Audit the result to the JSONL log.

The admin UI is embedded in the binary with go:embed and lives at http://127.0.0.1:11434/admin/. The codebase is small enough to audit in an afternoon. That's the point.

The project kept growing after that. The same binary can now run local GGUF models through a built-in llama.cpp engine (exposed as local/<model>), plus Whisper for speech-to-text and Piper for text-to-speech, all offline. One experiment chains them: a local model answers, and Piper turns the answer into voice, so it behaves a bit like a realtime voice model.

Limitations

  • The binary is not signed. Windows SmartScreen shows "Windows protected your PC" (More info, Run anyway). macOS Gatekeeper blocks the first launch (right-click, Open).
  • It's built for local development on 127.0.0.1, for one user. No distributed production setup.
  • Port 11434 clashes with a running Ollama. Change one of them in ~/.operatorlm/config.toml.
  • On headless Linux, the keyring needs a running Secret Service daemon and a D-Bus session. Running it as a proper Linux service on a server is the next thing I plan to do.
  • Local models need llama-server and run as a second process. Speed depends entirely on your GPU and RAM.

Try it

It's Apache 2.0, for Windows x64, macOS (Apple silicon and Intel) and Linux x64. The code is at github.com/aralde/operatorlm and the site is operatorlm.aralde.dev. If a provider fails over badly for you (an error that looks like one thing and is another), an issue with the response attached helps the most.

Top comments (0)