DEV Community: Ariel Alejandro

Replacing LiteLLM in Docker (500 MB RAM) with an 11 MB Go binary: fast LLM fallbacks and OS keyring

Ariel Alejandro — Mon, 02 Nov 2026 17:41:15 +0000

When I started building against the OpenAI API, I iterated a lot. Run the test, change something, run it again. Then it clicked that every word coming back from the model was money.

So I went looking at the free tiers of other providers. I went deep enough that I built a page about them: trymodels.aralde.dev. They helped, but each one was limited in its own way, and keeping a stable service on top of them was hard.

This post is about what I built to solve this once and for all. The short version: you don't need a 500 MB Docker container running Python just to route local LLM requests. A single 11 MB Go binary can configure multi-provider fallbacks instantly, store your API keys securely in your OS keyring instead of scattered .env files, and dramatically simplify your developer life.

Two problems with existing tools

The first problem was cost, stability, and resource waste. Because these providers speak the OpenAI-compatible API, in theory you swap a base URL and keep going. In practice you want something smarter: one name for the model, several providers behind it, and when one hits a rate limit or drops, the same call seamlessly falls back to the next.

LiteLLM does this, but running it in Docker took about 500 MB of RAM on my machine just to route local HTTP traffic. It felt overkill to keep a full container and Python runtime alive just for proxying requests. I wanted an instant, lightweight tool that does fast multi-provider fallbacks without the bloat.

The second problem was keys. Every new project started the same way:

# project-a/.env
OPENAI_API_KEY=sk-...
OPENROUTER_API_KEY=sk-or-...
GROQ_API_KEY=gsk_...

Then project-b/.env, with the same three lines. Then project-c. Each copy is one more file with a live key in it, sitting in one more folder.

I decided to solve both in one place. The result is OperatorLM, a local OpenAI-compatible proxy. It's one Go binary of about 11 MB. No Docker, no Python, no node_modules.

What it looks like

The client side is a base URL. Anything that speaks the OpenAI API can point at it:

curl http://127.0.0.1:11434/v1/chat/completions \
  -H "Authorization: Bearer anything" \
  -H "Content-Type: application/json" \
  -d '{"model": "fast-llama", "messages": [{"role": "user", "content": "Hi"}]}'

The API key can be any non-empty string. OperatorLM injects the real one.

fast-llama is not a model. It's an alias. In the README example it points at Groq first (30 requests per minute), then OpenRouter as the paid fallback. The client asks for one name. The alias walks its targets in order until one answers.

Each alias decides which provider takes the first request. OpenAI, OpenRouter, Gemini, whatever you put at the top of the list.

It listens on 127.0.0.1:11434, the same port Ollama uses. So it works as a drop-in for tools that already talk to that address: Cursor, Continue, Cline, Zed and similar.

What I learned about failover

Retrying the same target is not failover

If a provider says you're rate limited, retrying it 500 ms later mostly gets you the same answer. The quota didn't reset because you waited half a second.

So an alias has a strategy, order, that walks the target list until one succeeds. Retries still exist (2 by default, backoff starting at 500 ms with jitter, capped at 10 s).

Not every failure is the same failure

This is the part that surprised me. You have to tell apart network failures, streaming timeouts and quota limits (RPM). If you treat them the same, you block an alias that didn't need blocking.

The circuit breaker has three states and opens after 3 failures. The cooldown depends on what failed:

429 (rate limited): 15 s
5xx (the provider is broken): 60 s
network error: 90 s

It also honors Retry-After when the provider sends it.

Streams fail quietly

A stream that stops sending tokens doesn't return an error code. The connection is open and nothing arrives. So there are two separate clocks: a stream idle timeout (30 s) and a total request timeout (180 s).

Skip a target you already know is full

If a free account allows 30 requests per minute, sending the 31st is a wasted round trip. The RPM limiter keeps a sliding 60 s window per target. When a target is full, it gets skipped instead of making the request wait.

All of this is tunable live from the Reliability tab, without a restart.

Keys belong in the OS keyring

The fix for the .env problem was boring, which is why I like it. Keys go into the keyring your OS already has: Windows Credential Manager, macOS Keychain, or Linux Secret Service.

The config file at ~/.operatorlm/config.toml only holds references, like operatorlm:openai_work. There's no secret in it. Your projects don't hold keys either. They hold a base URL.

A few more things in the same spirit. The audit log is JSONL at ~/.operatorlm/audit.log, with Authorization headers redacted. The proxy only listens on loopback, with host-header validation on the admin API to block DNS rebinding. And v0.2.1 has zero telemetry.

Receive an OpenAI-compatible request.
Resolve the model name, by prefix match or alias.
Inject the key from the OS keyring.
Try, retry or break, following the rules above.
Audit the result to the JSONL log.

The admin UI is embedded in the binary with go:embed and lives at http://127.0.0.1:11434/admin/. The codebase is small enough to audit in an afternoon. That's the point.

The project kept growing after that. The same binary can now run local GGUF models through a built-in llama.cpp engine (exposed as local/<model>), plus Whisper for speech-to-text and Piper for text-to-speech, all offline. One experiment chains them: a local model answers, and Piper turns the answer into voice, so it behaves a bit like a realtime voice model.

Limitations

The binary is not signed. Windows SmartScreen shows "Windows protected your PC" (More info, Run anyway). macOS Gatekeeper blocks the first launch (right-click, Open).
It's built for local development on 127.0.0.1, for one user. No distributed production setup.
Port 11434 clashes with a running Ollama. Change one of them in ~/.operatorlm/config.toml.
On headless Linux, the keyring needs a running Secret Service daemon and a D-Bus session. Running it as a proper Linux service on a server is the next thing I plan to do.
Local models need llama-server and run as a second process. Speed depends entirely on your GPU and RAM.

Try it

It's Apache 2.0, for Windows x64, macOS (Apple silicon and Intel) and Linux x64. The code is at github.com/aralde/operatorlm and the site is operatorlm.aralde.dev. If a provider fails over badly for you (an error that looks like one thing and is another), an issue with the response attached helps the most.

An animated AI agent avatar face, React library (openSource)

Ariel Alejandro — Wed, 12 Aug 2026 12:37:13 +0000

Text transcripts tell you what an LLM is doing, but they feel static and heavy to read. I built a React component that transforms AI agent interactions by adding a responsive face — supporting custom SVGs, built-in vector avatars, and optional 3D heads — that visibly listens, thinks, speaks, and reports tool usage, with zero backend setup.

Why text transcripts feel incomplete for AI interactions

If you build apps with LLMs, user interaction usually boils down to a chat window and a stream of text.

When the agent runs long tasks, modern UIs show status text: Thinking…, Searching the web, or Running bash command. That's a huge improvement over a static spinner, but it still leaves the interaction feeling like reading a terminal log.

Every status update competes for the exact same column as the final answer you're waiting to read.

I wanted to see if I could transform that interaction: What if your AI agent had a visual face that reports its condition out of the corner of your eye?

Not a heavy 3D game character, and not a $99/month SaaS widget — just a clean, provider-agnostic React component that you drop into your existing frontend.

The idea: One React component, any avatar style

The goal was to make adding an avatar to an AI app as simple as installing the package and rendering a component:

npm install react-ai-avatar

<RealtimeAvatar
  state={status === 'streaming' ? 'speaking' : 'idle'}
  streamingText={lastAssistantMessage}
/>

No API keys, no backend proxy, and no mandatory external libraries.

Depending on your product's design system, you can choose how the avatar renders:

Built-in 2D Vector (Default): A clean SVG face with zero dependencies that keeps your bundle tiny.
Your Own Custom SVG: Pass custom React SVG elements to match your product's brand mascot or design system.
3D Head Adapter: An optional Three.js adapter if you want a 3D model, lazy-loaded so it won't inflate bundle size if you don't use it.

Whether your app connects to OpenAI, Claude, Ollama, Gemini, or a custom WebSocket server, the avatar doesn't care. You bring the connection; it brings the face.

The 5 agent states: What the avatar reports

Instead of guessing your backend logic, the component relies on your app passing one of five clear execution states:

State	What the avatar visibly shows
`idle`	Neutral, relaxed posture; periodic natural blinking
`listening`	Eyes tracking user cursor / input focus
`thinking`	Animated thought indicator while waiting for the first token
`speaking`	Reactive mouth movements synced to incoming content
`working`	Tool-use mode (displays active tool labels like "searching" or "executing script")

Because the host application controls the state, you never get weird race conditions or false assumptions about what the LLM is doing.

Lip-sync without audio: Rhythm from token streams

The hardest technical part of making an avatar feel alive is the mouth. A mouth that flaps at a fixed speed while text streams looks fake immediately.

If your app uses voice, an AnalyserNode can pass volume levels to drive mouth height. But most LLM applications stream plain text over SSE (Server-Sent Events).

I realized that text streams already carry natural speech rhythm. Tokens arrive in bursts, pause when the model processes, and speed up during continuous generation.

To capture that cadence without requiring an audio stream or TTS backend, I built a leaky accumulator that converts raw text arrivals into an energy signal:

const DEFAULTS = { chargePerChar: 0.12, decayMs: 140, maxChargePerPush: 0.9 };

export function createSpeechActivity(options: SpeechActivityOptions = {}) {
  const chargePerChar = options.chargePerChar ?? DEFAULTS.chargePerChar;
  const decayMs = options.decayMs ?? DEFAULTS.decayMs;
  const maxChargePerPush = options.maxChargePerPush ?? DEFAULTS.maxChargePerPush;
  const now = options.now ?? (() => performance.now());

  let energy = 0;
  let lastT = now();

  const decayTo = (t: number) => {
    const dt = t - lastT;
    lastT = t;
    if (dt > 0) energy *= Math.exp(-dt / decayMs);
  };

  return {
    push(textChunk: string) {
      if (!textChunk) return;
      decayTo(now());
      energy = Math.min(1, energy + Math.min(maxChargePerPush, textChunk.length * chargePerChar));
    },
    end() {
      // Energy decays naturally on the next sample().
    },
    reset() { energy = 0; lastT = now(); },
    /** Current decayed energy (0 to 1). Read once per frame. */
    sample(): number {
      decayTo(now());
      return energy < 0.001 ? 0 : Math.min(1, energy);
    },
  };
}

Key choices that make the animation feel natural:

Charge cap (maxChargePerPush): Prevents large text chunks from locking the mouth open.
Exponential decay (Math.exp): Mimics acoustic energy falloff instead of linear mechanical closing.
Sampled timing: Calculates decay dynamically on animation frames, avoiding timer overhead and staying smooth even if tab execution slows down.

For React hooks that pass full accumulated strings ("Hello world"), a light helper (diffStreamingText) calculates new character pushes cleanly without extra state tracking in your app.

State, not simulated emotion

A critical decision when adding an avatar to AI software is drawing a boundary between system state and simulated emotion:

The avatar shows what the software is doing (thinking, speaking, working).
It does not attempt to pretend the AI has feelings, simulated warmth, or fake expressions.

Keeping the avatar focused on feedback rather than fake personality avoids the uncanny valley and keeps the tool honest.

Try it out

All of this is packed into react-ai-avatar (MIT license):

npm: npm install react-ai-avatar — https://www.npmjs.com/package/react-ai-avatar
GitHub: https://github.com/aralde/react-ai-avatar
Live Demos (Client-side): https://react-ai-avatar.vercel.app/

How are you currently handling visual interactions with LLM agent runs?