I Explored Respan Gateway: A Plugin-First, Open-Source LLM Gateway

Every time I connect an application to a new LLM provider, I run into the same problem: how much provider-specific code do I have to add? One SDK expects one request format, another provider uses a different API, and switching models can mean changing far more than a model name. Then come retries, fallback logic, usage tracking, and debugging when a request fails.

Most gateways answer this with the same pitch: one endpoint, many providers. Respan Gateway is more interesting for how it’s built than for that pitch. It is an open-source Rust gateway with a very small core, where nearly everything, including the providers, is a plugin. I went through the documentation and the launch post to see what that design means for a developer and what I’d check before relying on it.

One caveat first: the project describes itself as pre-alpha, and its config and APIs may still change.

Why the Architecture Is the Interesting Part

Respan says it ran its hosted gateway on LiteLLM since 2024. As customers needed different API-key settings, routes, and limits, its wrapper code grew to more than 100,000 lines. Then, this January, a routine LiteLLM upgrade caused intermittent connection errors and forced a rollback. Respan says it never found the cause and now backports upstream fixes by hand. That is the company’s own account, and I haven’t tested LiteLLM myself. It does explain the design goal: add features without breaking existing ones.

The result is a small core inspired by Nginx, responsible only for dispatching events. Providers, protocols (chat, real-time voice, MCP), modalities (images, video), and gateway features like routing and fallbacks are all plugins. A provider is a short file. Respan cites Mistral’s at 22 lines, and provider quirks, such as Groq reporting token usage in its own field, are separate plugins with their own tests.

Here is how a request moves through it:

1. A client adapter hands each HTTP or WebSocket connection to the first client format that recognizes it.
2. That format exposes the request as a two-way stream of typed events.
3. Plugins run in registration order. They can choose a provider, filter either stream, make extra calls, return a response, or refuse the request.
4. A provider format opens the upstream connection.

When client and provider speak the same API, and no plugin changes the payload, the original bytes and provider-specific fields pass through untouched. When the formats differ, requests translate through shared types for messages, tool calls, results, streams, and usage.

The trade-off: plugins are natively compiled Rust crates, not dynamically loaded or interpreted. Config-only template plugins cover common needs like adding a vendor or format without a rebuild. Custom logic means compiling it in. In the Hacker News launch thread, a commenter pointed out that this means a CI step instead of pulling a container image, and raised supply-chain concerns. Respan replied that compiled-in crates can at least be reviewed, and said it pulled its Docker image before launch because some APIs may still change. Official builds are prebuilt binaries on GitHub Releases.

Installing and Making a First Request

Respan Gateway can be installed in three ways: by downloading a binary from GitHub Releases, using Cargo, or installing the Python package from PyPI. Linux x86_64 is shown here (macOS, arm64, and Windows builds are also available), using the version shown in the release:

VERSION=v0.1.0
curl -fsSL "https://github.com/respanai/respan-gateway/releases/download/$VERSION/respan-gateway-$VERSION-x86_64-unknown-linux-gnu.tar.gz" | tar xz
sudo mv "respan-gateway-$VERSION-x86_64-unknown-linux-gnu/respan-gateway" /usr/local/bin/

Alternatively, install it with Cargo:

cargo install --git https://github.com/respanai/respan-gateway respan-gateway

Or install the Python package from PyPI:

pip install respan-gateway

A source build has no web UI unless you build it first. Check the current release before copying the version number.

With no config file, every provider key in your environment becomes a provider:

export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export GEMINI_API_KEY="..."
respan-gateway

export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export GEMINI_API_KEY="..."
respan-gateway

The gateway prints the providers it found, the endpoints it serves, and a request to try. The model name in each request decides where it goes, and a prefix like openai/gpt-5.5 forces a provider. The docs’ curl example sends a request in OpenAI format to Gemini:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "gemini-3.6-flash", "messages": [{"role": "user", "content": "Hello!"}]}'

The same idea works from an SDK. Here, the OpenAI client talks to Claude:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="unused")
response = client.chat.completions.create(
    model="claude-sonnet-4-5-20250929",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)

With no client keys configured, the gateway answers only on your own machine.

The documentation describes the startup output and request log available at /ui/requests. I have not included a screenshot or claimed a local test result here.

One File: gateway.toml

Everything beyond relaying a request is configured in one gateway.toml. respan-gateway init writes a commented starter from the keys in your environment. This block is adapted from the configure docs:

client_keys = ["env:GATEWAY_KEY"]

[providers.openai]
vendor = "openai"
key = "env:OPENAI_API_KEY"

[providers.anthropic]
vendor = "anthropic"
key = "env:ANTHROPIC_API_KEY"

[providers.gemini]
vendor = "gemini"
key = "env:GEMINI_API_KEY"

[routes.cheap-and-good]
to = ["openai/gpt-5.5", "anthropic/claude-sonnet-4-5-20250929", "gemini/gemini-3.6-flash"]
pick = "cheapest"
needs = ["tools"]
tries = 2

[routes.fast]
to = ["anthropic/claude-sonnet-4-5-20250929", "openai/gpt-5.5"]
hedge = "800ms"

A client asks for cheap-and-good as if it were a model name. These are the documented behaviors that matter:

• pick sets the order candidates are tried in: first, cheapest (input price plus three times output price), fastest, weighted, or random.
• needs filters candidates to models that list capabilities like tools, reasoning, vision, or pdf.
• Fallback is on by default, so a failed candidate falls through to the next.
• tries allows 1 to 10 attempts per candidate. Backoff starts at 250ms, doubles each time, and caps at 10s.
• hedge races the candidates instead. Each starts after the delay, and the first answer wins.

Fallback models may differ in context length, tool support, or quality. Retries can waste tokens and add latency. Hedging deliberately sends more than one upstream request, which can increase cost.

Run respan-gateway check before starting (--online also calls each provider and MCP server). You can reload without a restart by sending SIGHUP, starting with --watch, or using the web UI. An invalid file is refused while the old config keeps serving. To see where a model would go and why, open /ui/api/routes/explain?model=cheap-and-good.

Before deployment, run respan-gateway check and inspect /ui/api/routes/explain?model=cheap-and-good to confirm the configuration and routing behavior in your environment.

For a failure test, configure an invalid key or unreachable provider as the first candidate, send a request, and inspect the request log to verify the actual retry and fallback behavior.

What It Can Proxy

The documented scope is chat, images, speech (including transcription), video, and real-time voice. The docs give examples like the Anthropic SDK calling a GPT model, and an OpenAI Realtime client talking to Gemini Live. It handles tools, images, streaming, and reasoning parameters across formats. Embeddings aren’t on that list, so retrieval pipelines will still need a separate path.

Sixteen providers are built in, including OpenAI, Anthropic, Gemini, Groq, DeepSeek, Mistral, xAI, OpenRouter, and Ollama. Any other OpenAI-, Anthropic-, or Gemini-style endpoint works with a base_url. Translation has limits, and Respan itself says not every feature carries over between providers. Test the exact models and parameters you use.

MCP Tools Behind the Gateway

Each [mcp.NAME] block puts an MCP server behind the gateway, with its credentials staying on the gateway. The tool_loop plugin lets any chat model use those tools mid-request. The gateway runs the tool, feeds the result back, and returns only the final answer, so the client needs no MCP support.

This is powerful and worth treating carefully. By default, any client key can make the model call any tool you offer, using the server’s credentials. Use deny lists for destructive tools, and per-key tools globs to limit what each client can reach.

Request Logs and Tracing

The web UI shows each request’s provider, latency, tokens, and cost. By default, it keeps the last 1,000 requests in memory. The request_log plugin can also append each finished request to a JSONL file, without bodies or keys.

The optional respan plugin sends each request to Respan as a trace. Its content setting defaults to true, meaning messages and replies are sent if your prompts contain sensitive data; set content = false to send metadata only.

Where It Fits

• LiteLLM is the established self-hosted option, and Respan’s own history is built on it. A newer pre-alpha project can’t match its track record, and Respan’s pitch is mainly about extensibility.
• OpenRouter is a hosted router with one API and one bill. It isn’t a competitor in the self-hosted sense, since it’s one of the gateway’s built-in vendors and you can route to it from here.
• Respan’s hosted platform runs the gateway for you, adds observability and evals, and bills centrally. That is how the company makes money. The open-source binary is the self-run version, and you can combine them by adding a Respan key as a provider.

I’ve kept these comparisons to differences I could support. Check each project’s current docs before relying on them.

Before Using It in Production

Because it’s pre-alpha, I’d treat this as a proof of concept rather than a critical dependency for now. My checklist:

• Stability: Expect config and API changes. Pin versions and read release notes.
• Provider compatibility: Test streaming, tool calls, structured output, images, and any provider-specific parameters your feature uses.
• Failure behavior: Simulate rate limits, timeouts, and bad keys. Confirm what retries, what falls back, and what happens when every candidate fails.
• Latency: Measure end-to-end with and without the gateway, including time to first token.
• Deployment: There’s no official Docker image right now, and custom plugins mean a rebuild. Plan for how you’ll build and ship it.
• Security: Set client_keys before exposing it beyond localhost (it refuses to start on a public address without them). Review the auth code you depend on. A commenter on the launch thread questioned the project’s MCP OAuth implementation, and Respan said it used Claude to generate code but reviewed every line.
• State: Conversation state for translated Responses API clients lives in one gateway’s memory, so multi-instance setups need clients pinned to the right instance.
• Privacy: Decide what the request log and the respan plugin may record.

Verdict

Respan Gateway is worth a look if you want a small, extensible gateway with swappable providers and features as plugins, and you’re comfortable with compiled Rust extensions. It’s less suited to teams that need a drop-in container and a stable API today. Run it against your own models, break it on purpose, and then decide.

To explore it yourself, start with the GitHub repository, then the open-source gateway introduction and quickstart. For how it relates to the hosted offering, see the Respan AI Gateway overview.

Frequently Asked Questions

What is Respan Gateway?

An open-source Rust LLM gateway with a small core in which providers, protocols, and features are plugins. It translates between API formats and supports routing, retries, fallbacks, MCP tools, and a request log.

Is it open source?

Yes, under your choice of MIT or Apache-2.0. Confirm against the repository’s license files.

How do I install it?

Respan Gateway can be installed by downloading a binary from GitHub Releases, using cargo install --git https://github.com/respanai/respan-gateway respan-gateway, or installing the Python package from PyPI with pip install respan-gateway.

What does it support?

Chat, images, speech, video, and real-time voice across OpenAI, Anthropic, Gemini, and other providers, plus MCP tools. Embeddings aren’t supported.

Is it production-ready?

The project calls itself pre-alpha. Validate compatibility, failure handling, security, and deployment before relying on it.

Comments

[Devers]

The plugin-first design is what stood out to me too. Using Nginx’s small-core idea for an LLM gateway makes sense, and a provider being a short file with quirks like Groq’s usage field split into their own tested plugins sounds much easier to maintain than a 100k-line wrapper.

I also liked that you covered the trade-offs. Needing a CI step for custom plugins instead of pulling a container image is a real cost, and the checklist for pre-alpha software is useful: pin versions, break it on purpose, measure time to first token.

The MCP tool_loop section was the most interesting part. Keeping credentials on the gateway so the client needs no MCP support is convenient, but it’s good you flagged that any client key can call any tool by default. I’d want deny lists and per-key globs set up before trying it.

Have you tried the hedging setup (hedge = "800ms") under load? I’m curious how much it increases upstream cost compared to plain retries with fallback. Thanks for sharing!

[Dev Stack Hub]

Thanks for the thoughtful read! Glad the architecture section came through, since that’s what made this gateway more interesting to me than the usual “one endpoint, many providers” pitch.

On your question: I haven’t load-tested hedging yet. Everything in the post comes from the docs and the launch thread, and I didn’t want to claim results I hadn’t measured. From the docs, though, hedging should cost more than retry-with-fallback. Retries only fire after a failure, while a hedge starts a second request after the delay (800ms in the example) even if the first one is still on its way. For slow-but-healthy providers, you’d often pay for both calls to cut tail latency.

If you try it, I’d compare three setups: plain fallback, retries with tries = 2, and hedge. Check the request log for cost per request and p95 latency. I’d also love to hear what you find, and I’ll add a follow-up if I get numbers of my own.

And agreed on the MCP point: the default of any client key reaching any tool is the first thing I’d lock down.

[Rafid Bottler]

The part I found most useful was the section on trade-offs. Posts often stop at the setup steps, but listing what to verify before relying on the tool, such as failure behavior, latency, and what gets logged, is what actually helps someone decide. I also appreciated that you separated what the docs claim from what you tested yourself. Do you plan to follow up with real numbers once you’ve run it?

[Ivan Annovazzi]

The 100k lines of wrapper code story is the part that got me, that's such a familiar way to die. Curious how the plugin sandboxing works though, if a bad provider plugin hangs does the whole core stall or is dispatch isolated per event?

[Dev Stack Hub]

Thanks, glad the 100k-lines story landed. It’s the part that stuck with me too.

On sandboxing, I can’t give you a documented answer. I went through the docs and the launch post, and I haven’t run the gateway myself. What I can say:

• Plugins are natively compiled Rust crates, not WASM or separate processes. So there’s no sandbox in the usual sense. They run in the same process as the core.
• The docs describe each request as its own stream of typed events, with plugins running in registration order. That suggests dispatch is per request, not one global queue. But I didn’t find anything documenting per-plugin timeouts, cancellation, or circuit breaking, so I wouldn’t assume a hung provider plugin is contained.
• My own guess, which is general reasoning about async Rust and not something the docs say: a plugin that just awaits forever would likely stall only its own request. One that blocks a runtime thread, or spins on CPU, could hurt everything sharing that runtime. That’s exactly why it’s worth testing.

To check it, point a route’s first candidate at a provider that accepts the connection and never responds. Then send concurrent requests to a different route and watch latency in /ui/requests. Also try hedge and trieson that route to see how they behave with a hang versus a clean error.

If you test it before I do, I’d like to hear what you find.

[Mayur Upadhyay]

Appreciate you separating what the docs say from your own guess, that’s rare in posts like this. The “same process, no sandbox” point answers most of what I was wondering.

The distinction you drew between a plugin that awaits forever and one that blocks a runtime thread is the one I’d worry about most. The first only hurts its own request, but the second can quietly degrade everything on that runtime, and it won’t show up in a happy-path test. A per-plugin timeout or a spawn_blocking convention in the plugin guidelines would go a long way here, so it’s worth asking the maintainers whether either is planned.

The never-responding provider test is a good one, and I’d add a variant where the connection opens and then stalls mid-stream. A hung stream after the first token is a different failure from a hung connect, and I’d guess hedge and tries handle them differently.

[Harper Elise Callahan]

The 100,000+ lines of wrapper code and the rollback after a routine upgrade explain why Respan went plugin-first. It’s a useful reminder that a gateway turns into infrastructure the moment you depend on it.

What I like most is that the post treats it as a proof of concept, not a production recommendation. Pre-alpha, no official Docker image, and config that may still change are all fair reasons to wait. I’d pin the version and read the release notes before every upgrade.

One thing I’d add to your checklist: test what the respan plugin sends by default. Since content is true unless you turn it off, anyone routing prompts with customer data should flip it to false first.

Thanks for including the install steps and the gateway.toml example. They make it easy to try in an afternoon.

[Rafid Bottler]

The /ui/api/routes/explain endpoint is easy to overlook, but it’s one of the more practical parts of this setup. Being able to see where a model name would go and why is the kind of thing that saves hours when a route behaves unexpectedly.

I’m curious about the pick = "cheapest" logic. Weighting input price plus three times output price is a reasonable default, but real cost depends on your prompt-to-completion ratio. For workloads with long outputs, such as code generation, that heuristic could rank providers differently than your actual bill would.

Also, since embeddings aren’t supported yet, how are you handling retrieval pipelines alongside it? A second path just for embeddings feels like it undercuts the “one gateway” idea a bit.

Either way, a good read. The “break it on purpose, then decide” advice is the right way to evaluate any pre-alpha tool.

[Vinay Shah]

The “route to the cheapest model that supports tools” idea is what got my attention. It’s a small config line (pick = "cheapest" with needs = ["tools"]) that would normally require custom code in an app.

I’m curious how it behaves in practice, though. If a fallback model has a shorter context window or weaker tool calling, would you only notice from the request log? I’d want to test that before trusting it with real traffic.

Also, good call on /ui/api/routes/explain. Being able to see why a request went where it did is underrated for debugging.

[Mayur Upadhyay]

The Nginx-style small core, where even providers are plugins, is a smarter design choice than the usual “one endpoint, many providers” pitch.

The trade-off you flagged is the part I’d think hardest about. Compiled-in Rust plugins mean you get something reviewable, but every custom change becomes a CI and rebuild step instead of a config tweak. For small teams, that could decide whether they adopt it at all. The config-only template plugins should cover most cases, but I’d like to see how far they stretch before someone has to touch Rust.

I also liked the failure-test suggestion: put a bad key first in the route and watch the retry and fallback behavior in the request log. Hedging is the one I’d test carefully, since racing candidates quietly multiplies upstream cost.

Have you tried the tool_loop MCP setup yet? Having the gateway run the tools and return only the final answer is a neat idea, but the per-key tools globs look essential to keep it safe.

[Elsie Rainee]

The “pre-alpha, treat it as a proof of concept” framing is the right call, and the production checklist at the end is the most useful part of the post.

What stood out to me is the privacy default. The respan plugin sends message content unless you set content = false, and that’s easy to miss in a quick setup. Anyone routing customer prompts through a gateway should check that setting before the first real request goes out.

The state note also deserves more attention than it usually gets. Conversation state for translated Responses API clients living in one instance’s memory means multi-instance deployments need sticky routing, and that kind of detail tends to surface only after the first scale-up.

I’d be interested in a follow-up with real numbers: time to first token with and without the gateway, and how retries and fallbacks behave under actual rate limits.