Every time I connect an application to a new LLM provider, I run into the same problem: how much provider-specific code do I have to add? One SDK e...
For further actions, you may consider blocking this person and/or reporting abuse
The 100k lines of wrapper code story is the part that got me, that's such a familiar way to die. Curious how the plugin sandboxing works though, if a bad provider plugin hangs does the whole core stall or is dispatch isolated per event?
Thanks, glad the 100k-lines story landed. It’s the part that stuck with me too.
On sandboxing, I can’t give you a documented answer. I went through the docs and the launch post, and I haven’t run the gateway myself. What I can say:
To check it, point a route’s first candidate at a provider that accepts the connection and never responds. Then send concurrent requests to a different route and watch latency in
/ui/requests.Also tryhedgeandtrieson that route to see how they behave with a hang versus a clean error.If you test it before I do, I’d like to hear what you find.
Appreciate you separating what the docs say from your own guess, that’s rare in posts like this. The “same process, no sandbox” point answers most of what I was wondering.
The distinction you drew between a plugin that awaits forever and one that blocks a runtime thread is the one I’d worry about most. The first only hurts its own request, but the second can quietly degrade everything on that runtime, and it won’t show up in a happy-path test. A per-plugin timeout or a
spawn_blockingconvention in the plugin guidelines would go a long way here, so it’s worth asking the maintainers whether either is planned.The never-responding provider test is a good one, and I’d add a variant where the connection opens and then stalls mid-stream. A hung stream after the first token is a different failure from a hung connect, and I’d guess
hedgeandtrieshandle them differently.The plugin-first design is what stood out to me too. Using Nginx’s small-core idea for an LLM gateway makes sense, and a provider being a short file with quirks like Groq’s usage field split into their own tested plugins sounds much easier to maintain than a 100k-line wrapper.
I also liked that you covered the trade-offs. Needing a CI step for custom plugins instead of pulling a container image is a real cost, and the checklist for pre-alpha software is useful: pin versions, break it on purpose, measure time to first token.
The MCP
tool_loopsection was the most interesting part. Keeping credentials on the gateway so the client needs no MCP support is convenient, but it’s good you flagged that any client key can call any tool by default. I’d want deny lists and per-key globs set up before trying it.Have you tried the hedging setup (
hedge = "800ms") under load? I’m curious how much it increases upstream cost compared to plain retries with fallback. Thanks for sharing!Thanks for the thoughtful read! Glad the architecture section came through, since that’s what made this gateway more interesting to me than the usual “one endpoint, many providers” pitch.
On your question: I haven’t load-tested hedging yet. Everything in the post comes from the docs and the launch thread, and I didn’t want to claim results I hadn’t measured. From the docs, though, hedging should cost more than retry-with-fallback. Retries only fire after a failure, while a hedge starts a second request after the delay (800ms in the example) even if the first one is still on its way. For slow-but-healthy providers, you’d often pay for both calls to cut tail latency.
If you try it, I’d compare three setups: plain fallback, retries with
tries = 2, andhedge.Check the request log for cost per request and p95 latency. I’d also love to hear what you find, and I’ll add a follow-up if I get numbers of my own.And agreed on the MCP point: the default of any client key reaching any tool is the first thing I’d lock down.
The part I found most useful was the section on trade-offs. Posts often stop at the setup steps, but listing what to verify before relying on the tool, such as failure behavior, latency, and what gets logged, is what actually helps someone decide. I also appreciated that you separated what the docs claim from what you tested yourself. Do you plan to follow up with real numbers once you’ve run it?
The “route to the cheapest model that supports tools” idea is what got my attention. It’s a small config line (
pick = "cheapest"withneeds = ["tools"]) that would normally require custom code in an app.I’m curious how it behaves in practice, though. If a fallback model has a shorter context window or weaker tool calling, would you only notice from the request log? I’d want to test that before trusting it with real traffic.
Also, good call on
/ui/api/routes/explain. Being able to see why a request went where it did is underrated for debugging.The
/ui/api/routes/explainendpoint is easy to overlook, but it’s one of the more practical parts of this setup. Being able to see where a model name would go and why is the kind of thing that saves hours when a route behaves unexpectedly.I’m curious about the
pick = "cheapest"logic. Weighting input price plus three times output price is a reasonable default, but real cost depends on your prompt-to-completion ratio. For workloads with long outputs, such as code generation, that heuristic could rank providers differently than your actual bill would.Also, since embeddings aren’t supported yet, how are you handling retrieval pipelines alongside it? A second path just for embeddings feels like it undercuts the “one gateway” idea a bit.
Either way, a good read. The “break it on purpose, then decide” advice is the right way to evaluate any pre-alpha tool.
The 100,000+ lines of wrapper code and the rollback after a routine upgrade explain why Respan went plugin-first. It’s a useful reminder that a gateway turns into infrastructure the moment you depend on it.
What I like most is that the post treats it as a proof of concept, not a production recommendation. Pre-alpha, no official Docker image, and config that may still change are all fair reasons to wait. I’d pin the version and read the release notes before every upgrade.
One thing I’d add to your checklist: test what the
respanplugin sends by default. Sincecontentistrueunless you turn it off, anyone routing prompts with customer data should flip it tofalsefirst.Thanks for including the install steps and the
gateway.tomlexample. They make it easy to try in an afternoon.The Nginx-style small core, where even providers are plugins, is a smarter design choice than the usual “one endpoint, many providers” pitch.
The trade-off you flagged is the part I’d think hardest about. Compiled-in Rust plugins mean you get something reviewable, but every custom change becomes a CI and rebuild step instead of a config tweak. For small teams, that could decide whether they adopt it at all. The config-only template plugins should cover most cases, but I’d like to see how far they stretch before someone has to touch Rust.
I also liked the failure-test suggestion: put a bad key first in the route and watch the retry and fallback behavior in the request log. Hedging is the one I’d test carefully, since racing candidates quietly multiplies upstream cost.
Have you tried the
tool_loopMCP setup yet? Having the gateway run the tools and return only the final answer is a neat idea, but the per-keytoolsglobs look essential to keep it safe.The “pre-alpha, treat it as a proof of concept” framing is the right call, and the production checklist at the end is the most useful part of the post.
What stood out to me is the privacy default. The
respanplugin sends message content unless you setcontent = false, and that’s easy to miss in a quick setup. Anyone routing customer prompts through a gateway should check that setting before the first real request goes out.The state note also deserves more attention than it usually gets. Conversation state for translated Responses API clients living in one instance’s memory means multi-instance deployments need sticky routing, and that kind of detail tends to surface only after the first scale-up.
I’d be interested in a follow-up with real numbers: time to first token with and without the gateway, and how retries and fallbacks behave under actual rate limits.