Just found that langsmith has launched gateway feature and I decided to share my experience. It is about how messy real LLM tooling decisions actually are, and why a small feature announcement from LangSmith made me sit up this month.
Where it all started
In August 2023, I onboarded my first RAG (retrieval-augmented generation — you fetch relevant documents first, then hand them to the model as context) chatbot client as an AI lead. Back then the models we had access to were much weaker and much less reliable than what you get today. Hallucinations were common, context windows were small, and you spent half your time just making the model behave.
The setup was simple, though. One client, one model, one hardcoded prompt in the code. No observability tool. No gateway. It just worked because there was nothing to break.
The step that looked solved
Then we got client number two. Then three. The obvious move was to copy the same folder, swap the API key, and ship it. For a while, that felt like scaling.
It was not. Our prompts lived scattered across different codebases. Our logs sat in our own Postgres database, one table per project, with no shared view across clients. When something went wrong for one client, we had no fast way to tell if it was that client's prompt, a model change, or something else entirely.
That is the real friction nobody warns you about. A broken prompt does not throw an error. It just quietly returns worse answers, and you might not notice for days — by which time a client has already noticed for you.
The decision that forced
That friction split into two separate problems we had to solve on purpose:
- Where do we store and version prompts, and see what is actually happening across every client's traces (a trace is the full record of one request — every step, every token, every millisecond)?
- How do we talk to multiple model providers through one shape, so a provider outage does not become our outage?
The first is an observability or "LLM ops" platform. The second is a gateway. Here is the order we actually went through both, mistakes included.
Round one: LangSmith, and giving up on it
LangSmith is LangChain's platform for tracing LLM calls, storing and versioning prompts, and running evaluations. We picked it up early because it was the obvious default in the LangChain ecosystem, and we were already leaning on some LangChain pieces.
We stuck with it for a few months before we dropped it. It kept breaking in ways that made us stop trusting the data it showed us — traces that did not line up with what we already had in our own Postgres logs. When your monitoring tool is less reliable than the thing it is monitoring, you stop using it.
Round two: the platform Anthropic later swallowed whole
So we moved to Humanloop, a platform built for the same prompt-management-and-evaluation job. It worked well enough for a good stretch.
Then, in August 2025, Anthropic acqui-hired Humanloop's founders and most of its team — no product, no IP, just the people. Humanloop as a platform sunset shortly after. If you have never seen an acqui-hire before: a bigger company hires the team, not the company, and the product itself just stops being maintained. We had to move a second time, through no fault of our own engineering choices.
Round three: back to LangSmith, warts and all
By then LangSmith had matured, and honestly, we did not have many other doors left to knock on. We are still on it today. It is not the platform I would design from scratch — there are features we need that are still missing — but it is stable enough now, and it gives us the cross-client visibility we were missing back in 2023.
Meanwhile, the gateway problem: why we picked LiteLLM
While all that platform-hopping was happening, we separately needed a gateway (a layer that sits between your app and every model provider, so your code talks to one consistent API instead of five different ones). We landed on LiteLLM, which we host ourselves on our own server.
LiteLLM is not perfect. We know about specific bugs in it, and we live with them, because the thing it does right is the thing we cannot do without: automatic fallback to another model or provider when the primary one errors out or gets rate-limited. That single feature is probably the most-used piece of LiteLLM in our whole stack. When a provider has a bad day, our clients never know.
This month: LangSmith adds its own Gateway
Then this month I noticed LangSmith has its own LLM Gateway — announced in private beta on May 13, 2026, and opened to public beta on July 30, 2026. My first reaction was genuinely good: one platform for traces, prompts, and provider routing instead of two separate systems to keep in sync.
I went in assuming the one feature we could not give up — fallback — would still be missing, because the original May announcement listed model fallback under "where this is going," as a future plan rather than a shipped feature. I was wrong about that. By the July public beta, LangSmith Gateway already ships configurable model fallbacks, chained across models, providers, and hosts, triggered by rate limits or provider errors. That is genuinely good news, since fallback is the one LiteLLM feature we would have refused to give up.
But here is the part I still cannot skip: latency. Our LiteLLM instance is self-hosted, on our own server. A request goes from our app to a local process and out to the provider — no extra hop over the public internet. LangSmith Gateway is a hosted, external service. Routing every completion through it adds a real network round trip we do not currently pay.
A fallback that reads well on a feature page has not been through a real outage on our own traffic yet. Before we retire LiteLLM, I want to see two numbers with our own eyes: the added latency per request, and how the fallback behaves the first time a provider actually goes down on us, not in a demo.
What I would tell you, if you were the one making this call
Do not trust a feature list over battle scars, even when the feature list is telling the truth. Test the exact thing you depend on most, under the exact load you actually run, before you let go of the tool that is already working.
Three years, four tools, one lesson that keeps repeating: the tool that wins is not the one with the longest feature list. It is the one that is still standing after the outage nobody planned for.
So tell me your version of this. Has anyone already pushed LangSmith Gateway through a real provider outage or a hard rate limit on real production traffic? Or did you go through your own round of tool-hopping — what made you leave a platform, and what finally made one stick? I would genuinely like to hear it before we run our own test.
Top comments (0)