AI agents that can call LLMs freely are a budget and compliance risk waiting to happen. LangSmith's new LLM Gateway addresses both without forcing you to rewrite your agent logic.
The Idea: Governance as Infrastructure, Not Afterthought
Most teams bolt on cost controls and data-privacy rules after something goes wrong - a runaway agent burns $400 in a weekend, or a support bot logs a customer's social security number into a trace. The LangSmith LLM Gateway moves those guardrails into the request path itself, sitting between your agent and the underlying model provider.
Spend limits are blocked at runtime - meaning an agent that exceeds its token budget stops before the API call goes out, not after you audit last month's invoice. PII redaction works the same way: sensitive patterns are stripped from the request and from the stored trace, so the compliance problem never reaches the model or the log in the first place. Crucially, trace continuity is preserved, so you still get full observability - you can see what the agent tried to do, just with the sensitive fields masked rather than missing.
Real Example: Wiring a Spend Limit Into Your Chain
If you're already using LangChain + LangSmith, the gateway integrates at the client initialization level. A minimal setup looks roughly like this:
from langsmith import Client
client = Client(
api_key="ls-...",
gateway_config={
"spend_limit_usd": 10.0, # hard stop per session/agent run
"pii_redaction": True, # strip emails, SSNs, phone numbers
"fallback_on_limit": "error", # or "warn" to log and continue
}
)
Once the gateway is in place, every LLM call your agent makes flows through it. The spend counter increments per token usage; hitting the ceiling raises a controlled exception your agent can catch and handle gracefully - returning a partial result or escalating to a human - instead of silently running up charges. PII redaction runs on both the outbound prompt and the inbound completion before either touches the LangSmith trace store.
This is particularly useful for Forward Deployed Engineers shipping agents to enterprise customers who have strict data-handling requirements baked into their contracts, and for product teams running multi-agent workflows where a single misconfigured node could drain a shared budget.
Key Takeaways
- Runtime governance means the guardrail executes during the agent's API call, not in a post-hoc audit - which is the only layer that actually prevents harm.
- PII redaction at the gateway level keeps sensitive data out of your trace store, which directly reduces compliance surface area without sacrificing observability.
- Spend limits paired with a configurable fallback give agents a graceful degradation path instead of an uncontrolled failure or an unlimited bill.
If you're already tracking agent runs in LangSmith, what's your current approach to capping per-run LLM spend - token budget in the prompt, external middleware, or something else?
Sources referenced: LangChain Blog - LangSmith LLM Gateway announcement
Top comments (0)