A practical look at routing, security inspection, and the one architecture decision that determines whether your LLM traffic stays yours
Most teams bolt an AI gateway onto their stack the same way they bolt on a load balancer. Stand it up, point traffic at it, move on. That works fine until someone asks which model handled a specific failed request last Tuesday, or whether a support prompt leaked a customer's phone number. If the gateway can't answer that, it isn't really a gateway. It's a pass-through with extra steps.
Here's what's actually going on under the hood, and why one design decision matters more than the rest combined.
The request path
An AI gateway sits between your app and whatever model providers you use. It's not a dumb proxy. It reads the payload, not just the headers. A request typically moves through five stages:
- Auth - who's calling, what are they allowed to hit, what's their rate limit
- Routing - which model actually handles this
- Security inspection - prompt injection checks, PII detection, policy enforcement, run again on the response
- The upstream call - the actual model invocation
- Logging - timestamp, model, tokens, latency, cost, policy outcome
A well-built version of this pipeline adds well under 100ms of overhead. If auth fails, nothing past step 1 executes, which is easy to get wrong if you're rolling your own.
Routing is the interesting part
Calling it a "gateway" undersells it. The routing engine is really an AI router, and three strategies cover most production needs:
Load balancing across multiple instances of the same model, useful once you're hitting provider rate limits.
Fallback chains, where the gateway retries against the next provider in an ordered list if the primary errors out or times out. Your app never has to know a failure happened.
Cost-based routing, where cheap, low-complexity requests get routed to a smaller model and anything above a threshold goes to the capable one. This is one of the more direct ways teams cut LLM spend without touching application code.
A minimal fallback config looks something like this:
{
"route": "chat-completion",
"primary": "gpt-5-turbo",
"fallbacks": ["claude-sonnet-5", "llama-4-70b"],
"retry_on": [429, 503, "timeout"],
"max_attempts": 3
}
Nothing exotic. The value is in the gateway enforcing this consistently across every service that calls a model, instead of every team writing its own retry logic.
The decision that actually matters: control plane vs data plane
This is where architecture stops being an implementation detail and starts being a compliance conversation.
A single-process gateway is simple to stand up, but it usually means your prompts and completions physically transit the vendor's cloud. For a lot of teams that's a non-starter, especially with the EU AI Act's obligations for high-risk systems taking effect from August 2026, or anything falling under NIST's AI Risk Management Framework guidance on access control and data handling.
A split-plane design separates the two concerns. The control plane manages policy, routing rules, and configuration, and never touches actual traffic. The data plane enforces those policies and runs inside your own VPC, cluster, or on-prem environment. Your prompts never leave your infrastructure, only policy updates flow between the planes.
NeuralTrust's original deep dive on this covers the tradeoffs against sidecar deployments in more depth, but the short version is that split-plane gets you sovereignty without the operational cost of deploying a gateway instance alongside every service.
Treat it as a pipeline, not a monolith
The best gateways are a chain of small, swappable steps: rate limiter, auth handler, PII detector, injection scanner, router, cost tracker, response filter, audit logger. Enable what you need per route. A guardrail, in this model, is just another plugin in the chain, not a bolted-on separate system. NeuralTrust's write-up on gateway security goes deeper into what the inspection layer specifically needs to catch, and prompt injection remains the top-ranked risk in the OWASP Top 10 for LLM Applications, so that plugin isn't optional in practice.
If you're mapping out the broader agent security landscape beyond gateways, agentsecurity.com is a decent starting point for orienting yourself before picking tools.
Bottom line
If you're evaluating or building an AI gateway, don't just check whether it routes and logs. Check where the data plane actually runs. That single detail decides whether you're building infrastructure you control or renting a black box you'll have to explain to a compliance team later. If you're comparing options, NeuralTrust's gateway product is one implementation of the split-plane pattern worth looking at alongside whatever else is on your shortlist.
Top comments (0)