LiteLLM Gets You Routing. It Doesn't Get You a Security Story.

Guardrails cover the pattern-matching basics. Compliance, jurisdiction, and multi-agent traffic need something more.

If you're running LiteLLM in front of your models, you already know why it's popular. One proxy, 100+ providers, automatic failover, per-team spend tracking. It solves the "how do I not rebuild an SDK integration for every model" problem cleanly.

What it doesn't solve, by design, is security. LiteLLM ships a guardrails system, and it's genuinely useful, but it stops at pattern matching. If your deployment needs to survive an audit, that gap matters.

What's actually in the box

LiteLLM's guardrails block in config.yaml plugs into 40+ third-party providers for prompt injection detection, PII and secret masking, and content policy checks. Each guardrail attaches to one of three hook points:

• pre_call: runs on the input before the model sees it
• during_call: runs in parallel with the model call
• post_call: runs on the response, input and output both

A minimal config looks like this:

guardrails:
  - guardrail_name: "pii-mask"
    litellm_params:
      guardrail: presidio
      mode: "pre_call"
      pii_entities_config:
        CREDIT_CARD: "BLOCK"
        EMAIL_ADDRESS: "MASK"
    default_on: true

You can scope guardrails per API key or per team, which is handy for giving customer-facing endpoints stricter rules than internal tooling. For prototypes and most internal use cases, this is enough.

Where it stops being enough

The gaps show up once you're past prototyping and into something a compliance team has to sign off on:

1. Pattern matching, not intent. A prompt injection scanner catches known signatures. It won't catch a user extracting your system prompt one innocuous message at a time across a ten-turn conversation.
2. No jurisdiction awareness. LiteLLM routes on latency, cost, and load. It has no concept of "this is EU personal data, it can't go to a US endpoint without a legal basis."
3. Callback logs aren't audit trails. Sending events to Langfuse or S3 is application logging. A regulator wants immutable, inference-level records: exact input, output, model version, timestamp.
4. Guardrails only see the outer API call. Once you're running agent chains, tool calls, and orchestrator-to-subagent hops, native guardrails are blind to everything happening inside that chain.
5. You can't guardrail what you haven't tested for. The OWASP Top 10 for LLM Applications lists ten attack categories. Config-based guardrails cover a few. The rest need deliberate adversarial testing before you ship, not detection after the fact.

None of this is a knock on LiteLLM. It's a routing and cost-management tool that happens to expose a plugin point for security. Expecting it to be a full AI security layer is the actual mistake here, and it's worth reading NeuralTrust's longer breakdown of the specific enterprise gaps if you want the full list with FAQ-level detail.

Bolting on an actual security layer

If you already have LiteLLM in production and don't want to rip it out, the pragmatic move is to add a security layer as a custom guardrail rather than replacing the proxy. NeuralTrust's TrustGuard does this: it's a CustomGuardrail class that calls out to an /v1/evaluate endpoint on pre_call and post_call, and returns allow, block, or transform.

The integration is three files' worth of work: a Python guardrail class next to your config, an entry in guardrails:, and two env vars (TRUSTGUARD_API_BASE, TRUSTGUARD_API_KEY). No client-side changes, since every app is already pointing at the proxy.

Two config decisions matter more than the rest:

Fail open vs fail closed. If TrustGuard is unreachable, fail_open: false returns a 503 and blocks the request from reaching the model. fail_open: true lets traffic through uninspected and logs a warning. Default to closed unless availability trumps inspection for your use case, and if you do run open, alert on the "traffic NOT inspected" log line.

Inspection scope. current_turn checks only the latest message plus recent tool results, which keeps payload size constant. transcript checks the full conversation history on every turn, which catches slow-burn attacks but means one flagged message can block every later turn in that session.

If you're evaluating gateways from scratch rather than retrofitting one, it's worth noting NeuralTrust also ships TrustGate, a gateway with security built in natively instead of layered on. The two aren't meant to run in series, it's one or the other depending on whether you're starting fresh.

The actual takeaway

LiteLLM and a security layer solve different problems. Routing and cost control on one side, intent-based enforcement, jurisdiction rules, immutable audit trails, and multi-agent visibility on the other. If you're building anything that needs to answer "prove this is inspected" to a regulator or a security team, plan for both from the start, and if agent-to-agent and tool-call security specifically is your gap, agentsecurity.com is worth a look alongside whatever gateway you land on.

Either way: don't confuse "I added a guardrails block" with "this is secured." They're not the same claim.