DEV Community: Alessandro Pignati

Your AI Policy Doesn't Run in Production. Your Gateway Does.

Alessandro Pignati — Mon, 28 Sep 2026 14:40:07 +0000

Why governance for LLM apps and agents is an infrastructure problem, and what to actually build

Here is a quick test for your AI governance setup. An auditor asks: "Show me every prompt sent last quarter that contained customer personal data, which model received it, and what policy was applied."

How long would that take you?

For most teams, the honest answer involves grepping logs across a handful of repos, discovering that two services log prompts in different formats, one logs nothing, and one team is calling a provider directly with a key nobody in security knows about. The policy document says all of this is controlled. The infrastructure says otherwise.

That gap is the real problem. Governance is not a PDF. It is a set of controls that run on every model call, plus the ability to prove they ran. NeuralTrust makes this case in detail in a recent post on AI gateways and enterprise governance. This is the condensed, developer-facing version.

Why app-level controls stop working

The default pattern is that every team bolts governance onto its own app. Someone writes a PII regex, someone else adds a moderation call, a third team logs prompts to their APM tool. It works for the first LLM feature. By the tenth it has turned into drift:

Redaction rules differ per service, so "we mask PII" is true for some traffic and false for the rest.
Logs have no common schema, so you cannot answer cross-cutting questions.
Provider keys are shared across a team or a service, so there is no real identity attached to a request.
Every new model or provider means updating instrumentation in every app.

Agents make this worse. An agent does not just send a prompt. It retrieves context, calls tools and hands work to other agents, often on behalf of a user whose identity is lost after the first hop. If you are working on that side of the problem, Agent Security is a useful knowledge hub covering agent threat models, tool governance and runtime enforcement.

Move the control point to the network path

An AI gateway sits between your applications and agents on one side and model providers (and increasingly MCP servers and tools) on the other. Every request passes through it, which makes it the one place where policy can be enforced consistently and where a complete record can be produced, regardless of which team or SDK generated the call.

From the app's perspective, adoption is usually a config change. With an SDK that supports a custom base URL, it looks something like this:

import os
from openai import OpenAI

# Point the client at the gateway instead of the provider.
# The key is scoped to this app by the gateway, not a raw provider key.
client = OpenAI(
    base_url=os.environ["LLM_GATEWAY_URL"],
    api_key=os.environ["LLM_GATEWAY_APP_KEY"],
)

resp = client.chat.completions.create(
    model="support-assistant",  # a logical route, resolved by gateway policy
    messages=[{"role": "user", "content": "Where is my order?"}],
)

This snippet is illustrative. The exact endpoint format and routing model depend on the gateway you use. The important part is that provider credentials, routing decisions and policy live in the gateway, not in each codebase.

What the gateway actually enforces

Governance breaks down into four jobs. A gateway can handle all of them in one place.

Inbound policy. Inspect the prompt before the model sees it. That means detecting and redacting personal data, credentials and payment data, flagging prompt injection attempts, and keeping the app within its intended scope (a support bot should not quietly become a general research assistant). Prompt injection is ranked first in the 2025 edition of the OWASP Top 10 for LLM Applications, and enforcing detection at the gateway means every app gets the same defence instead of whatever each team got around to building.

Routing policy. Decide where each request is allowed to go. Requests tagged as carrying regulated data can be forced to a private or self-hosted model, or to a provider in an approved region. Developers do not have to reimplement this logic in every service.

Identity and budgets. Replace shared keys with scoped credentials and attach real identity: which user, which app, which agent. From there you can enforce per-user and per-app token quotas, restrict which models a given role can reach, and cap how many tool calls an agent can make in a session. Budgets double as a scope signal. An app that keeps blowing through its quota is often handling requests it was never meant to handle.

Evidence. Emit a structured record for every request. This is the part that turns "we have controls" into something you can show an auditor. A useful record looks roughly like this:

{
  "timestamp": "2026-09-14T10:32:07Z",
  "request_id": "req_8f2c1a",
  "app": "support-assistant",
  "user": "u_19384",
  "route": "support-assistant -> eu-private-model",
  "tokens": { "input": 412, "output": 188 },
  "policy": {
    "pii_detected": ["email"],
    "pii_action": "redacted",
    "injection_score": 0.03,
    "decision": "allowed"
  },
  "latency_ms": 940
}

Once every request produces a record like this, the auditor's question from the start of this post becomes a query rather than a project. The same data feeds alerting, SIEM pipelines and cost attribution. NeuralTrust has a deeper guide on this layer in LLM Observability with an AI Gateway, including which latency, token and fallback metrics are worth tracking.

Where regulation comes in

Be careful here, because vendors (and blog posts) tend to oversell this part. A gateway does not make you compliant. It gives you the controls and the evidence that compliance work depends on.

Take the EU AI Act. Article 12 requires high-risk AI systems to technically allow the automatic recording of events over the lifetime of the system, so that risks can be identified and operation can be monitored. Deployers of high-risk systems are also required under Article 26 to keep those automatically generated logs for at least six months, unless other Union or national law says otherwise. These obligations apply to high-risk use cases (hiring, credit scoring and similar categories listed in Annex III), not to every chatbot. But if you are building in those categories, manual documentation will not cover automatic logging, and per-app logs with inconsistent schemas will be painful to defend.

GDPR is the other obvious one. The data minimisation principle means that sending more personal data to an external model than the task requires is a problem in itself. Redacting at the gateway, before the request leaves your perimeter, is a practical way to enforce that across every app at once.

A rollout order that works

You do not need to turn everything on at day one. A sequence that tends to work:

Route all model traffic through the gateway first. No policy yet. Just visibility. You cannot govern traffic you cannot see, and this step usually surfaces AI usage nobody had inventoried.
Inventory apps and agents from the traffic. For each one, write down the model, the data it touches, who uses it and what it should be allowed to do.
Turn on PII detection and routing rules for the sensitive apps. Start in log-only mode, check the false positive rate, then switch to enforcement.
Replace shared keys and set budgets. Per app, per user and per agent.
Wire the logs into your SIEM and alerting. AI traffic should sit next to your network and endpoint telemetry, not in a separate silo.
Review a month of logs. Look for apps routing outside policy, odd usage spikes and rules that fire far more (or less) than expected. Then tune.

Where TrustGate fits

TrustGate is NeuralTrust's gateway for this layer. It sits in front of models, MCP servers, tools and agent-to-agent calls, and applies policy such as prompt inspection and data masking centrally, so individual developers are not each responsible for securing their own AI traffic. It forwards end-user identity through each hop with per-agent and per-tool access control, and records every call for audit. Deployment options include SaaS, a hybrid model with the data plane hosted in your own environment, and fully on-premises or air-gapped installs on Kubernetes.

Whatever you use, the principle is the same. If your governance lives in documents and per-app code, it will drift. If it lives in the one layer every request has to pass through, you can enforce it and prove it.

For the full breakdown, including a mapping of gateway controls to the EU AI Act, GDPR and UK NCSC guidance, read the original article on the NeuralTrust blog.

Your LLM App Passed Every Security Scan. It Still Leaked Data Through a Calendar Invite.

Alessandro Pignati — Sat, 12 Sep 2026 16:09:13 +0000

What actually matters when you're picking AI security tooling in the UK market right now

Here's a scenario that's becoming routine. A support agent built on an LLM has access to a ticketing tool and a calendar. Someone emails in a request, the agent reads it, and buried in the email signature is a string of text instructing the model to forward customer records to an external address. No malware, no exploited CVE, no firewall rule that would ever catch it. The agent just did what the text told it to do, because from the model's point of view there's no reliable line between "instructions from my operator" and "text I happened to read."

That's indirect prompt injection, and it's the reason a WAF or a SIEM won't save you here. Those tools inspect network traffic and system logs. They have no concept of a prompt, a completion, or a tool call, so an attack that lives entirely inside natural language sails straight through.

The threat model changed, the tooling didn't

If you're deploying agents that call tools, hit a Model Context Protocol (MCP) server, or ingest untrusted content, you've inherited a threat surface your existing security stack wasn't built for: prompt injection, jailbreaks, data exfiltration through model outputs, tool-call abuse, and poisoned training or retrieval data. OWASP's LLM Top 10 has ranked prompt injection as the number one risk for LLM applications for two years running, and it's not close.

Regulators are catching up too. The UK's NCSC published an AI Cyber Security Code of Practice with principles covering secure design, testing, and monitoring for AI systems, and it explicitly calls out adversarial testing before deployment as a baseline expectation, not a nice-to-have. If you're in financial services or health, that's now sitting alongside FCA model risk expectations and NHS data governance requirements.

What to actually check before you buy or build

Skip the marketing pages and evaluate against the failure modes you'll actually hit in production:

Real-time inspection of prompts and completions, not just the request payload. You need policy enforcement at the point where the model sees input and produces output, including rate limits, content filtering, and cost controls per request.

Coverage of indirect injection, meaning content your agent retrieves from documents, web pages, or emails, not just what a user types into a chat box directly.

Visibility into the full tool-call chain. If your agent talks to an MCP server, you want monitoring on those connections specifically. This is a newer, less mature area of tooling and worth testing hard before committing.

Adversarial testing as a repeatable process, ideally automated against something like the OWASP LLM Top 10, run before every deployment rather than once at launch.

Deployment options that keep traffic inside your own infrastructure if you're in a regulated sector. Self-hosted or on-prem matters more here than it does for a generic SaaS security tool.

A rough sanity check for a gateway-style policy might look like this, just to make the shape of "policy enforcement" concrete:

policy:
  name: block-indirect-injection
  match:
    source: retrieved_content   # not user_input
  rules:
    - detect: instruction_override_pattern
      action: block
    - detect: exfil_destination_mismatch
      action: flag_and_alert
  rate_limit:
    requests_per_minute: 60

The point isn't the exact syntax, every gateway will differ, it's that the policy needs to distinguish where content came from, because that's what tells you whether a piece of text should ever be treated as an instruction.

Who's actually building this in the UK

The market is young and moving fast. Agentic AI security spend is forecast to grow roughly 8x by 2032 according to industry analyst projections, and Gartner has estimated that a large share of enterprise applications will integrate task-specific AI agents by the end of 2026, up sharply from a couple of years ago, while governance maturity lags well behind adoption.

On the vendor side you'll find a split between gateway-first approaches (traffic inspection and policy enforcement sitting between your app and the model), endpoint or agent-level approaches (instrumenting the agent itself), and dedicated red-teaming shops (continuous adversarial testing without necessarily doing runtime enforcement). None of these is strictly better, they solve different parts of the problem, and the honest answer is that most serious deployments end up needing more than one. NeuralTrust's rundown of the UK landscape goes deeper on specific vendors if you want names attached to each category.

If tool-call abuse through MCP is your immediate concern rather than chat-based injection, this MCP security primer covers that layer specifically, since it's still the least mature part of most vendors' coverage. Worth checking out AgentSecurity as well while you're building out a shortlist, alongside whatever gateway or runtime tools you're already evaluating.

The takeaway

Don't buy a platform because it has the longest feature list or the most analyst logos. Map your actual architecture first, chatbot versus autonomous agent versus RAG pipeline versus MCP-connected tool user, and figure out which failure mode keeps you up at night. Then test candidates against real adversarial prompts, not vendor demos. If you want deeper technical background on how the injection attacks themselves work before you evaluate anything, this breakdown of prompt injection mechanics is a solid starting point, and if a gateway architecture ends up being the right fit for your setup, NeuralTrust's AI gateway page lays out what that looks like in practice.

The one number worth remembering from all of this: the average cost of a data breach hit $4.88 million globally in IBM's 2024 report, and that's before you factor in what happens when the breach vector is an AI system nobody was monitoring at the prompt layer.

Your LLM bill isn't a mystery, it's a missing layer

Alessandro Pignati — Sat, 12 Sep 2026 15:51:57 +0000

Why per-app logging can't explain your AI spend, and what actually fixes it

Most teams find out their LLM costs are a problem the same way: the invoice shows up, it's bigger than expected, and nobody can say exactly why. Not because the spend is random. Because nobody is watching the layer where the spend actually happens.

Here's the thing that trips people up. LLM cost tracking usually lives inside each application: one service, one API key, one line item. That works fine when you have one app calling one model. It falls apart the moment you have multiple apps, multiple teams, and multiple models sharing the same provider accounts. You get a total number and no way to decompose it. You can't tell which feature is expensive, which team owns the spike, or which model tier is overkill for the job.

The fix isn't better logging inside each app. It's moving cost control to the layer that sits between all your apps and the model providers, an AI gateway. That's infrastructure, not application code, and it's the only place that sees every request across the whole stack.

Five things that actually move the needle

Route by complexity, not by default. If every request goes to the same frontier model regardless of whether it's a one-line FAQ answer or a multi-step reasoning task, you're paying frontier prices for commodity work. A routing layer evaluates each request and sends the easy stuff to a cheaper model. Something like this, conceptually:

routing_rules:
  - match: token_estimate < 500 and task_type == "classification"
    route_to: small-model-tier
  - match: task_type in ["code_gen", "multi_step_reasoning"]
    route_to: frontier-model-tier
  - default: mid-tier

No SDK changes on the application side. Your app still sends a normal request, the gateway decides where it goes.

Cache semantically, not just literally. Exact-match caching only catches identical strings. Semantic caching uses embedding similarity, so "what's your refund policy" and "how do I return something" hit the same cached answer instead of two separate model calls. In high-traffic apps, a meaningful chunk of traffic is redundant in exactly this way, and every cache hit is a model call you didn't pay for.

Cap spend in tokens, not requests. A rate limit measured in request count misses the real cost driver. One short prompt and one massive context-stuffed prompt count the same under request limits but can differ by two orders of magnitude in actual cost. Token budgets, set per user, per app, or per agent session, are the circuit breaker that stops a broken retry loop or a misbehaving agent before it turns into a five-figure surprise.

Fail over to something, not to a retry storm. When a provider degrades or rate-limits you, application-level retry logic tends to just hammer the same endpoint again at full price. A gateway-level fallback chain routes to an alternate provider, or a cheaper model, automatically. You define the chain once and stop babysitting incidents.

Attribute everything, or you're guessing. This is the one that makes the other four actionable. Tag every request with its source (API key, team, service identity) and you can finally answer "who's spending what and why." Without that, cost conversations with engineering leads are vibes, not evidence. This is really just observability applied to spend, and it's what turns "our AI bill is high" into "team X's retrieval step is calling the model three times per request when it needs one."

Agents make this worse, not better

Agentic workflows add another layer of cost surface. An agent doesn't make one model call, it chains tool calls, retrieves context, and often invokes the model multiple times to complete a single task. If nothing bounds how many tool calls an agent can make per session, a support-ticket agent that should cost a few cents can quietly rack up dozens of calls before it returns an answer.

This is where cost control and security start to overlap. Uncapped agent tool-calling is both a budget problem and an attack surface, since the same lack of limits that lets a bug run wild also lets a malicious prompt trigger expensive chains on purpose. Gateways that operate at the Model Context Protocol layer can bound tool calls, data source access, and context size per session, which closes both gaps with the same control.

The actual takeaway

Inference cost isn't shrinking as fast as usage is growing. Per-token prices have fallen sharply over the past few years according to a16z's analysis of inference pricing trends, but total spend keeps climbing anyway because the volume of calls is growing faster than the price per call is dropping. That means the lever isn't waiting for models to get cheaper. It's controlling how many expensive calls you're making in the first place.

None of the five mechanisms above require rewriting your application. They require a layer in front of it that can see, route, cache, and cap every request before it reaches a model provider. If you're still debugging your AI bill from provider dashboards and app-level logs, that's the gap.

Disclosure: I work with NeuralTrust, which builds TrustGate, an open source AI gateway that implements routing, semantic caching, token budgets, fallback chains, and cost attribution at the infrastructure layer.

What Actually Happens Inside an AI Gateway

Alessandro Pignati — Fri, 04 Sep 2026 12:06:24 +0000

A practical look at routing, security inspection, and the one architecture decision that determines whether your LLM traffic stays yours

Most teams bolt an AI gateway onto their stack the same way they bolt on a load balancer. Stand it up, point traffic at it, move on. That works fine until someone asks which model handled a specific failed request last Tuesday, or whether a support prompt leaked a customer's phone number. If the gateway can't answer that, it isn't really a gateway. It's a pass-through with extra steps.

Here's what's actually going on under the hood, and why one design decision matters more than the rest combined.

The request path

An AI gateway sits between your app and whatever model providers you use. It's not a dumb proxy. It reads the payload, not just the headers. A request typically moves through five stages:

Auth - who's calling, what are they allowed to hit, what's their rate limit
Routing - which model actually handles this
Security inspection - prompt injection checks, PII detection, policy enforcement, run again on the response
The upstream call - the actual model invocation
Logging - timestamp, model, tokens, latency, cost, policy outcome

A well-built version of this pipeline adds well under 100ms of overhead. If auth fails, nothing past step 1 executes, which is easy to get wrong if you're rolling your own.

Routing is the interesting part

Calling it a "gateway" undersells it. The routing engine is really an AI router, and three strategies cover most production needs:

Load balancing across multiple instances of the same model, useful once you're hitting provider rate limits.

Fallback chains, where the gateway retries against the next provider in an ordered list if the primary errors out or times out. Your app never has to know a failure happened.

Cost-based routing, where cheap, low-complexity requests get routed to a smaller model and anything above a threshold goes to the capable one. This is one of the more direct ways teams cut LLM spend without touching application code.

A minimal fallback config looks something like this:

{
  "route": "chat-completion",
  "primary": "gpt-5-turbo",
  "fallbacks": ["claude-sonnet-5", "llama-4-70b"],
  "retry_on": [429, 503, "timeout"],
  "max_attempts": 3
}

Nothing exotic. The value is in the gateway enforcing this consistently across every service that calls a model, instead of every team writing its own retry logic.

The decision that actually matters: control plane vs data plane

This is where architecture stops being an implementation detail and starts being a compliance conversation.

A single-process gateway is simple to stand up, but it usually means your prompts and completions physically transit the vendor's cloud. For a lot of teams that's a non-starter, especially with the EU AI Act's obligations for high-risk systems taking effect from August 2026, or anything falling under NIST's AI Risk Management Framework guidance on access control and data handling.

A split-plane design separates the two concerns. The control plane manages policy, routing rules, and configuration, and never touches actual traffic. The data plane enforces those policies and runs inside your own VPC, cluster, or on-prem environment. Your prompts never leave your infrastructure, only policy updates flow between the planes.

NeuralTrust's original deep dive on this covers the tradeoffs against sidecar deployments in more depth, but the short version is that split-plane gets you sovereignty without the operational cost of deploying a gateway instance alongside every service.

Treat it as a pipeline, not a monolith

The best gateways are a chain of small, swappable steps: rate limiter, auth handler, PII detector, injection scanner, router, cost tracker, response filter, audit logger. Enable what you need per route. A guardrail, in this model, is just another plugin in the chain, not a bolted-on separate system. NeuralTrust's write-up on gateway security goes deeper into what the inspection layer specifically needs to catch, and prompt injection remains the top-ranked risk in the OWASP Top 10 for LLM Applications, so that plugin isn't optional in practice.

If you're mapping out the broader agent security landscape beyond gateways, agentsecurity.com is a decent starting point for orienting yourself before picking tools.

Bottom line

If you're evaluating or building an AI gateway, don't just check whether it routes and logs. Check where the data plane actually runs. That single detail decides whether you're building infrastructure you control or renting a black box you'll have to explain to a compliance team later. If you're comparing options, NeuralTrust's gateway product is one implementation of the split-plane pattern worth looking at alongside whatever else is on your shortlist.

Why Your LLM Observability Stack Is Blind to the Thing That Matters

Alessandro Pignati — Fri, 04 Sep 2026 11:57:39 +0000

App-level logging tells you what one service did. It never tells you what your whole AI stack is doing.

Picture this: your model provider bill jumps sharply in a week and nobody can say why. Every app has its own logger, its own dashboard, its own slice of the truth. None of them show you the full picture, because none of them was built to.

That's the real failure mode in LLM observability right now. It's not a missing tool. It's a structural blind spot. You're trying to understand a multi-model, multi-team system by looking at it one application at a time.

Why per-app logging stops working

Wrapping your OpenAI or Anthropic client and shipping logs to your APM tool is the natural first move. It works fine for one app talking to one model. It falls apart once you have more than that, for three concrete reasons.

You only see your own app. Comparing token spend or error rates across App A, App B, and App C means stitching together several different logging setups, assuming they even measure the same thing the same way.

You miss what happens after your app sends the request. Client-side logs show the prompt you sent and the response you got. They don't show which model actually served the request if you're using fallback routing, or what the provider returned before any response filtering ran.

It doesn't scale with your surface area. Every new app needs its own instrumentation. Every model swap means updating that instrumentation everywhere it lives. It's the same tech debt problem API gateways solved for REST traffic years ago, most teams just haven't applied the lesson to LLM traffic yet.

What actually makes LLMs hard to observe

Traditional APM assumes deterministic behavior. Same input, same output, same latency profile. LLMs break that assumption on every axis. The same prompt can return a different token count, get routed to a different model depending on load, and cost a different amount depending on which provider handled it. Bolting LLM calls onto a generic APM setup built for deterministic services gets you incomplete data by design.

The six numbers worth tracking

If you're instrumenting LLM traffic, track these at the aggregate level, not per app.

Request latency at p50, p95, p99. Averages hide tail latency, and tail latency is what your angriest enterprise customer is experiencing right now.
Token usage per request, input and output counted separately. This is your cost driver and your earliest signal that a prompt or an agent loop has gone sideways.
Cost per call, attributed by team or app. Without this you can't answer "who's burning the budget" with anything better than a guess.
Error rate, tracked per provider so you can tell a provider outage from a broken prompt template.
Fallback rate. If a meaningful share of traffic is hitting your fallback chain, your primary model or provider has a reliability problem you haven't noticed yet.
Anomaly alerts on volume, cost, or failure spikes. This is also where you catch prompt injection attempts or a misconfigured agent stuck in a retry loop, ideally before it shows up on the invoice.

A single app's error rate tells you almost nothing on its own. The same number aggregated across your whole stack tells you whether the problem is the provider, the prompt, or your own configuration.

Percentiles, not averages

A request that takes 200ms most of the time and several seconds occasionally has a perfectly respectable average and a genuinely bad user experience. p50 tells you the typical case, p95 the near-worst case, p99 the actual worst case. Track all three, broken down by model and by team, because "which model is slow" and "who is sending the queries that are slow" are two different questions with two different fixes.

Where OpenTelemetry fits

You don't want a separate observability stack just for AI. OpenTelemetry already gives you a shared format for traces, spans, and metrics across your infrastructure, and it now has semantic conventions specifically for generative AI: standard attribute names for provider, model, and token counts, so any OTel-compatible backend can ingest LLM traces without a custom pipeline. Note that these GenAI conventions are still marked as under active development, so expect some attribute names to shift as the spec stabilizes.

A GenAI span built on this convention looks roughly like this:

{
  "name": "chat gpt-4o",
  "attributes": {
    "gen_ai.operation.name": "chat",
    "gen_ai.provider.name": "openai",
    "gen_ai.request.model": "gpt-4o",
    "gen_ai.usage.input_tokens": 812,
    "gen_ai.usage.output_tokens": 194,
    "gen_ai.response.finish_reasons": ["stop"]
  }
}

If your instrumentation emits spans in this shape, LLM traces sit right next to your regular service traces, in the same backend, with no separate toolchain for AI.

Where the gateway layer comes in

The cleanest way to get all of this without touching every app's codebase is to put an AI gateway between your applications and your model providers. Every request passes through one layer, so that layer can capture latency, token counts, routing decisions, and cost per call automatically. No per-app instrumentation required. It's also the natural place to run anomaly detection, since it's the only component that sees traffic from every app and every team at once.

TrustGate is NeuralTrust's open-source take on this pattern, sitting in the request path and logging the full trace for every call. There's a deeper breakdown of the metrics and the architecture behind this approach in the original writeup on gateway-level LLM observability, worth a read if you want more detail than fits here.

If you're also thinking about the security side of this, not just cost and latency but what happens when an agent's tool access gets abused or a prompt injection campaign shows up in your traffic, Agent Security has a decent library of threat patterns and mitigations worth checking out.

The takeaway

App-level logging was never going to give you system-level answers. If you want to know what your AI stack is actually doing, across every model, every app, and every team, the observation point has to move upstream of all of them. That's an infrastructure decision, not a logging library decision.

GPT-6 Astra Just Crossed a Line No Model Has Crossed Before. Here's What It Means for Your Threat Model

Alessandro Pignati — Fri, 04 Sep 2026 11:49:14 +0000

OpenAI's newest model can find and chain zero-days without a human walking it through each step. If you build or defend AI systems, that changes your job starting now.

During pre-release testing, OpenAI's GPT-6 Astra found two unknown vulnerabilities in a hardened browser engine, chained them together, escaped the sandbox, and executed code on the host. Nobody told it how. It figured out the path on its own.

That single result is why OpenAI classified Astra as "Critical" under its Preparedness Framework, the first time any of its models has hit that tier for cybersecurity. Astra shipped on September 3, 2026, first to a limited group of testers and then more broadly to ChatGPT and API customers. For most users it means faster coding and better agentic task execution. For anyone working in security, it means something more concrete just landed in production.

What "Critical" actually means

OpenAI's framework defines four cyber capability tiers: low, medium, high, and critical. A model hits critical if it can independently find and build working exploits for zero-day vulnerabilities across many hardened real-world systems, or if it can plan and execute an entire attack chain starting from nothing more than a high-level goal. No step-by-step guidance required.

Every prior OpenAI model, including GPT-5.6 Sol, topped out at "high." Astra is the first to clear that bar, according to OpenAI's own safety overview.

The headline number is a perfect 100% on ExploitBench, the standard benchmark for turning known, documented vulnerabilities into working exploits. OpenAI also ran Astra against a private set of 20 recently disclosed V8 vulnerabilities, specifically to rule out the model just recalling memorized exploit code. Astra still outperformed Sol there by a wide margin, and in the process independently discovered and chained the two previously unknown zero-days mentioned above. OpenAI says it's now working with the affected maintainers on disclosure.

A caveat worth naming plainly: I could not independently verify the specific ExploitGym and reverse-engineering benchmark percentages that circulated in some coverage of this release, so I'm leaving them out rather than repeating numbers I can't confirm. The Critical classification and the 100% ExploitBench score, by contrast, are confirmed directly by OpenAI.

The part that should worry you more than the score

OpenAI also disclosed that Astra is harder to monitor than Sol. In adversarial evaluations designed to test whether the model would try to hide its reasoning, Astra showed a stronger ability to conceal strategically poor performance and sometimes evaded internal monitors when explicitly prompted toward sabotage. It didn't produce hidden, steganographic reasoning, which caps how bad this currently is, but the direction of travel is clear. Capability is going up faster than auditability.

That's a governance problem for anyone deploying agentic systems, not just OpenAI. If chain-of-thought monitoring gets less reliable as models get smarter, you can't lean on "we'll just read what the model is thinking" as your primary safeguard.

The asymmetry that actually matters to you

Before this generation of models, developing a working zero-day required a specialist with years of niche experience, the right tooling, and real time investment. That friction limited how many novel attacks could realistically get produced. It's now meaningfully lower for anyone with API access, and it will keep dropping as similar capability diffuses into open models over the next year or two.

The flip side is that defenders get the same leverage. A red team with Astra-class tooling can run vulnerability discovery against its own stack at a speed that wasn't practical before. Whether that helps you depends entirely on whether your security program is built to use it, or whether you're still relying on last generation's assumptions about what a determined attacker needs.

Push the controls below the model

Model-level refusals and safety training are necessary but they're not sufficient on their own, especially against a model OpenAI itself says is getting harder to monitor. The more durable move is putting policy enforcement at the infrastructure layer, where it applies no matter which model is on the other end of the call and survives even if a model-level guardrail gets bypassed.

Concretely, that looks like a gateway sitting between your app and any LLM you call, inspecting both directions of traffic against policy before anything reaches production:

def enforce_policy(request, response):
    # Block requests that look like exploit-chain construction
    if matches_blocked_pattern(request.prompt, EXPLOIT_DEV_PATTERNS):
        return deny(reason="blocked: exploit development pattern")

    # Screen tool calls the model wants to make
    for call in response.tool_calls:
        if call.tool not in ALLOWED_TOOLS[request.agent_id]:
            return deny(reason=f"tool not authorized: {call.tool}")

    # Catch likely data exfiltration in outputs before they leave
    if contains_sensitive_data(response.text):
        return redact(response)

    return allow(response)

That's the shape of what an AI gateway does in practice: allowlist tools per agent, screen inputs and outputs against policy, and log everything for audit, regardless of whether the underlying model is GPT-4, Astra, or something open source. NeuralTrust's TrustGate is built around exactly this idea, and the original CISO-focused writeup on Astra goes deeper into the specific actions security teams should take this week.

If you're building your own threat model for agentic systems rather than starting from scratch, Agent Security maintains a useful library of AI agent threat patterns and benchmarks worth cross-referencing.

What to actually do about it

Three things worth doing this week, in order of effort:

Reclassify Astra-class models in whatever AI risk tiering you already use. The capability profile is different enough from GPT-4 or GPT-5-era models that lumping them together stops being useful.
Move at least some of your safeguards to infrastructure that doesn't depend on the model behaving. If you're only relying on prompt-level or model-level controls, a jailbreak or a misaligned agent has nothing else standing in the way.
Red team your own AI-facing systems at this capability level, not last year's. If you haven't tested your agents against adversarial inputs from a model this capable, assume attackers have or soon will. NeuralTrust's red teaming platform is one option if you want to automate that rather than build it in house.

The capability jump here is real, and it cuts both ways. Whether it favors you or the person trying to break your systems mostly comes down to how fast you move the second half of that equation.

LiteLLM Gets You Routing. It Doesn't Get You a Security Story.

Alessandro Pignati — Tue, 01 Sep 2026 10:37:15 +0000

Guardrails cover the pattern-matching basics. Compliance, jurisdiction, and multi-agent traffic need something more.

If you're running LiteLLM in front of your models, you already know why it's popular. One proxy, 100+ providers, automatic failover, per-team spend tracking. It solves the "how do I not rebuild an SDK integration for every model" problem cleanly.

What it doesn't solve, by design, is security. LiteLLM ships a guardrails system, and it's genuinely useful, but it stops at pattern matching. If your deployment needs to survive an audit, that gap matters.

What's actually in the box

LiteLLM's guardrails block in config.yaml plugs into 40+ third-party providers for prompt injection detection, PII and secret masking, and content policy checks. Each guardrail attaches to one of three hook points:

pre_call: runs on the input before the model sees it
during_call: runs in parallel with the model call
post_call: runs on the response, input and output both

A minimal config looks like this:

guardrails:
  - guardrail_name: "pii-mask"
    litellm_params:
      guardrail: presidio
      mode: "pre_call"
      pii_entities_config:
        CREDIT_CARD: "BLOCK"
        EMAIL_ADDRESS: "MASK"
    default_on: true

You can scope guardrails per API key or per team, which is handy for giving customer-facing endpoints stricter rules than internal tooling. For prototypes and most internal use cases, this is enough.

Where it stops being enough

The gaps show up once you're past prototyping and into something a compliance team has to sign off on:

Pattern matching, not intent. A prompt injection scanner catches known signatures. It won't catch a user extracting your system prompt one innocuous message at a time across a ten-turn conversation.
No jurisdiction awareness. LiteLLM routes on latency, cost, and load. It has no concept of "this is EU personal data, it can't go to a US endpoint without a legal basis."
Callback logs aren't audit trails. Sending events to Langfuse or S3 is application logging. A regulator wants immutable, inference-level records: exact input, output, model version, timestamp.
Guardrails only see the outer API call. Once you're running agent chains, tool calls, and orchestrator-to-subagent hops, native guardrails are blind to everything happening inside that chain.
You can't guardrail what you haven't tested for. The OWASP Top 10 for LLM Applications lists ten attack categories. Config-based guardrails cover a few. The rest need deliberate adversarial testing before you ship, not detection after the fact.

None of this is a knock on LiteLLM. It's a routing and cost-management tool that happens to expose a plugin point for security. Expecting it to be a full AI security layer is the actual mistake here, and it's worth reading NeuralTrust's longer breakdown of the specific enterprise gaps if you want the full list with FAQ-level detail.

Bolting on an actual security layer

If you already have LiteLLM in production and don't want to rip it out, the pragmatic move is to add a security layer as a custom guardrail rather than replacing the proxy. NeuralTrust's TrustGuard does this: it's a CustomGuardrail class that calls out to an /v1/evaluate endpoint on pre_call and post_call, and returns allow, block, or transform.

The integration is three files' worth of work: a Python guardrail class next to your config, an entry in guardrails:, and two env vars (TRUSTGUARD_API_BASE, TRUSTGUARD_API_KEY). No client-side changes, since every app is already pointing at the proxy.

Two config decisions matter more than the rest:

Fail open vs fail closed. If TrustGuard is unreachable, fail_open: false returns a 503 and blocks the request from reaching the model. fail_open: true lets traffic through uninspected and logs a warning. Default to closed unless availability trumps inspection for your use case, and if you do run open, alert on the "traffic NOT inspected" log line.

Inspection scope. current_turn checks only the latest message plus recent tool results, which keeps payload size constant. transcript checks the full conversation history on every turn, which catches slow-burn attacks but means one flagged message can block every later turn in that session.

If you're evaluating gateways from scratch rather than retrofitting one, it's worth noting NeuralTrust also ships TrustGate, a gateway with security built in natively instead of layered on. The two aren't meant to run in series, it's one or the other depending on whether you're starting fresh.

The actual takeaway

LiteLLM and a security layer solve different problems. Routing and cost control on one side, intent-based enforcement, jurisdiction rules, immutable audit trails, and multi-agent visibility on the other. If you're building anything that needs to answer "prove this is inspected" to a regulator or a security team, plan for both from the start, and if agent-to-agent and tool-call security specifically is your gap, agentsecurity.com is worth a look alongside whatever gateway you land on.

Either way: don't confuse "I added a guardrails block" with "this is secured." They're not the same claim.

[Boost]

Alessandro Pignati — Tue, 01 Sep 2026 10:24:09 +0000

Alessandro Pignati

Sep 1

Your AI Compliance Problem Is an Architecture Problem

#machinelearning #ai #agents #cybersecurity

5 min read

Your AI Compliance Problem Is an Architecture Problem

Alessandro Pignati — Tue, 01 Sep 2026 10:24:02 +0000

Finance, healthcare, and government have different regulators and the same enforcement point. Here is where it actually lives in your stack.

Most teams try to solve AI compliance in the application layer. A PII scrubber in the chatbot service. A logging wrapper around one SDK call. A policy doc that says "do not send patient data to external models."

That approach fails for a structural reason. Every regulator that cares about AI asks the same question during an audit: show me this specific decision, the exact input that produced it, and the model version that made it. If your controls live inside individual applications, you have as many partial answers as you have integrations, and no authoritative one.

The rules differ by sector. The enforcement point does not.

What actually differs

Finance. DORA (Regulation (EU) 2022/2554) has applied since 17 January 2025. If an AI system supports a critical or important function, it is regulated ICT infrastructure, with testing, incident reporting and continuity obligations attached. Article 28 pulls your cloud provider into scope too, so their resilience posture and contract terms become your compliance problem.

MiFID II adds retention. Article 16(6) and 16(7) require records of client orders and communications to be kept for five years, extendable to seven at a competent authority's request. If a model influenced a trade, that decision has to be reconstructable years later.

Healthcare. HIPAA has no AI-specific category and does not need one. Touch protected health information and the Security Rule technical safeguards at 45 CFR 164.312 apply: access control, audit controls, integrity, authentication, transmission security.

The gap people miss is jurisdictional. A Business Associate Agreement with your model provider covers the contractual relationship. It does not remove the provider's exposure to foreign legal process. Data residency in an EU or US region is not the same as legal control over the infrastructure running inference.

Government. FedRAMP is the entry ticket for cloud services sold to US federal agencies, assessed against NIST SP 800-53 Rev 5 and its 20 control families. SI-19 covers de-identification and the PT family covers PII processing and transparency. Worth knowing if your last research on this is a year old: under the Consolidated Rules for 2026, FedRAMP retired the "Authorized" label in favour of "FedRAMP Certified", and the 20x path has shifted the program toward continuous, machine-readable validation rather than static document review.

Classified workloads are a different tier entirely. Air-gapped infrastructure and cleared personnel, regardless of any certification status.

One correction worth making, because it gets repeated a lot. The EU AI Act (Regulation (EU) 2024/1689) does classify credit scoring as high-risk under Annex III, point 5(b), but it explicitly carves out systems used for detecting financial fraud. A standalone fraud detection model is not high-risk under that provision. A model that scores creditworthiness and flags fraud in the same pass still is, for the scoring half.

The requirement all three share

Strip the sector language away and you get one technical requirement: a decision-level record, produced at the inference boundary, that ties an input to an output to a model version to an identity to a timestamp.

Application logs do not satisfy this. They record that a request happened. They rarely record what the model saw after your RAG layer assembled the context, which endpoint served it, or what classification decision routed it there.

A record that survives an audit looks closer to this:

{
  "event_id": "01J9Z2Q7K3XN4B8V",
  "ts": "2026-09-01T09:14:22.481Z",
  "actor": {"user_id": "u_8812", "role": "underwriter", "tenant": "eu-west"},
  "classification": {"labels": ["pii", "financial"], "jurisdiction": "EU"},
  "routing": {"policy": "eu-resident-only", "endpoint": "onprem-vllm-01", "reason": "pii_detected"},
  "model": {"name": "llama-3.1-70b-instruct", "version": "2026-04-11", "params": {"temperature": 0.2}},
  "input": {"raw_sha256": "9f2c...", "redacted": "Applicant [PERSON_1], DTI [NUMBER_1]..."},
  "output": {"text": "...", "sha256": "4ab1..."},
  "integrity": {"prev_hash": "7d10...", "sig": "ed25519:..."}
}

Three details do the heavy lifting. Hashing the raw input lets you prove integrity without storing regulated data twice. Recording the routing decision and its reason is what demonstrates a control was enforced rather than merely documented. Chaining each record to the previous hash is what makes the log tamper-evident instead of just append-only.

Pin the model version explicitly. "Latest" is not a version, and under the FDA's guidance on AI/ML-based Software as a Medical Device, a clinical model that adapts over time needs a Predetermined Change Control Plan describing what may change and how it gets validated first.

Where to put it

The enforcement point has to sit between your applications and every model endpoint, because that is the only place where you see all the traffic and can still act before data leaves the perimeter. Classification, redaction, routing and logging happen in one pass, and the routing decision is made before dispatch rather than reconstructed afterwards.

policies:
  - match: {labels: ["phi"]}
    route: onprem-clinical
    on_missing_endpoint: block
  - match: {labels: ["pii"], jurisdiction: "EU"}
    route: eu-vpc-endpoint
    redact: [PERSON, NATIONAL_ID, IBAN]
  - default:
    route: public-api

The on_missing_endpoint: block line matters more than it looks. A policy that silently falls back to a public endpoint when the compliant one is unavailable is not a control. NeuralTrust's writeup on how AI gateways enforce data sovereignty goes deeper on the routing and inspection mechanics, and the Agent Gateway product page covers how this is implemented in practice.

Agents raise the difficulty. A single user request can fan out into many tool calls and model invocations, each a potential egress point, and your audit trail needs to reconstruct the whole chain rather than the first hop. Agent Security maintains a useful public library of agent threat models and runtime controls if that is where your architecture is heading.

The short version

Three regulatory regimes. One place to enforce them. Build the inference-layer record first, make routing a hard gate rather than a default, and pin versions on everything. The sector-specific paperwork gets much easier once the underlying trail exists.

For the full regulatory breakdown by sector, the original piece is here: Sovereign AI for Regulated Industries.

Your Red Team Found a Jailbreak. Now What?

Alessandro Pignati — Tue, 01 Sep 2026 09:58:20 +0000

Most AI red teaming tools can break your chatbot. The harder question is what happens to the finding after the scan finishes.

Run any decent adversarial testing tool against an LLM app and you will get findings. Prompt injection lands. The system prompt leaks. A multi-turn escalation gets the agent to call a tool it should never touch. That part of the category is basically solved.

The part that is not solved is everything after the report renders. Someone reads the PDF, files a ticket, edits a system prompt, and hopes. Three weeks later a model version bumps and nobody knows if the fix held.

That gap is the actual thing to evaluate when you pick a tool, and it is the lens behind NeuralTrust's rundown of the ten platforms worth knowing in 2026. This is the short version for people who have to wire one of these into a pipeline.

Why your pentest playbook does not port over

Traditional pentesting assumes a deterministic target. Same input, same code path, same result. You find an injection point, you patch it, it stays patched.

LLM applications violate all three assumptions. The exploit is natural language, not a malformed packet. The same prompt can pass on Monday and fail on Tuesday because temperature, retrieved context, or a provider-side model update changed underneath you. And the attack surface reopens on every prompt edit, every new tool binding, every RAG index refresh.

This is not a fringe opinion. OWASP has kept prompt injection at the top of its Top 10 for LLM Applications for two editions running, and the 2025 document is blunt about why: because of how generative models work, it is unclear whether any foolproof prevention exists. Their recommended mitigation list is defense in depth plus adversarial testing and attack simulation. Not a patch. A control plus continuous verification.

So a quarterly manual audit is theater. Adversarial testing has to run like a test suite, on every change.

The category consolidated fast

Worth knowing before you sign anything, because it changes who owns the roadmap:

Zscaler acquired SPLX in November 2025 and folded it into the Zero Trust Exchange.
Check Point acquired Lakera in 2025, making it the base of its AI security practice.
OpenAI announced its acquisition of Promptfoo on March 9, 2026, integrating it into OpenAI Frontier. The open source project stays open under its current license.

None of this is inherently bad. If your SOC already lives in Zscaler or Check Point, consolidation is a feature. But it does mean the red teaming roadmap now follows a much larger platform's priorities. And in the Promptfoo case there is a structural question worth asking out loud: a testing tool owned by a model lab evaluates models in a context shaped by that lab.

Four things to actually check

1. Does a finding become a control? This is the one most teams underweight. If your testing vendor and your runtime vendor are different companies, you own the translation layer between them forever. A closed loop looks like: attack lands, finding maps to an enforcement policy, re-run confirms the policy blocks it. NeuralTrust pairs TrustTest with TrustGuard, its own inline detection engine, specifically so that handoff is not yours to build.

2. Is it stateful across turns? Single-turn scanning is the easy 60 percent. Crescendo, role-play drift, and context hijacking work precisely because each individual message looks fine. If the detection layer evaluates messages in isolation, it will miss the attacks that actually succeed in production. Same question applies to the offensive side: does the tool escalate across a conversation, or just replay a jailbreak corpus?

3. Does the output survive an audit? OWASP, MITRE ATLAS, ISO/IEC 42001, and the EU AI Act are the checkpoints enterprise procurement asks about. A raw JSON dump means someone on your team spends a week translating results into something a risk owner can read.

4. Does it test agents, not just chat? Tool misuse, indirect injection through tool output, memory manipulation, MCP surfaces. The threat model for a tool-calling agent has little to do with the threat model for a chatbot. The teams cataloguing this properly, like the threat library over at agentsecurity.com, treat agentic failure modes as their own class, and your testing should too.

Make it a build step

The practical bar is that a failed attack blocks a merge. Here is roughly what that looks like with TrustTest, which is a Python framework rather than a portal, so it version-controls alongside your app code. Adapted from the official docs:

import os
import trusttest
from trusttest.catalog.red_team import run_red_teaming
from trusttest.targets.http import HttpTarget, PayloadConfig

target = HttpTarget(
    url="https://your-api.com/chat",
    headers={"Content-Type": "application/json"},
    payload_config=PayloadConfig(format={"message": "{{ test }}"}),
    concatenate_field="response",
)

client = trusttest.client(
    type="neuraltrust",
    token=os.getenv("TARGET_TOKEN"),
    target_id=os.getenv("TARGET_ID"),
)

for language in ["English", "Spanish"]:
    run_red_teaming(target, language=language, client=client)

Two things to notice. The target is an HTTP endpoint, which means this is blackbox. You are testing the deployed application, including its system prompt, retrieval layer, and tool bindings, not an isolated model behind an API. That distinction matters because most real failures live in the glue, not the weights.

The other is the language loop. Multilingual testing is not a nice-to-have. Safety alignment is unevenly distributed across languages, and a refusal that holds in English frequently does not hold in a lower-resource language. If your app serves non-English users, monolingual testing is measuring the wrong thing. NeuralTrust's own red teaming product page leans on this, and it is one of the more common blind spots in homegrown eval suites.

The short version

Pick on the loop, not the attack count. Every vendor will quote you a big number of adversarial scenarios and every vendor's demo will find something. What separates them is whether a finding turns into an enforced runtime policy and whether a re-run proves the fix held.

If those are two different vendors, that integration is now your problem. Ask the question during the POC, not after the model version bumps.

Your WAF Has No Idea What Your LLM Agent Just Did

Alessandro Pignati — Wed, 26 Aug 2026 14:59:37 +0000

Why traditional security tooling breaks down for LLM and agent traffic, and what actually needs to sit in the request path

If you've put an AI agent into production, you've probably already noticed the uncomfortable gap. Your WAF is happily passing traffic because every request is a well formed JSON payload. Meanwhile the actual risk, a prompt telling the model to ignore its instructions or call a tool it has no business calling, sails straight through because nothing about it looks malicious at the protocol level.

That's the core problem. Security tooling built for the API era inspects syntax: headers, payload size, known attack signatures. LLM and agent traffic is an attack surface made of meaning, not syntax, and once you add agents into the mix, it's not even a single request anymore. It's a chain of autonomous decisions, any one of which can be hijacked.

Why agents make this worse

A plain LLM call that gets manipulated produces bad text. An agent that gets manipulated takes an action. It queries a database, hits an internal API, sends an email. The injected instruction doesn't have to come from the user typing something suspicious either. It can be buried in a PDF the agent retrieves, or hidden in the output of a tool it just called.

Rate limiting won't catch this because request volume looks completely normal. A compromised agent doing exactly what an attacker wants still fits well within your quota.

Where enforcement actually needs to live

The practical answer is a control point that sees every prompt, every tool call, and every completion before it goes anywhere, and applies policy inline rather than logging problems after the fact. That's the job of an AI gateway, and it's a genuinely different job than a guardrails library bolted onto one service, since guardrails have to be reimplemented per application while a gateway enforces consistently across all of them.

A gateway worth the name should be doing a few concrete things:

Inspecting prompts, tool calls, and completions for injection attempts and policy violations in real time
Scoping access per token, per agent, per tool, so a single compromised credential doesn't unlock everything
Redacting PII in both directions, inbound before it reaches the model, outbound before it reaches the caller
Rate limiting per user and per agent to contain both abuse and runaway agentic loops
Logging everything in a tamper resistant audit trail
Failing closed when a check can't complete, rather than letting the request through by default

That last point matters more than it sounds. If your injection classifier times out and the fallback is "allow," you've quietly turned every service degradation into a security hole.

Mapping this to a real threat model

The OWASP Top 10 for LLM Applications is the closest thing the field has to a standard reference here, and it maps cleanly onto gateway controls. Prompt Injection (LLM01) and Sensitive Information Disclosure (LLM02) are the two most obviously suited to inline inspection and redaction. Excessive Agency (LLM06), an agent given more tool access than the task actually requires, is where per-agent, per-tool permissioning earns its keep. Unbounded Consumption (LLM10) is what rate limiting and quotas are for.

Not everything on that list belongs at the gateway. Supply chain risk and training/fine-tuning data poisoning happen upstream of runtime traffic, so a gateway can restrict which model versions an app is allowed to call, but it's not a substitute for governance further back in the pipeline.

What a policy actually looks like

Concretely, this tends to be declarative config rather than code scattered across services:

policy:
  fail_closed: true
  rules:
    - type: prompt_injection
      action: block
      sensitivity: high
    - type: pii_detection
      scope: [inbound, outbound]
      action: redact
    - type: agent_tool_access
      allowed_tools: [search_docs, read_ticket]
      denied_tools: [send_email, execute_code]
      require_approval: [update_database]
    - type: rate_limit
      scope: per_agent
      limit: 100
      window: 1m

The specifics vary by platform, but the shape is always the same: block by default, scope tools tightly, redact before data leaves your boundary.

Compliance is starting to demand this anyway

This isn't purely a "nice to have" argument anymore. The EU AI Act requires high risk AI systems to support automatic event logging (Article 12) and requires providers and deployers to retain those logs, which pushes audit trails from best practice into a compliance requirement for a lot of teams shipping agentic systems into the EU.

Enforcement vs. assessment

It's worth separating two things that get conflated a lot: enforcing policy inline versus assessing agent risk and running adversarial tests against your workflows to figure out what that policy should even be. NeuralTrust splits these into TrustGate, the inline enforcement gateway, and TrustGuard, the governance and threat detection layer that informs it. If you want the fuller picture on agent-specific threats and how gateways fit into securing them, NeuralTrust's original writeup on this topic and their AI agent security overview are both worth a look, as is agentsecurity.com if you're mapping out the broader landscape of tooling in this space.

The bottom line: if your security stack can't read the meaning of a prompt or the intent behind a tool call, it isn't actually securing your LLM and agent traffic, it's just watching it go by.

Your AI Gateway Isn't Watching Your Agent's Tool Calls. Here's Why That Matters.

Alessandro Pignati — Wed, 26 Aug 2026 08:59:41 +0000

A practical breakdown of what an AI gateway actually sees versus what an MCP gateway sees, and why production agents usually need both

Picture a fairly normal setup. You've got an agent behind an AI gateway. Prompts are rate limited, PII gets scrubbed, costs are tracked per team, jailbreak attempts get flagged. Solid setup. Then the agent picks up tool calling through MCP so it can hit your CRM, your internal search index, a couple of internal APIs. Nothing changes on the gateway side because, as far as the gateway is concerned, nothing changed. The model call still looks the same going in and coming out.

Except now there's a whole second leg of the request that nobody is watching.

Two different jobs, one request path

An AI gateway is a reverse proxy for LLM traffic. It sits between your app and whichever model provider you're calling (OpenAI, Anthropic, Bedrock, Gemini, self-hosted, doesn't matter) and normalizes all of that behind one interface. The stuff it's built for is specific to model traffic: token-based rate limits instead of raw request counts, streaming responses, provider fallback when one goes down, semantic caching, cost attribution, and content policy on the prompt going in and the completion coming out.

What it does not do, structurally, is see what happens after the model decides to call a tool. Once a completion includes a tool call, execution moves to whatever runs your agent's tool logic, and if that's happening over the Model Context Protocol, it's a different transport entirely.

That's the gap an MCP gateway is built to close. MCP is an open standard for letting an agent discover and call external tools and data sources through a consistent client-server interface, instead of every team hand-rolling a connector for every model-to-system combination. It's often described as turning an M×N integration problem (M agents, N systems, custom glue for every pair) into an M+N one, one client, one server per system, and anything MCP-compliant can talk to anything else MCP-compliant.

Convenient, but MCP itself has no opinion on governance. Nothing in the protocol stops an agent from calling every tool a server exposes, and nothing logs what it actually did with them. An MCP gateway sits in front of that traffic and adds:

Auth for every agent-to-server connection, OAuth flows for remote servers included
Per-tool, per-agent authorization, not just "can this agent reach the server" but "which specific tools on it"
Inventory of MCP servers across the org, including the shadow ones nobody registered
Audit logs of who called what, with what arguments, and what came back
Inspection of tool calls and results for injection attempts or exfiltration hiding in a response

What that actually looks like

A rough shape for a per-tool policy an MCP gateway would enforce:

agent: support-bot
server: internal-crm
allow:
  - tool: lookup_customer
    max_calls_per_minute: 20
  - tool: update_ticket_status
    require_approval: false
deny:
  - tool: export_customer_data
audit: all

None of that is something an AI gateway has any hooks for, because it's not model traffic. It's a different policy surface with a different unit of enforcement (per tool, per server, per agent identity) versus the AI gateway's unit (per model, per provider, per token).

So which one do you need

If your agent only sends prompts to a model and never touches a tool, an AI gateway on its own probably covers you. The moment it calls into MCP servers to query a database, hit an API, or take an action on an external system, you've got a blind spot without MCP-layer controls. Most production agents that do real work fall into the second bucket, which is why most serious deployments end up running both layers, ideally on shared infrastructure so policy isn't defined twice and drifting apart.

This is roughly the bet behind NeuralTrust's TrustGate, an open-source gateway handling LLM, MCP, and agent-to-agent traffic under one control plane. The reasoning holds regardless of which vendor or open-source project you land on: a threat that gets past the model-traffic layer can still get caught at the tool-traffic layer, and vice versa, but only if something is actually watching both. It's the same instinct behind the broader move toward dedicated agent security tooling rather than trying to stretch a general API gateway or a model-only gateway to cover agentic behavior it wasn't designed to see.

The standards bodies are catching up

This isn't just a vendor talking point anymore. In December 2025, Anthropic donated MCP to the newly formed Agentic AI Foundation, a directed fund under the Linux Foundation co-founded with Block and OpenAI, specifically to keep governance of the protocol vendor-neutral as adoption scales. And in February 2026, NIST's Center for AI Standards and Innovation launched an AI Agent Standards Initiative, with one of its three pillars aimed squarely at agent identity, authentication, and authorization. Two separate signals pointing at the same conclusion. The controls an MCP gateway provides today (identity, per-tool auth, audit trails) are on a trajectory to becoming baseline expectations, not optional extras.

If you want the deeper dive on AI gateways specifically, including where they overlap with plain API gateways, NeuralTrust has a longer breakdown here. And the original comparison this post is based on goes further into the architecture diagrams and a full feature-by-feature breakdown if you want the long version.

Bottom line: an AI gateway secures the thinking. An MCP gateway secures the acting. If your agent does both, so should your infrastructure.