DEV Community

Cover image for Your LLM App Passed Every Security Scan. It Still Leaked Data Through a Calendar Invite.
Alessandro Pignati
Alessandro Pignati

Posted on

Your LLM App Passed Every Security Scan. It Still Leaked Data Through a Calendar Invite.

What actually matters when you're picking AI security tooling in the UK market right now

Here's a scenario that's becoming routine. A support agent built on an LLM has access to a ticketing tool and a calendar. Someone emails in a request, the agent reads it, and buried in the email signature is a string of text instructing the model to forward customer records to an external address. No malware, no exploited CVE, no firewall rule that would ever catch it. The agent just did what the text told it to do, because from the model's point of view there's no reliable line between "instructions from my operator" and "text I happened to read."

That's indirect prompt injection, and it's the reason a WAF or a SIEM won't save you here. Those tools inspect network traffic and system logs. They have no concept of a prompt, a completion, or a tool call, so an attack that lives entirely inside natural language sails straight through.

The threat model changed, the tooling didn't

If you're deploying agents that call tools, hit a Model Context Protocol (MCP) server, or ingest untrusted content, you've inherited a threat surface your existing security stack wasn't built for: prompt injection, jailbreaks, data exfiltration through model outputs, tool-call abuse, and poisoned training or retrieval data. OWASP's LLM Top 10 has ranked prompt injection as the number one risk for LLM applications for two years running, and it's not close.

Regulators are catching up too. The UK's NCSC published an AI Cyber Security Code of Practice with principles covering secure design, testing, and monitoring for AI systems, and it explicitly calls out adversarial testing before deployment as a baseline expectation, not a nice-to-have. If you're in financial services or health, that's now sitting alongside FCA model risk expectations and NHS data governance requirements.

What to actually check before you buy or build

Skip the marketing pages and evaluate against the failure modes you'll actually hit in production:

Real-time inspection of prompts and completions, not just the request payload. You need policy enforcement at the point where the model sees input and produces output, including rate limits, content filtering, and cost controls per request.

Coverage of indirect injection, meaning content your agent retrieves from documents, web pages, or emails, not just what a user types into a chat box directly.

Visibility into the full tool-call chain. If your agent talks to an MCP server, you want monitoring on those connections specifically. This is a newer, less mature area of tooling and worth testing hard before committing.

Adversarial testing as a repeatable process, ideally automated against something like the OWASP LLM Top 10, run before every deployment rather than once at launch.

Deployment options that keep traffic inside your own infrastructure if you're in a regulated sector. Self-hosted or on-prem matters more here than it does for a generic SaaS security tool.

A rough sanity check for a gateway-style policy might look like this, just to make the shape of "policy enforcement" concrete:

policy:
  name: block-indirect-injection
  match:
    source: retrieved_content   # not user_input
  rules:
    - detect: instruction_override_pattern
      action: block
    - detect: exfil_destination_mismatch
      action: flag_and_alert
  rate_limit:
    requests_per_minute: 60
Enter fullscreen mode Exit fullscreen mode

The point isn't the exact syntax, every gateway will differ, it's that the policy needs to distinguish where content came from, because that's what tells you whether a piece of text should ever be treated as an instruction.

Who's actually building this in the UK

The market is young and moving fast. Agentic AI security spend is forecast to grow roughly 8x by 2032 according to industry analyst projections, and Gartner has estimated that a large share of enterprise applications will integrate task-specific AI agents by the end of 2026, up sharply from a couple of years ago, while governance maturity lags well behind adoption.

On the vendor side you'll find a split between gateway-first approaches (traffic inspection and policy enforcement sitting between your app and the model), endpoint or agent-level approaches (instrumenting the agent itself), and dedicated red-teaming shops (continuous adversarial testing without necessarily doing runtime enforcement). None of these is strictly better, they solve different parts of the problem, and the honest answer is that most serious deployments end up needing more than one. NeuralTrust's rundown of the UK landscape goes deeper on specific vendors if you want names attached to each category.

If tool-call abuse through MCP is your immediate concern rather than chat-based injection, this MCP security primer covers that layer specifically, since it's still the least mature part of most vendors' coverage. Worth checking out AgentSecurity as well while you're building out a shortlist, alongside whatever gateway or runtime tools you're already evaluating.

The takeaway

Don't buy a platform because it has the longest feature list or the most analyst logos. Map your actual architecture first, chatbot versus autonomous agent versus RAG pipeline versus MCP-connected tool user, and figure out which failure mode keeps you up at night. Then test candidates against real adversarial prompts, not vendor demos. If you want deeper technical background on how the injection attacks themselves work before you evaluate anything, this breakdown of prompt injection mechanics is a solid starting point, and if a gateway architecture ends up being the right fit for your setup, NeuralTrust's AI gateway page lays out what that looks like in practice.

The one number worth remembering from all of this: the average cost of a data breach hit $4.88 million globally in IBM's 2024 report, and that's before you factor in what happens when the breach vector is an AI system nobody was monitoring at the prompt layer.

Top comments (0)