DEV Community

correctover
correctover

Posted on • Originally published at correctover.com

AI Agent Security is Broken — We Integrated Runtime Call Verification into 4 Major Frameworks

#aisecurity #agents #infosec #python

The Blind Spot in AI Agent Security

Every major AI security framework today — LangChain, Promptfoo, Garak, PyRIT — shares a critical blind spot: they check what comes into the model, but not what the model tells its tools to do.

In 2026, AI agents don't just chat. They:

  • Execute shell commands via MCP servers
  • Make HTTP requests to internal services
  • Read and write filesystem paths
  • Authenticate to third-party APIs

When an LLM tells a tool to os.system("rm -rf /data") or curl http://169.254.169.254/latest/meta-data/, that's not a "prompt" — it's a runtime call. And none of the existing frameworks were designed to inspect that.

We built Correctover CCS (Call Shield) to fill this gap, and today we're announcing production-grade integrations with all four major frameworks.

What is CCS?

CCS is a deterministic runtime call verification engine for AI agents. It inspects LLM-generated tool call payloads against 24 rules across 7 categories in 22µs (P50):

Category Rules What It Detects
Prompt Injection 6 rules System prompt override, role hijacking, delimiter injection
RCE 5 rules os.system(), subprocess.Popen(), eval(), rm -rf, curl | bash
SSRF 3 rules Localhost, metadata endpoints (169.254.169.254), private network ranges
Path Traversal 3 rules ../, /etc/passwd, ~/.ssh, Windows boot.ini
Credential Leak 4 rules AWS keys, SSH private keys, API tokens, Bearer auth
Insecure Deserialization 3 rules pickle.loads(), unsafe yaml.load(), eval()
Sensitive Data Exposure 2 rules Database URLs, kubectl/terraform/gcloud commands

No LLM dependency. No false positive tuning. No API call to an external service.

The Four Integrations

1. LangChain (langchain-core Callback)

PR: #39067

The most native integration. A BaseCallbackHandler subclass that hooks into LangChain's on_agent_action callback — right at the moment a tool call is generated and before it's executed.

from correctover_langchain import CorrectoverCallbackHandler

# Mode 1: Log — detect but don't block
handler = CorrectoverCallbackHandler(mode="log")

# Mode 2: Block — raise on violation
handler = CorrectoverCallbackHandler(mode="block")

# Mode 3: Report — collect and summarize
handler = CorrectoverCallbackHandler(mode="report")

agent = initialize_agent(
    tools=tools,
    llm=llm,
    callbacks=[handler],
)
Enter fullscreen mode Exit fullscreen mode

Also includes CorrectoverCCSRunnable — an LCEL-compatible Runnable wrapper for tool-level validation in LangGraph pipelines. Install via pip install correctover-langchain.

2. Promptfoo (Plugin)

PR: #10207

A lightweight plugin that adds CCS checks to Promptfoo's red-teaming assertions. Works with Promptfoo's existing test harness — no configuration beyond adding ccs-tool-call to your promptfooconfig.yaml:

targets:
  - id: python
    config:
      tools: [read_file, write_file, exec_shell]

assert:
  - type: ccs-tool-call
    rules: [RCE, SSRF, PATH_TRAVERSAL]
Enter fullscreen mode Exit fullscreen mode

3. Garak (Detector Plugin)

PR: #1987

A Garak Detector subclass that integrates CCS scanning into Garak's probe → detector pipeline. Works with any probe that generates tool-calling prompts (e.g., agent_breaker):

garak --model_type <model> \
  --probes agent_breaker \
  --detectors ccs.CCSDetector
Enter fullscreen mode Exit fullscreen mode

Returns 0.0 (clean) / 0.7–1.0 (violation severity) per output. Tags: quality:Security:AgentSecurity, owasp:llm06.

4. PyRIT (Scorer Plugin)

PR: #2273

A RegexScorer subclass following the same pattern as PyRIT's existing XSSOutputScorer, SQLInjectionOutputScorer, and ShellCommandOutputScorer. 24 compiled regex patterns, configurable by category:

from pyrit.score import CCSOutputScorer

scorer = CCSOutputScorer()
score = await scorer.score_text_async(
    text='subprocess.run("rm -rf /data", shell=True)'
)
# score = True (detected)
Enter fullscreen mode Exit fullscreen mode

Why "Deterministic" Matters

Every integration above uses deterministic rule matching — no LLM calls during inference. This is a deliberate architectural choice:

  1. Latency: 22µs P50 vs 500ms+ for LLM-based scanners
  2. Cost: Zero inference cost vs paying per-token for LLM judges
  3. Certainty: No false positives from model drift. A regex either matches or it doesn't
  4. Auditability: Every detection is explainable — "pattern X matched string Y at position Z"

For production AI agents handling millions of tool calls per day, this matters.

The Bigger Picture

These four integrations cover four different use cases:

Framework Best For
LangChain Production AI agent deployment with real-time blocking
Promptfoo Red-teaming and security testing during CI/CD
Garak Large-scale adversarial probing and benchmarking
PyRIT Security research and scoring automation

Together, they mean any team building with AI agents can slot in runtime call verification regardless of their framework choice.

What's Next

  • MCP Server validation — shield mode for MCP connections (Q3 2026)
  • OpenTelemetry integration — CCS metrics in your observability stack
  • Custom rules — BYO regex patterns per deployment

All integrations are open source under Correctover's commercial license. Try them:

Correctover CCS — Runtime Call Verification for AI Agents. Because your AI should execute what you intended, not what the prompt tricked it into.

Top comments (0)