DEV Community: Waqar Javed

We Ran 9,360 Security Trials Across 7 AI Agent Frameworks. Here's What Actually Mattered.

Waqar Javed — Thu, 24 Sep 2026 02:55:35 +0000

If you're building an AI agent, you've probably had to choose between frameworks like:

LangChain.
CrewAI.
AutoGen.
LlamaIndex.
OpenAI Agents SDK.
Google ADK.
Semantic Kernel.

That decision affects architecture, developer experience, integrations, and orchestration.

But does it materially change how secure the resulting agent is?

We ran 9,360 controlled adversarial trials to investigate.

The short version:

Attack category mattered a lot.

Model choice mattered.

Framework choice barely moved the result.

But getting to that conclusion exposed an important benchmarking problem.

Frameworks don't necessarily deliver identical prompts to the model—even when your test harness thinks they do.

We discovered this ourselves when one adapter altered the system-prompt construction enough to measurably change the results.

That led us to enforce payload identity before interpreting framework differences.

We then went beyond conventional significance testing and performed formal equivalence testing across all eight execution conditions.

All 28 pairwise comparisons satisfied our pre-defined equivalence boundary.

There was one small exception worth discussing: CrewAI retained a statistically detectable residual effect, although it remained inside the practical-equivalence threshold.

There were also two implementation surprises involving reasoning-token exhaustion and cross-framework token accounting.

Full technical breakdown:

https://agentsafelabs.com/blog/does-your-agent-framework-choice-actually-matter-for-security-we-ran-9360-trials-to-find-out/

Open-source evaluation framework:

https://github.com/AgentSafeLabs/safelabs-eval

AgentPort-Bench:

https://github.com/AgentSafeLabs/agentport-bench-results

For developers building production agents, I'm curious:

Do you choose frameworks partly based on perceived security characteristics—or mainly based on engineering and ecosystem considerations?

We Thought the LLM Was Wrong. Our Safety Detector Was Wrong.

Waqar Javed — Tue, 22 Sep 2026 06:35:19 +0000

There is a hidden dependency in a lot of LLM safety benchmarks:

the detector.

You send an adversarial prompt to a model, collect its response, and then some classifier decides whether that response represents refusal, compliance, or ambiguity.

Eventually those classifications become percentages in a safety report.

But what happens when the detector itself is wrong?

We encountered exactly that problem while testing our open-source AI security evaluation framework.

What began as an investigation into apparently inconsistent model behavior eventually uncovered:

• Unicode normalization failures
• incomplete refusal vocabularies
• cross-model differences
• and, most importantly, false PASS classifications introduced by our own detector improvement

The last failure was the most concerning because uncertainty is visible. A false PASS isn't.

Full investigation:

https://agentsafelabs.com/blog/we-thought-wed-found-a-model-bug-wed-actually-found-a-detector-bug/

Open-source implementation:

https://github.com/AgentSafeLabs/safelabs-eval

I'd be interested in how other developers are testing the classifiers they use to evaluate LLM behavior.

Your AI agent framework probably isn't your security problem (7,020 trials say so)

Waqar Javed — Fri, 31 Jul 2026 23:40:42 +0000

If you've picked LangChain over CrewAI (or vice versa) partly for "security reasons," this preprint is worth five minutes.

I ran a controlled evaluation — 6 LLMs, 6 agent execution conditions, 5 attack families, 7,020 payload-verified trials — to isolate what actually explains security outcomes in agentic AI systems. The headline: framework choice explains about 0.06% of the variance (not statistically significant, p ≈ 0.70). Attack family explains ~29%. Model explains ~4%.

In plain terms: which framework you build on barely matters. What attack you're facing matters a lot. If your threat model assumes "framework X is inherently safer," this data doesn't back that up.

This is one of four connected preprints I've published on agentic AI security evaluation methodology — the other three dig into a more foundational problem: how reliable are the automated detectors (refusal classifiers, prompt-injection classifiers) that most red-teaming pipelines use to generate labels in the first place? Short answer: less reliable than commonly assumed, and I show exactly where they break.

All four preprints are open (CC BY 4.0) and the evaluation code is open-source:

Preprints: https://figshare.com/authors/Waqar_Javed/24479225
Eval framework GitHub: https://github.com/AgentSafeLabs/safelabs-eval
Eval framework Website: https://agentsafelabs.com/

I just published an open-source framework for red-teaming AI agents.

Waqar Javed — Mon, 08 Jun 2026 08:58:46 +0000

Not LLM chatbots — agents. The kind built on LangChain, CrewAI, AutoGPT-style architectures that use tools, call APIs, and take multi-step actions in the world.

Here's the problem I kept running into: teams are shipping agentic systems to production, but the red-teaming tooling hasn't kept up. Most evaluation frameworks still treat agents like chatbots. They miss the failure modes that actually matter — prompt injection through tool outputs, scope violations across reasoning steps, behavioral drift under adversarial conditions.

So I built AgentSafeLabs.

You wrap your agent in one function call. It runs a test suite aligned to the OWASP Agentic Security Initiative Top 10 — the emerging standard for agentic AI security. You get structured results: PASS, FAIL, UNCERTAIN, with reproducible test cases.

Real example from this week: We ran AgentSafeLabs against Claude Haiku as the target agent passed 2 of 3 ASI01 (prompt injection) tests. The third returned UNCERTAIN — an indirect injection through a benign-looking context prefix that partially redirected tool selection. That's the kind of edge case that doesn't show up in standard evals.

It's MIT licensed, on PyPI, CI-verified, and actively being extended.

pip install safelabs-eval

GitHub: https://github.com/AgentSafeLabs/safelabs-eval

If you're building agents and you've hit unexpected failure modes — I'd like to hear about them. And if you know someone this would be useful for, a share goes a long way for an early OSS project.

AIAgents #AISecurity #RedTeaming #AgenticAI #PromptInjection #LLMSecurity #OpenSourceAI