If you're building an AI agent, you've probably had to choose between frameworks like:
- LangChain.
- CrewAI.
- AutoGen.
- LlamaIndex.
- OpenAI Agents SDK.
- Google ADK.
- Semantic Kernel.
That decision affects architecture, developer experience, integrations, and orchestration.
But does it materially change how secure the resulting agent is?
We ran 9,360 controlled adversarial trials to investigate.
The short version:
Attack category mattered a lot.
Model choice mattered.
Framework choice barely moved the result.
But getting to that conclusion exposed an important benchmarking problem.
Frameworks don't necessarily deliver identical prompts to the model—even when your test harness thinks they do.
We discovered this ourselves when one adapter altered the system-prompt construction enough to measurably change the results.
That led us to enforce payload identity before interpreting framework differences.
We then went beyond conventional significance testing and performed formal equivalence testing across all eight execution conditions.
All 28 pairwise comparisons satisfied our pre-defined equivalence boundary.
There was one small exception worth discussing: CrewAI retained a statistically detectable residual effect, although it remained inside the practical-equivalence threshold.
There were also two implementation surprises involving reasoning-token exhaustion and cross-framework token accounting.
Full technical breakdown:
Open-source evaluation framework:
https://github.com/AgentSafeLabs/safelabs-eval
AgentPort-Bench:
https://github.com/AgentSafeLabs/agentport-bench-results
For developers building production agents, I'm curious:
Do you choose frameworks partly based on perceived security characteristics—or mainly based on engineering and ecosystem considerations?
Top comments (0)