What Changed
For years, the evaluation of AI-driven security agents has been dominated by a singular, high-level metric: peak success rate. Whether the task involves vulnerability discovery, exploit development, or penetration testing, benchmarks have historically prioritized the ability of a model to complete a challenge regardless of the resources consumed. This approach, while effective for measuring the theoretical ceiling of a model’s capabilities, is increasingly disconnected from the realities of operational security.
Recent research titled "Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents" marks a significant shift in this paradigm. The authors argue that in real-world security operations, every reasoning step, tool invocation, telemetry query, and data enrichment request carries a tangible cost. By evaluating language-model security agents through a cost-success lens, the research demonstrates that the current obsession with "best-case" performance masks critical inefficiencies. The study introduces a framework that decomposes agent performance by inference spend and tool spend, providing a more granular view of how models actually perform in environments where budgets are finite and operational efficiency is paramount.
Technical Details
The researchers evaluated security agents against two distinct sets of challenges: offensive Cybench tasks and defensive Splunk BOTS v1 investigation challenges. By moving away from unrestricted inference budgets, the study compared models at fixed cost levels, allowing for a direct assessment of economic efficiency.
The methodology reveals distinct scaling regimes for red-team and blue-team tasks. In offensive scenarios, such as Capture The Flag (CTF) challenges, performance shows a strong correlation with additional test-time compute. As models are given more budget to reason and iterate, their ability to navigate complex exploit paths improves significantly. Interestingly, the study notes that scaled open-weight models can approach the performance of frontier proprietary systems when evaluated under these cost-competitive conditions, suggesting that the gap between open and closed models may be narrower than peak-performance benchmarks suggest.
Conversely, defensive Security Operations Center (SOC) investigation tasks do not follow this same scaling trajectory. The research indicates that success in defensive tasks is less dependent on raw reasoning power or extensive test-time compute and more reliant on disciplined tool use, the ability to navigate complex telemetry, and the efficiency of selective data enrichment. A model that simply consumes more compute to reason about an alert does not necessarily become a better investigator; rather, it often becomes a more expensive one without a proportional increase in detection or response accuracy.
Developer Implications
For developers building security agents, these findings necessitate a fundamental shift in design philosophy. If you are developing an offensive agent, the focus should remain on optimizing for reasoning depth and leveraging test-time compute to solve complex, multi-step challenges. Because offensive tasks scale with compute, developers can justify higher inference costs if the resulting success rate increases linearly or super-linearly.
However, for developers working on defensive agents, the strategy must be entirely different. The data suggests that throwing more compute at a defensive agent is an inefficient way to improve performance. Instead, developers should prioritize the "tool-use discipline" of their agents. This involves refining the agent's ability to interface with SIEMs, EDRs, and other SOC tools to retrieve only the most relevant telemetry. Reducing the noise in tool calls and improving the accuracy of selective enrichment will likely yield better results than simply scaling the underlying LLM or increasing the number of reasoning tokens.
Furthermore, the research highlights that benchmarks must evolve. Developers should stop relying solely on "success rate" as a North Star metric. Instead, they should incorporate cost-efficiency metrics into their internal evaluation pipelines. By measuring the cost-per-successful-investigation, teams can better identify which models provide actual value in a production SOC environment versus those that are simply "smart" but prohibitively expensive to operate.
Bottom Line
The era of evaluating security agents based solely on their ability to solve a challenge at any cost is coming to an end. As AI agents move from research benchmarks to production environments, economic efficiency is becoming a primary constraint. The research into cost-aware evaluation provides a much-needed reality check for the industry, distinguishing between tasks that benefit from raw scale and those that require operational discipline. For the developer, the takeaway is clear: building a useful security agent is no longer just about the model's intelligence; it is about the efficiency of its integration with the security stack. Future benchmarks will need to reflect this reality, rewarding agents that provide the most insight for the lowest operational spend.
Top comments (0)