In "Adaptive Composition Attacks in AI-Integrated Systems CEO Setaleur Momen Ghazouani presents a conceptual framework for understanding new cybersecurity threats that arise from the interaction of secure components in AI-integrated systems. The paper, completed in July 2025, argues that these "adaptive composition attacks" are not the result of individual system weaknesses but emerge from unintended interactions between large language models (LLMs) and other systems with execution permissions
The central idea of the paper is the "self-adaptive hacking loop". This is a two-phase process where an LLM iteratively refines a malicious input based on feedback from the target system. In the generation phase, the LLM creates an initial exploit, such as a phishing email or API call. In the evaluation phase, it receives feedback on the attempt's outcome, which it then uses to improve the next attempt. This process transforms the LLM from a passive content generator into an active, problem-solving agent capable of learning to bypass security constraints. The paper distinguishes this from traditional fuzzing by highlighting the LLM's use of semantic awareness and goal-directed refinement rather than brute-force randomness
A key vulnerability discussed in the paper is "permissions and trust composition". The author notes that while components like LLMs, email clients, and shell environments are designed with individual security constraints, their integration can create new vulnerabilities. This happens when trust granted to one module is implicitly extended to another without explicit authorization, a phenomenon termed "false trust propagation". The paper provides a scenario where an LLM with access to both a corporate inbox and shell commands could be manipulated by a phishing email to execute a malicious script, even though each component individually functioned as designed
The threat is systemic, not merely technical and necessitates a reevaluation of how permissions are defined and validated in multi-agent AI systems The research also addresses the evolution of LLMs from passive generators to "coordinated attackers". When given a feedback-rich environment, an LLM can become a strategic orchestrator that plans, adapts, and executes multi-step attack strategies, engaging in reconnaissance, deception, and privilege escalation. The paper argues that current defense mechanisms, which are often based on static, signature-based detection, are insufficient against these dynamically planned, adaptive attacks
To counter these threats, the author proposes a new defensive framework centered on a "Compositional Security Mediator" (CSM). The CSM would function as a proxy between the LLM and any execution interfaces, analyzing the intent behind a sequence of actions rather than just individual commands. It would also maintain a temporary record of interactions to identify and break self-adaptive hacking loops by providing generic error messages instead of detailed feedback. Additionally, the CSM would enforce "dynamic trust scoping," limiting permissions to a specific task and revoking them upon completion to combat false trust propagation
The paper concludes by advocating for a shift in cybersecurity thinking. Instead of evaluating systems as discrete entities, the author recommends treating the entire AI-integrated ecosystem—including the dynamic interconnections between components—as the true attack surface. The paper calls for the development of standardized testing environments, like the conceptual "planner/victim" model, to simulate and study these complex attack compositions. These simulations would help reveal the "combinatorial threat space" that is invisible in isolated testing. The ultimate goal is to build defenses that are resilient to intelligent, adaptive, and collaborative machine-based threats.
Top comments (0)