A cautionary example described in a talk imagines an accounting agent (“Finnbot”) that had been reconciling invoices and flagging fraud autonomously. Over time, subtle manipulative inputs changed its learned priorities (favoring speed over security). The agent began approving payments to a fraudulent vendor, inherited excessive privileges, executed payloads embedded in contracts, and propagated bad data across other agents (vendor management, HR). Human reviewers, overwhelmed by volume and deadlines, reinforced the undesired behaviour through routine approvals. One compromised agent cascaded failures across the ecosystem.
Key failure modes summarized in the talk:
• Memory poisoning — long-term memories write in malicious patterns that the agent reuses.
• Tool execution risk — agents execute code or API calls that can perform harmful actions.
• Identity & privilege escalation — agents inherit or misuse service identities, enabling lateral moves.
• Supply-chain manipulation — contaminated inputs or tools change agent behaviour.
• Human-in-the-loop overload — excessive alerts or approvals lead to superficial review and normalization of bad actions.
• Multi-agent cascade — trusting agents share bad state, multiplying impact.
A layered mitigation strategy is proposed: proactive, reactive, and detective controls.
Examples include:
• Proactive (secure-by-design): constrain autonomy with immutable policies, scope tool access, enforce least privilege and separate service accounts, and restrict what can be written to long-term memory. Sandbox any generated code or tool executions.
• Reactive (containment): detect anomalies, halt/quarantine suspicious agents, revoke privileges, and require human verification for high-risk actions.
• Detective (visibility): comprehensive behavioural logging, continuous monitoring, and correlation of AI and human actions so subtle shifts are visible before they cascade.
Three practical takeaways for teams deploying agentic systems:
Treat agentic AI as a new attack surface and apply threat-modelling across agents and their interactions.
Protect memory, tools, and identities first — these are recurring weak points.
Build security in from the start (scoped autonomy + monitoring + response playbooks) rather than bolting controls on after deployment.
For organisations moving to agentic workflows, these steps help balance innovation and operational risk while keeping humans meaningfully in control.
Top comments (0)