The rapid advancement of frontier artificial intelligence has placed agentic AI systems at the center of modern AI development. Unlike traditional language models that simply generate responses, agentic AI systems can reason, plan, use tools, interact with dynamic environments, and take actions over extended periods of time. As benchmarks improve and autonomous capabilities expand, agentic AI is increasingly shaping how advanced AI systems are designed, deployed, and integrated into real-world applications.
This rapid progress raises a fundamental question for researchers, engineers, and policymakers alike: how can agentic AI systems be built to remain safe, secure, and robust in real-world and adversarial environments? This question is the focus of a lecture in the Massive Open Online Course (MOOC) on agentic AI, which explores the emerging AI safety and AI security challenges associated with increasingly autonomous systems.
The discussion is grounded in the findings of the International AI Safety Report, led by Yoshua Bengio and authored by approximately 100 AI researchers from over 30 countries. The report emphasizes that AI risk is not a single issue but a broad and evolving spectrum of risks that grow alongside system capability. Historical precedent shows that attackers rapidly adapt to new technologies. In the case of AI, the incentives are especially high, since agentic systems can access multiple tools, influence external systems, and make decisions independently, amplifying the potential impact of misuse or compromise.
A central theme of the lecture is the distinction and interaction between AI safety and AI security. AI safety focuses on preventing harm that an AI system may cause to people, infrastructure, and the broader environment. AI security focuses on protecting the system itself from malicious actors who attempt to exploit vulnerabilities, manipulate behavior, or gain unauthorized access. In practice, these domains are inseparable. Safety mechanisms must function reliably even under active attack, and alignment techniques must be resilient to adversarial inputs and adaptive exploitation strategies.
Agentic AI systems differ fundamentally from both traditional software systems and standalone large language models. While an LLM typically maps inputs to outputs, an agentic system is a hybrid architecture composed of conventional software components and neural components such as LLMs. These systems can observe environments, maintain short-term and long-term memory, retrieve external information, invoke tools, and execute actions that have real-world consequences. This flexibility enables powerful capabilities, but it also dramatically increases the attack surface.
The lecture outlines how failures can occur across the entire agent lifecycle. Risks may emerge during deployment if models, tools, or data sources are flawed or poisoned. User inputs and environmental data may be malicious or untrusted. Model-generated outputs can themselves become attack vectors when used in tool calls, control-flow decisions, or code execution. Long-running autonomous agents may also face availability risks, including resource exhaustion and denial-of-service scenarios. A recurring issue is the implicit trust placed in model outputs within larger systems.
A major focus is placed on prompt injection attacks, both direct and indirect. Direct prompt injection occurs when an attacker provides input that overrides system-level instructions, potentially leading to sensitive data leakage or unsafe behavior. Indirect prompt injection is often more subtle and widespread. In these cases, malicious instructions are embedded in external content such as documents, resumes, web pages, emails, or calendar entries. When an agent processes this content without clearly separating data from instructions, it may unknowingly execute attacker-controlled actions.
Beyond prompt injection, the lecture examines backdoor attacks and poisoning of retrieval-augmented generation systems and agent memory. Even minimal amounts of carefully crafted malicious content can introduce conditional behaviors that activate only under specific triggers, making such attacks difficult to detect while remaining highly effective.
Given these challenges, the lecture emphasizes the need for systematic evaluation and risk assessment tailored to agentic AI. Traditional benchmarks that focus solely on model performance are insufficient for systems operating in open-ended and adversarial environments. Techniques such as automated red teaming, adversarial testing, and continuous evaluation are essential to understanding real-world agent behavior under realistic threat models.
Effective defense requires a layered security strategy. No single mechanism is sufficient. Recommended approaches include defense in depth, least privilege, privilege separation, runtime input and output guardrails, execution monitoring, and secure-by-design architectures. Limiting tool access, enforcing security policies during execution, and reducing the blast radius of compromised components are critical for building resilient agentic AI systems.
The overarching objective is not to slow innovation, but to ensure that the benefits of agentic AI are realized responsibly and distributed safely. As these systems become more autonomous and embedded in critical workflows, AI safety and AI security must be treated as foundational design requirements, not post-deployment considerations.
The full lecture and additional learning materials are available through the MOOC on agentic AI:
https://agenticai-learning.org/f25
Top comments (0)