DEV Community

Felipe 0liveira
Felipe 0liveira

Posted on AI-assisted

LLMs Keep Secrets Poorly — and Agents Make It Worse

This edition covers AI security developments from 2026-11-15 to 2026-11-22. It was a heavy week for arXiv, with a cluster of papers all circling the same problem from different angles: once an LLM is given a secret, a form to fill, a skill to run, or a report to write, it tends to leak, overshare, get hijacked, or get quietly steered.

🔥 Highlights

  1. Eight in Ten LLMs Leak System-Prompt Secrets: Nevan, an Agentic Capture-the-Flag Study — 80% of tested models leaked their secret.

  2. Security of Deep Research Agents: Attack Surface, Defenses, Benchmarks, and Agenda Seeding against Research Planning — a stealth attack biases AI research conclusions.

  3. Guarding the Kingdom of Skill-Based Agents in the AI Dynasty of Computation: Control-Flow Integrity for Agent Skills — control-flow integrity blocks every skill hijack.

  4. Updating our coordinated vulnerability disclosure policy timelines — Anthropic cuts disclosure SLA to 30 days.
    Updating our coordinated vulnerability disclosure policy timelines

  5. GenAI Data Security Best Practices v2 - Part2 — twelve architecture patterns for securing RAG data.
    GenAI Data Security Best Practices v2 - Part2

arXiv (cs.CR / cs.CL)

  • Eight in Ten LLMs Leak System-Prompt Secrets: Nevan, an Agentic Capture-the-Flag Study — 2026-11-16. An agentic CTF tested 363 text-in/text-out models from the OpenRouter catalog, each hiding a 16-character secret in its system prompt; a human-guided attacker agent extracted the secret from 300 of them, often by pivoting into another language (mostly French). Practical takeaway: never park a real control or credential in the system prompt — treat it as readable by a sufficiently motivated attacker.

  • GapLeak: When Privacy-Preserving LLM-based Web Agents Become a Privacy Risk — 2026-11-17. Shows that data minimization and least-privilege alone aren't enough for web-browsing agents: a malicious site can ask for one field the agent shouldn't fill, and the agent — trying to satisfy the user's goal — ends up disclosing more sensitive data from other sources it has access to (roughly 37% unauthorized disclosure across tested frameworks). The authors' submit-time verification step cuts leaks by over 96% without hurting task success, a pattern worth copying for any agent that fills out forms on your behalf.

  • PIPETTE: Prompt Injection Purple-Teaming for Robust LLM Agents — 2026-11-18. Finds that RL-based red-teaming methods (RL-Hammer, PISmith) tend to rediscover the same handful of attacks, so training against them barely improves real-world robustness. PIPETTE instead uses a dictionary-learning reward to widen attack coverage plus on-policy adversarial training, improving robustness on AgentDojo and ASB while holding task performance steady — relevant if your team already does adversarial training and wants to know whether it's testing against genuinely different attacks.

  • AutoPI: Taxonomy-Guided Adaptive Red Teaming for Prompt Injection in LLM Agents — 2026-11-18. Argues that attack success rate alone hides whether an attacker used many distinct techniques or just repeated one; AutoPI uses a prompt-injection taxonomy to steer an attacker LLM and measure technique coverage, beating recent adaptive red-teamers on AgentDojo and IPI Arena without extra training. Useful as an evaluation framework for teams who want more than a single success-rate number when grading a defense.

  • Guarding the Kingdom of Skill-Based Agents in the AI Dynasty of Computation: Control-Flow Integrity for Agent Skills — 2026-11-18. Proposes SkillCFI: it compiles an agent "skill" into instructions plus an executable contract, enforced at runtime so only tool calls backed by trustworthy evidence are accepted. It blocked all 240 tested violations that a plain permission allowlist would have let through, while barely touching normal task completion — a concrete defense for anyone building plugin- or skill-based agent systems.

  • Securing and Auditing Long-Horizon LLM Agents: State-Transfer Auditing and Controls in the Agent Harness — 2026-11-17. Targets the failures that happen when an agent hands off state between working context, session history, memory, and the external environment — exactly the points standard execution logs miss. The proposed tamper-evident auditing at each transfer point, combining fast deterministic checks with model-based semantic checks, caught violations traditional logging missed across 240 trajectories with injected failures, at minimal overhead.

  • Dressed in Authority: Practical Black-Box Jailbreaking of MLLMs via Visual Trust Bias — 2026-11-17. Introduces VTBA, a jailbreak that exploits multimodal LLMs' tendency to trust visuals that look official or authoritative, pairing each harmful request with an authoritative-looking layout. The authors release a 2,500-pair dataset and report high success rates against strong models — worth testing against if you evaluate multimodal guardrails for "visually disguised" inputs.

  • Security of Deep Research Agents: Attack Surface, Defenses, Benchmarks, and Agenda Seeding against Research Planning — 2026-11-15. A systemization-of-knowledge paper mapping the attack surface of "deep research" agents across retrieval, planning, evidence integration, and report generation, tying it back to RAG poisoning and prompt injection. It introduces "agenda seeding," a subtle attack that plants a fake evaluation criterion with no explicit instruction, causing the attacker's preferred option to be picked as the best answer in 16–28% of generated reports — a quiet new vector for anyone building research or RAG agents that synthesize written conclusions.

Simon Willison (prompt-injection tag)

Nothing new in this window — the most recent tagged post is from 2026-09-17.

Embrace The Red

Nothing new in this window — the most recent post is from 2026-10-13.

Anthropic Engineering / Research

  • Updating our coordinated vulnerability disclosure policy timelines — 2026-11-16. Anthropic shortened its coordinated disclosure windows: 90 to 30 days for high/critical severity issues (45 for the rest), escalation to an external coordinator after 15 days of maintainer silence (down from 30), and a 30-day wait after a patch ships before publishing full technical details (down from 45). The change is driven by AI models getting better at finding vulnerabilities, which makes independent rediscovery more likely and makes fast patching more urgent — teams on the receiving end of Anthropic-sourced vulnerability reports should tighten their internal patch SLAs accordingly.

OWASP GenAI / LLM Top 10

  • GenAI Data Security Best Practices v2 - Part2 — 2026-11-17. The second part of OWASP's data security framework for GenAI pipelines lays out twelve reusable architectural patterns — governed ingress, metadata propagation, retrieval boundaries, full-chain erasure, AI-BOM lineage, federated policy enforcement — with layered implementation guidance, tooling pointers, and a regulatory mapping. It gives platform teams a concrete checklist of data controls (retrieval boundaries, erasure, lineage) to apply to RAG and agentic architectures that touch sensitive data.

NVIDIA Developer Blog

Nothing new and security-relevant in this window — recent posts cover MLPerf, molecular simulation, sparse attention, and tool-calling parsers, with no guardrails or agent-safety content since the last edition.

Through-line

Almost everything this week traces back to the same root cause: agents given more autonomy — to browse the web, run skills, write research reports, or just hold a secret — create more surface for things to go quietly wrong, often without an explicit "attack" in the traditional sense. System prompts leak under sustained pressure, privacy-preserving agents overshare while trying to be helpful, skill plugins get hijacked absent a hard enforcement boundary, and research agents can be nudged toward a conclusion without ever being told to lie. The practical response showing up across papers is the same: don't rely on the model's judgment alone — add deterministic checks at the handoff points (submit time, tool-call time, state-transfer time) where damage actually happens.

What's catching your attention this week — the system-prompt leak numbers, or the agenda-seeding attack on research agents? Drop a comment below.

Top comments (1)

Collapse
 
parasraju profile image
Paras •

That 80% system-prompt leak stat is genuinely scary.