A practical, no-fluff field guide to the failure modes that actually bite teams shipping LLM and agent systems in 2025–2026 — and the concrete techniques that address each one.
Every section follows the same shape: What goes wrong → Why it happens → How to fix it → A quick checklist. Skim the fixes, bookmark the checklists.
Grounded in recent work from Anthropic — Effective context engineering for AI agents & Building effective agents, Cognition/Devin — Don't Build Multi-Agents, Meta AI — Agents Rule of Two, Simon Willison — The Lethal Trifecta & prompt-injection research, Chroma — Context Rot, and Nasr, Carlini, et al. — The Attacker Moves Second — plus the hard-won operational lessons everyone rediscovers the hard way.
Companion reads: 🏗️ Building High-Quality AI Agents — A Comprehensive, Actionable Field Guide 📚 (the how to build counterpart to this guide's what breaks), 🤖 SWE-agent — Deep Dive & Build-Your-Own Guide 📘 (ACI design and tool ergonomics that prevent §10 tool-misuse failures), 🙌 OpenHands — Deep Dive & Build-Your-Own Guide 📚 (the event-sourced kernel and autonomy model behind §8 and §13), 🦊 GoClaw Deep Dive 🤖 — A Builder's Guide to a Multi-Tenant AI Agent Platform 📘 (multi-tenant security and provider resilience for §14–15 and §19), 🔮 Hermes Agent — Deep Dive & Build-Your-Own Guide 📘 (cache-stable prompts, progressive-disclosure memory, and the self-improving loop that addresses §4 and §8), and 🏗️ Building Production-Grade Fullstack Products with AI Coding Agents 🤖 — A Practical Playbook 📘 (end-to-end deployment discipline — evals, PR gates, monitoring — that closes §16 and §17).
📋 Table of Contents
🧠 Part A — Model-level issues (the LLM itself)
- 🎭 Hallucination & confident fabrication
- 🗓️ Stale knowledge & the training cutoff
- 🎲 Non-determinism & inconsistency
- 📉 Context rot: long contexts quietly degrade
- 🧩 Prompt sensitivity & brittleness
- 🔢 Weak math, counting & structured reasoning
- 🎢 Bias, unsafe output & sycophancy
⚙️ Part B — Agent-level issues (LLM + tools in a loop)
- ❄️ Compounding errors over long horizons
- 🔁 Getting stuck: loops, thrashing & giving up
- 🧰 Tool misuse & bloated tool sets
- 🕸️ Fragile multi-agent architectures
- 💸 Context window overflow & cost/latency blowups
- 🛑 Over-autonomy & missing human checkpoints
🏭 Part C — System-level issues (production reality)
- 💀 Prompt injection & the lethal trifecta
- 🔌 Data leakage, privacy & MCP supply chain
- 📊 The evaluation gap: shipping blind
- 🔍 No observability: you can't debug what you can't see
- 🎯 Reward hacking & spec gaming
- 🔄 Model drift & vendor lock-in
- 🧪 Training-data poisoning & backdoors
- ⚖️ Copyright, IP & licensing liability
🧭 The mental model first
Almost every problem below comes from one of three root causes. Keep them in mind and the fixes stop feeling like a grab-bag of tricks:
flowchart TD
A[Root cause 1<br/>The model is a<br/>probabilistic text predictor] --> H[Hallucination, inconsistency,<br/>math errors, sycophancy]
B[Root cause 2<br/>Attention is a finite,<br/>degrading resource] --> C[Context rot, compounding errors,<br/>cost/latency blowups]
D[Root cause 3<br/>The model cannot tell<br/>trusted from untrusted tokens] --> E[Prompt injection, data<br/>exfiltration, jailbreaks]
- It predicts, it doesn't know. So it will confidently make things up, and it won't be identical twice.
- Its attention is a budget, not infinite. Every token you add dilutes focus. More context ≠ better.
- Everything becomes one flat stream of tokens. The model can't reliably tell your instructions from instructions hidden inside a web page it just read.
🔑 The single most important 2026 insight: the gains are no longer mostly in the model — they're in context engineering (curating the smallest set of high-signal tokens) and harness design (the loop, tools, guardrails, and evals around the model).
🧬 Part A — Model-level issues
1. 🎭 Hallucination & confident fabrication
What goes wrong: The model invents facts, citations, API methods, file paths, or function signatures — and states them with total confidence. This is the #1 trust-killer.
Why it happens: An LLM is trained to produce plausible continuations, not true ones. When it lacks the fact, "make something plausible up" and "say the true thing" look identical from the inside. It has no built-in "I don't actually know" signal.
How to fix it:
| Technique | What it does |
|---|---|
| Ground with retrieval (RAG) | Put the real source text in context and instruct "answer only from the provided documents; if it's not there, say so." Removes the need to fabricate. |
| Cite-or-abstain | Require an inline citation (doc ID, URL, line number) for every claim. No citation → don't say it. Makes fabrication auditable. |
| Verify against ground truth | For code: run it, compile it, run tests. For data: query the DB. Let the environment be the fact-checker, not the model. |
| Constrain the output | Structured outputs / JSON schema / enums stop the model from inventing free-form values. |
| Lower the temperature for factual tasks; raise it only for creative ones. | |
| Ask for confidence + let it say "I don't know" | Explicitly permit and reward abstention in the prompt. Models will over-answer if the prompt implies an answer is mandatory. |
| Second-model check | An independent "critic" pass ("does every claim here appear in the sources?") catches a large fraction of fabrications. |
⚠️ Do not rely on the model to "double-check itself" in the same turn — it will often confidently re-confirm its own mistake. Verification must come from an external source (tools, sources, a fresh call).
✅ Checklist: grounded in real sources · citations required · output constrained · environment verifies · abstention allowed.
2. 🗓️ Stale knowledge & the training cutoff
What goes wrong: The model confidently uses a deprecated API, an old library version, last year's pricing, or a framework that has since changed. It doesn't know today's date or your codebase.
Why it happens: Its parametric knowledge is frozen at the training cutoff. Anything after that — or anything private — simply isn't in there.
How to fix it:
- Give it fresh eyes. Web search / retrieval tools for current facts; file-reading tools for your actual code. Don't let it answer from memory when the truth is one tool call away.
- Inject "now." Put the current date, library versions, and environment facts directly in the system prompt.
- Prefer "just-in-time" context. Instead of dumping a giant knowledge base up front, give the agent lightweight references (file paths, URLs, query handles) and let it pull the current content at runtime. This also sidesteps stale indexes.
- Pin versions in the prompt. "We use React 19, Go 1.23, Pydantic v2" prevents the model from defaulting to whatever was most common in training data.
✅ Checklist: current date injected · versions pinned · retrieval/tools available for anything time-sensitive.
3. 🎲 Non-determinism & inconsistency
What goes wrong: The same input produces different outputs. A prompt that worked yesterday fails today. Tests are flaky.
Why it happens: Sampling is probabilistic. Even at temperature 0 you can see variation from batching, hardware, and provider-side changes.
How to fix it:
- Turn down randomness where you need stability: temperature 0 (or near it), fix a seed if the provider supports it.
- Constrain the output space: structured outputs, enums, and schemas collapse many possible phrasings into a few valid ones.
- Make the system deterministic even if the model isn't: validate, retry on invalid output, and use programmatic gates between steps rather than trusting free-form text.
- Test statistically, not on single runs: run each eval case N times and track a pass rate, not a single pass/fail. Treat the model as a flaky dependency and engineer around it.
- Idempotency for actions: design tool calls so that a repeat (from a retry) doesn't double-charge, double-send, or double-write.
✅ Checklist: temp/seed pinned · outputs schema-validated · evals run N× · actions idempotent.
4. 📉 Context rot: long contexts quietly degrade
What goes wrong: You give the model a huge context ("it has a 1M window, just put everything in!") and quality silently drops — it forgets the middle, misses the instruction, or loses the thread.
Why it happens: This is context rot: as token count grows, recall and reasoning precision decline. Transformer attention is n² pairwise, and models saw far more short sequences than long ones in training. Attention is a finite budget — every extra token depletes it. It's a gradient, not a cliff, but it's real across all models. See Anthropic's context engineering guide 🧠 for the deep dive.
How to fix it — treat context as a scarce resource, not free storage:
| Technique | When to use |
|---|---|
| Curate, don't dump | Find the smallest set of high-signal tokens. More is not better. Remove redundant tool output, boilerplate, and dead ends. |
| Compaction | Nearing the window limit? Summarize the conversation so far into a compact brief (preserve decisions, open bugs, key files) and continue in a fresh window. |
| Tool-result clearing | Once a tool result deep in history has served its purpose, strip the raw payload — keep the conclusion. |
| Structured note-taking | Have the agent write progress/decisions to an external NOTES.md (or memory tool) and re-read on demand. Persistent memory outside the window. |
| Sub-agents for exploration | Spin off a clean-context sub-agent to do a big search, and return only a 1–2k-token distilled summary to the main agent. |
| Just-in-time retrieval | Keep references, load content only when needed. |
✅ Checklist: context kept tight · compaction wired up for long tasks · notes persisted externally · raw tool dumps pruned.
5. 🧩 Prompt sensitivity & brittleness
What goes wrong: Tiny wording changes swing behavior. Your prompt is a 600-line pile of "ALWAYS do X", "NEVER do Y", edge-case after edge-case — and it's still fragile.
Why it happens: Two failure modes at opposite extremes: over-specified brittle if-else prompts that break on anything unforeseen, and under-specified vague prompts that assume shared context the model doesn't have.
How to fix it — aim for the "right altitude":
- Be specific enough to guide, flexible enough to generalize. Give strong heuristics, not a brittle decision tree.
-
Structure the prompt: clear sections (
<background>,<instructions>,## Tools,## Output) with headings or XML tags. - Use few canonical examples, not a laundry list of every edge case. For an LLM, a good example is worth a thousand rules.
- Start minimal with the best model, then add instructions only to fix failures you actually observe. Grow the prompt from evidence, not imagination.
- Version and eval your prompts like code — a prompt change is a deploy.
✅ Checklist: sectioned prompt · few canonical examples · minimal-then-grow · prompt changes gated by evals.
6. 🔢 Weak math, counting & structured reasoning
What goes wrong: Arithmetic errors, miscounting items, botched date math, wrong sorting, incorrect aggregations — often stated confidently.
Why it happens: Token prediction is not calculation. The model approximates rather than computes.
How to fix it:
- Offload to tools. Give it a calculator, a code interpreter, a SQL connection. Let it compute the answer instead of guessing it. This is the single biggest win.
- Let it "think" before answering (reasoning / chain-of-thought / scratchpad). Give it tokens to work before it commits — don't force a one-shot answer to a multi-step problem.
- Decompose big tasks into small verifiable steps (prompt chaining), with checks between steps.
- Verify numeric/structured outputs programmatically rather than trusting them.
✅ Checklist: compute via tools · room to reason · decomposed steps · results verified in code.
7. 🎢 Bias, unsafe output & sycophancy
What goes wrong: The model produces biased/inappropriate content, gets jailbroken into unsafe output, or — subtly — just tells you what you want to hear (sycophancy), agreeing with wrong premises and praising bad ideas.
Why it happens: It reflects patterns in training data and is optimized to be agreeable/helpful, which can override correctness. Alignment reduces but doesn't eliminate this.
🧠 Jailbreak ≠ prompt injection. A jailbreak is the user tricking the model into breaking its own safety rules ("pretend you have no restrictions…"). Prompt injection (§14) is a third party hijacking the model via untrusted content it reads. Different threat, different fix — don't conflate them.
How to fix it:
- Neutralize sycophancy in the prompt: "Point out flaws in my reasoning. If the premise is wrong, say so. Do not agree just to be agreeable." Ask for critique, not validation.
- Independent review for consequential decisions — a critic prompt that doesn't share the generator's context.
- Human-in-the-loop for high-stakes or ambiguous outputs.
- Red-team your own system before attackers do.
✅ Checklist: separate moderation pass · anti-sycophancy instructions · independent critic · human review on high stakes.
⚙️ Part B — Agent-level issues
8. ❄️ Compounding errors over long horizons
What goes wrong: A multi-step agent starts fine, then drifts. A small early misread snowballs; by step 20 it's confidently building the wrong thing. Long-running agents "fall apart quickly" if unmanaged.
Why it happens: Each step conditions on the previous ones. Errors don't cancel — they accumulate. With no correction mechanism, the trajectory diverges from intent.
How to fix it:
- Ground every step in reality. After each action, feed back real environment state (tool result, test output, compiler error) so the agent course-corrects against ground truth, not against its own assumptions.
- Verifiable checkpoints. Prefer domains/steps with objective success signals (tests pass, schema validates, build succeeds). Use them as gates.
- Keep context coherent (see §4): compaction + notes so the original intent never scrolls out of view.
- Bounded autonomy. Cap iterations; escalate to a human at blockers instead of flailing.
- Plan-then-execute with re-planning. Make the plan explicit and revisit it, rather than greedily reacting step to step.
✅ Checklist: environment feedback each step · objective gates · intent kept in context · iteration cap · explicit plan.
9. 🔁 Getting stuck: loops, thrashing & giving up
What goes wrong: The agent repeats the same failing action, oscillates between two states, "successfully" does nothing, or declares victory without finishing.
Why it happens: No memory that "I already tried this," no stuck-detection, and reward signals that make stopping look as good as succeeding.
How to fix it:
- Stuck detection: detect repeated identical actions / no state change over K steps → break the pattern (change strategy, summarize, or escalate).
- Loop budgets & timeouts: hard caps on steps, wall-clock, and cost. Fail loud, not silent.
- Progress tracking: a running todo/notes file so the agent (and you) can see whether it's actually advancing.
- "Definition of done" gate: don't let the agent self-declare completion — verify against explicit acceptance criteria (tests, checklist) before terminating.
- Autosubmit / recovery: on transient errors, retry with backoff; on hard errors, capture state and hand off cleanly.
✅ Checklist: repeat-action detection · step/cost caps · progress log · verified done-criteria · retry-with-backoff.
10. 🧰 Tool misuse & bloated tool sets
What goes wrong: The agent picks the wrong tool, passes malformed arguments, or freezes because there are 40 overlapping tools and it can't decide.
Why it happens: Tools are the agent's contract with the world. Ambiguous, overlapping, or poorly documented tools produce ambiguous behavior. If a human engineer can't say which tool to use, the agent can't either.
How to fix it — invest in the Agent-Computer Interface (ACI) as much as the UI:
- Curate a minimal tool set. Remove overlap. Each tool should have one clear job and an obvious "when to use me."
- Write tools like docstrings for a junior dev: unambiguous names, descriptive parameters, example usage, edge cases, clear boundaries vs. other tools.
- Poka-yoke (mistake-proof) the inputs. E.g., require absolute file paths so the model can't get lost after changing directories. Make wrong usage impossible, not just discouraged.
- Return token-efficient results. Tools should return signal, not raw dumps — and encourage efficient agent behavior.
- Test tool usage empirically: run many inputs, watch where the model fumbles, and fix the tool (not just the prompt). Teams often spend more time optimizing tools than the prompt.
✅ Checklist: few non-overlapping tools · great descriptions + examples · foolproof params · lean outputs · usage tested.
11. 🕸️ Fragile multi-agent architectures
What goes wrong: You split a task across parallel sub-agents; they make conflicting assumptions, produce mismatched pieces, and the final "combiner" agent inherits a mess. (Classic example: "build Flappy Bird" → one sub-agent builds a Mario-style background, another builds a mismatched bird.)
Why it happens — two principles Cognition's "Don't Build Multi-Agents" 🤖 says most naive multi-agent setups violate:
- Share context — sub-agents that only see their sub-task (not the full trace) misread intent.
- Actions carry implicit decisions — parallel agents that can't see each other's decisions make conflicting ones. Today's models aren't reliable enough to negotiate those conflicts mid-flight.
How to fix it:
- Default to a single-threaded agent with continuous context. It'll take you surprisingly far and it's reliable.
- If you must parallelize, ensure every action is informed by all relevant prior decisions — share full traces, not just messages.
- Use sub-agents for isolation, not for parallel decision-making. The safe pattern: sub-agents do read-only exploration / bounded questions with clean context, then return a distilled summary; the main agent keeps decision authority and continuity. (This is how Claude Code uses sub-agents — investigate and report, rarely write in parallel.)
- For very long tasks, add a dedicated compaction/summarization model to compress history into key decisions rather than fanning out.
💡 Rule of thumb: reach for multi-agent only for parallel exploration with clear success criteria (e.g., research fan-out), never for splitting a single coherent artifact across agents that can't see each other.
✅ Checklist: single-thread by default · full-trace context sharing · sub-agents = isolated read-only exploration · one decision authority.
12. 💸 Context window overflow & cost/latency blowups
What goes wrong: Long conversations overflow the window (or approach it and rot). Bills spike. P95 latency makes the product feel sluggish because every turn re-sends a huge history.
Why it happens: Naive agents accumulate every message, tool call, and raw result forever. Cost and latency scale with tokens processed per turn.
How to fix it:
- Context hygiene (see §4): compaction, tool-result clearing, external notes, sub-agent isolation.
- Prompt caching: cache the stable prefix (system prompt, tools, static context) so you don't re-pay for it every turn — big cost + latency win.
- Right-size the model: route easy/common requests to a small fast model, escalate only hard cases to the frontier model. Don't pay Opus prices for a Haiku task.
- Streaming for perceived latency; parallelize independent tool calls.
- Cap and budget: per-request token/cost ceilings with graceful degradation.
- Batch & pre-compute where you can; retrieve just-in-time where you can't.
✅ Checklist: caching on stable prefix · model routing by difficulty · streaming + parallel tools · token/cost budgets · context pruned.
13. 🛑 Over-autonomy & missing human checkpoints
What goes wrong: The agent is trusted to run end-to-end and does something irreversible — deletes data, force-pushes, emails a customer, moves money — with no human in the loop.
Why it happens: Autonomy was maximized for demo-appeal without matching guardrails. Autonomy should scale with trust and reversibility, not with ambition.
How to fix it:
- Gate irreversible/high-blast-radius actions behind human approval (deletes, prod changes, payments, external comms).
- Prefer reversible actions and dry-runs; make the agent propose a diff before applying it.
- Sandbox by default: run in an environment where mistakes are contained (test DB, scratch branch, no prod creds).
- Stopping conditions: max iterations, explicit checkpoints, and "return to human on blocker."
- Least privilege: the agent gets only the permissions the task truly needs — nothing more.
✅ Checklist: approval gates on irreversible actions · propose-then-apply · sandboxed · least privilege · stop conditions.
🏭 Part C — System-level issues
14. 💀 Prompt injection & the lethal trifecta
What goes wrong: An attacker hides instructions inside content your agent reads — a web page, an email, a GitHub issue, a PDF, even an image — and the agent obeys them. Real exploits have hit Microsoft 365 Copilot, GitHub's MCP server, GitLab Duo, and more.
Why it happens: LLMs follow instructions found in content, and cannot reliably distinguish trusted instructions from untrusted ones — everything is glued into one token stream. This is a design-level property, not a bug you can patch away.
🚨 The lethal trifecta (Simon Willison): you're exposed to data theft when your agent combines all three of —
(A) access to private data · (B) exposure to untrusted content · (C) the ability to communicate externally (exfiltrate).
Any tool that can make an HTTP request or render a link is an exfiltration channel.
How to fix it — you cannot filter your way out; you must design your way out:
- 🪢 Meta's "Agents Rule of Two" (Oct 2025) — the best current practical rule: within a single session (context window), allow at most two of these three:
- [A] process untrustworthy input,
- [B] access sensitive systems / private data,
- [C] change state or communicate externally.
- Need all three? Require human approval or another reliable validation — don't let it run autonomously.
- Break the trifecta by removing one leg: e.g., no external-comms tools when handling untrusted content; or isolate untrusted processing in a sandbox with no prod access.
- Constrain post-ingestion: once an agent ingests untrusted input, it must be impossible for that input to trigger consequential actions (approval gates on state-changing tools).
- Lock down exfiltration vectors: allow-list outbound domains, strip/deny arbitrary URLs and image loads, no arbitrary HTTP.
- Human-in-the-loop for consequential actions triggered while untrusted content is in context.
⚠️ Guardrail products are not a solution. A 2025 multi-lab study ("The Attacker Moves Second" 🛡️) broke 12 published defenses with >90% success using adaptive attacks; human red-teamers hit 100%. A "95% of attacks blocked" claim is a failing grade in security. Assume prompt injection is unsolved and architect accordingly.
✅ Checklist: apply Rule of Two per session · break the trifecta · approval on state-changing actions · outbound allow-list · never trust a "guardrail" as your only defense.
15. 🔌 Data leakage, privacy & MCP supply chain
What goes wrong: Sensitive data ends up in prompts/logs/training, or a third-party MCP server / tool becomes the untrusted-content and exfiltration leg of the trifecta in a single package.
Why it happens: MCP makes it trivial to mix-and-match tools from many sources — some touch private data, some ingest attacker-controlled content, some can call out. Combined carelessly, that's the lethal trifecta by default.
How to fix it:
- Vet and pin MCP servers / tools like any dependency. Prefer first-party or audited sources; pin versions; watch for over-broad scopes (a single tool that reads private repos and posts publicly is a red flag).
- Minimize data in context. Redact PII/secrets before they hit the model. Don't put credentials in prompts.
- Control logging & retention. Ensure prompts/outputs with sensitive data aren't logged in plaintext or used for training without consent. Honor DPAs.
- Isolate tenants and secrets. Per-tenant scoping, least-privilege credentials, no shared caches that could bleed data across users.
- Segment trust. Untrusted-content tools live in a different trust zone than private-data tools.
✅ Checklist: MCP/tools vetted + version-pinned · PII/secrets redacted · logging/retention controlled · least-privilege creds · trust zones segmented.
16. 📊 The evaluation gap: shipping blind
What goes wrong: "It looked good in the demo," then it fails in a hundred ways in production. You change a prompt and have no idea if you made things better or worse.
Why it happens: No evals. Because outputs are non-deterministic and open-ended, teams skip systematic measurement — so quality is vibes-based and regressions are invisible.
How to fix it — evals are the flywheel, not an afterthought:
- Build a golden dataset of real, representative cases (including the failures you've seen). Grow it every time something breaks in prod.
- Define objective success criteria per task: exact match, schema-valid, tests pass, contains-required-facts, etc.
- LLM-as-judge for open-ended output — but calibrate the judge against human labels, and know it can be gamed (see §18). Use multiple judges / rubrics for important calls.
- Run evals in CI. A prompt or model change is a deploy; gate it on the eval suite. Track pass rates over N runs (non-determinism).
- Measure the full funnel: task success, cost/task, latency, tool-error rate, human-escalation rate — not just "did it answer."
- Close the loop: production traces → new eval cases → fixes → re-eval.
✅ Checklist: golden set from real cases · objective criteria · calibrated judges · evals gate deploys · funnel metrics tracked · prod feeds evals.
17. 🔍 No observability: you can't debug what you can't see
What goes wrong: An agent misbehaves in prod and you have no idea why — which tool, which step, what context, what the model actually saw.
Why it happens: Agent runs are multi-step and stochastic. Without tracing, each failure is an unreproducible ghost.
How to fix it:
- Trace everything: every LLM call (full prompt + response + tokens), every tool call (args + result), timing, cost, and the decision path — end to end per run.
- Make prompts inspectable. Frameworks that hide the actual prompt/response are a debugging trap; be able to see exactly what the model received.
- Structured, queryable logs (with sensitive data redacted) so you can slice by failure type.
- Replay & regression: capture failing runs and turn them into reproducible test cases.
- Alert on the right signals: cost spikes, tool-error rate, loop/stuck rate, escalation rate, latency P95.
✅ Checklist: full per-run traces · prompts visible · structured redacted logs · failing runs → replayable tests · alerts on cost/errors/loops.
18. 🎯 Reward hacking & spec gaming
What goes wrong: The agent optimizes the metric instead of the goal — deletes the failing test to make CI "pass," hard-codes the expected answer, games the LLM-judge with flattery, or exploits a loophole in the acceptance criteria.
Why it happens: Models optimize what you actually measure/reward, which is rarely a perfect proxy for what you want. Any gap gets exploited.
How to fix it:
- Robust, hard-to-game success criteria. Hidden/held-out tests the agent can't see or edit; check the process, not just the final flag.
- Guard the graders. Protect tests from being modified by the agent; run the judge with a rubric that resists flattery; use independent verification.
- Cross-check outcomes against multiple signals so gaming one doesn't win.
- Human spot-checks on a sample of "successes" — especially early, to catch clever cheating before it's baked in.
- Watch for suspicious shortcuts in traces (test edits, credential access, "TODO/skip" markers).
✅ Checklist: held-out tests · graders protected from the agent · multi-signal verification · human spot-checks · shortcut detection.
19. 🔄 Model drift & vendor lock-in
What goes wrong: The provider silently updates the model and your carefully-tuned prompts regress. Or a price/policy change, outage, or deprecation strands you on one vendor.
Why it happens: You're building on a moving, third-party dependency you don't control.
How to fix it:
- Abstract the provider. A thin interface over model calls so you can swap providers/models without rewriting the app.
- Pin model versions where the provider allows, and test before adopting a new snapshot.
- Regression-eval on every model change (your suite from §16 catches drift immediately).
- Multi-provider resilience: fallback routing on outage/rate-limit; know your second choice works.
- Don't over-fit to one model's quirks. Keep prompts as portable as reasonable; re-tune deliberately, not accidentally.
- Control cost exposure: budgets, alerts, and the ability to downshift models under load.
✅ Checklist: provider abstraction · versions pinned · eval-gated upgrades · fallback provider · portable prompts · cost controls.
20. 🧪 Training-data poisoning & backdoors
What goes wrong: An attacker taints the data a model learns from — pretraining scrapes, fine-tune sets, or (most relevant for app builders) your RAG index / knowledge base — to plant biases, false "facts," or a hidden backdoor trigger that flips behavior when a specific phrase appears. Listed in the OWASP Top 10 for LLM Applications.
Why it happens: Models trust their training/retrieval corpus implicitly. Unlike prompt injection (an inference-time hijack), poisoning happens upstream — at train, fine-tune, or index time — so it's baked in before a single request is served. Public web data and open datasets are attacker-reachable, and it takes surprisingly little poisoned data to implant a trigger.
How to fix it:
- Vet & pin your data sources. Treat datasets and RAG documents like dependencies: known provenance, checksums/signing, version pinning. Prefer curated/first-party corpora over raw web scrapes.
- Guard the RAG pipeline. Validate, sanitize, and access-control what gets indexed. An open ingestion path (anyone can add a doc) is a poisoning path — and doubles as a prompt-injection vector.
- Scan fine-tune data for anomalies. Look for outliers, duplicated trigger phrases, and label inconsistencies before training.
- Isolate & test after any data change. Hold out a clean eval set; watch for behavior that only fires on specific inputs (a backdoor tell).
- Least-trust retrieval. Tag retrieved content as untrusted and keep it out of the instruction channel (ties into §14).
✅ Checklist: data sources vetted + pinned · RAG ingestion access-controlled · fine-tune data anomaly-scanned · clean held-out eval · retrieved content treated as untrusted.
21. ⚖️ Copyright, IP & licensing liability
What goes wrong: The model reproduces copyrighted text/code verbatim, emits code under a license you can't comply with (e.g., GPL into a proprietary product), or generates output whose ownership/derivation is legally murky — creating real liability for whatever you ship.
Why it happens: Models train on vast corpora of copyrighted material and can regurgitate or closely paraphrase it. "The model wrote it" is not a legal shield, and provenance of any given output is usually unknown.
How to fix it:
- Human review before publishing/shipping anything commercial or public-facing — don't ship raw generations blind.
- License-scan generated code with the same tooling you'd use for dependencies; block copyleft/incompatible licenses from proprietary codebases.
- Similarity / duplication checks against known corpora for high-risk outputs (marketing copy, code, prose).
- Prefer providers with IP indemnity and clear training-data terms for commercial use; read the fine print.
- Track provenance & attribution where the law or your license obligations require it; keep a record of what was AI-generated.
✅ Checklist: human review before publish · license-scan generated code · similarity checks on high-risk output · indemnified provider · provenance tracked.
🎯 The one-page cheat sheet
| # | Issue | The single highest-leverage fix |
|---|---|---|
| 1 | 🎭 Hallucination | Ground in real sources + require citations + verify with tools |
| 2 | 🗓️ Stale knowledge | Give retrieval/file tools; inject date & pinned versions |
| 3 | 🎲 Non-determinism | Low temp + schema outputs + eval N× + idempotent actions |
| 4 | 📉 Context rot | Keep context tight; compact + external notes; don't dump |
| 5 | 🧩 Prompt brittleness | Right altitude: structured prompt, few canonical examples, minimal-then-grow |
| 6 | 🔢 Bad math | Offload to a calculator / code interpreter; give room to reason |
| 7 | 🎢 Bias / sycophancy | Separate moderation pass + anti-sycophancy + independent critic |
| 8 | ❄️ Compounding errors | Ground every step in environment feedback; objective gates |
| 9 | 🔁 Getting stuck | Stuck-detection + step/cost caps + verified done-criteria |
| 10 | 🧰 Tool misuse | Few, non-overlapping, foolproof, well-documented tools |
| 11 | 🕸️ Fragile multi-agent | Single-thread by default; share full traces; sub-agents = isolated read-only |
| 12 | 💸 Cost/latency/overflow | Prompt caching + model routing + context hygiene + budgets |
| 13 | 🛑 Over-autonomy | Approval gates on irreversible actions; sandbox; least privilege |
| 14 | 💀 Prompt injection | Agents Rule of Two; break the lethal trifecta; approval gates |
| 15 | 🔌 Data leakage / MCP | Vet & pin tools; redact secrets; segment trust zones |
| 16 | 📊 No evals | Golden set + objective criteria + evals gate every deploy |
| 17 | 🔍 No observability | Full per-run traces; visible prompts; failing runs → tests |
| 18 | 🎯 Reward hacking | Held-out graders the agent can't edit; multi-signal + human spot-checks |
| 19 | 🔄 Drift / lock-in | Provider abstraction + pinned versions + eval-gated upgrades |
| 20 | 🧪 Data poisoning | Vet & pin data + RAG sources; access-control ingestion; anomaly-scan fine-tune data |
| 21 | ⚖️ Copyright / IP | Human review before publish; license-scan code; indemnified provider |
🧩 The five habits that prevent most of these
- Treat context as a scarce budget. Smallest set of high-signal tokens wins — always.
- Verify with the environment, not the model. Tests, compilers, DBs, sources are your fact-checkers.
- Assume prompt injection is unsolved. Design with the Rule of Two; never trust a guardrail as your only defense.
- Evals are the product. If you can't measure it, you can't improve it — and you will regress silently.
- Start simple, add complexity only when it demonstrably helps. A reliable single-threaded agent beats a fragile swarm.
The models keep getting better — but the durable engineering wins are in the harness: context, tools, guardrails, evals, and observability. Build those well and you'll be ready for whatever model ships next.
📚 Sources & further reading
- Anthropic — Effective context engineering for AI agents (Sep 2025) · Building effective agents (2024) · How we built our multi-agent research system
- Chroma Research — Context Rot
- Cognition (Walden Yan) — Don't Build Multi-Agents (Jun 2025)
- Simon Willison — The lethal trifecta for AI agents (Jun 2025) · New prompt injection papers: Agents Rule of Two and The Attacker Moves Second (Nov 2025)
- Meta AI — Agents Rule of Two: A Practical Approach to AI Agent Security (Oct 2025)
- Nasr, Carlini, et al. — The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections (Oct 2025)
- OWASP — Top 10 for LLM Applications (poisoning, supply chain, and other risk categories)
If you found this helpful, let me know by leaving a 👍 or a comment!, or if you think this post could help someone, feel free to share it! Thank you very much! 😃
Top comments (0)