There is a specific kind of confidence failure that shows up in agentic AI systems. It is not the dramatic hallucination - the confident fabrication of a person who never existed. It is quieter than that. The model understands the task. It understands the data. And then it decides it has a better idea.
We ran into this repeatedly while building the content health maintenance workflow for Synthadoc, an open-source AI-native wiki engine. The workflow's job sounds modest: scan wiki pages for structural problems like duplicate headings or thin sections, propose targeted fixes, and let the user approve them before applying anything. What we found, and eventually fixed, is a catalogue of ways an agent can go wrong even when the model is capable and the intent is clear - and why the solution is never just a better instruction.
The Problem We Were Solving
Synthadoc ingests raw documents - PDFs, web pages, transcripts, spreadsheets - and uses an LLM to synthesize them into a structured Markdown wiki. Over time, the same topic gets ingested from multiple sources. Without intervention, a single wiki page about Alan Turing can accumulate ten copies of ## Bletchley Park and Cryptanalysis and eight copies of ## Early Life and Education. The page is coherent in isolation but structurally broken: duplicate content, contradictory emphasis, and thin sections that exist only because a heading was created before the body text was written.
The content health scanner addresses this. It is deliberately a pure-Python heuristic with no LLM dependency - it detects duplicate_section, empty_section, thin_section, unclosed_code_block, orphaned_heading, and broken_list_nesting through deterministic regex. This part is reliable. The next part - using an LLM to fix the issues - is where the story gets interesting.
What We Intended the Workflow to Do
The design was explicit. For each flagged page:
- Load the page content and the list of detected issues
- Send both to the LLM with a carefully constructed prompt
- Show the user a side-by-side BEFORE/AFTER diff
- Wait for approval before writing anything
- Snapshot the previous version so a rollback is possible
Every step in this design has a purpose. The BEFORE/AFTER review is a human gate, not a formality. The snapshot is a reversibility guarantee. The heuristic scanner runs before the LLM so the issue list is precise, not speculative.
What actually happened when we first ran it told us exactly which parts of our hardening were missing.
What Went Wrong
Failure 1: The Model Rewrote the Page
The first problem emerged immediately. The LLM received the page content and a list of issues. There were twelve structural problems on the page about the Open Source Movement - three duplicate sections and nine thin sections. The AFTER content we received back bore no structural resemblance to the original. The original heading ## Richard Stallman and GNU (1983) was gone. The heading ## The Cathedral and the Bazaar (1999) was gone. In their place: ## Historical Framing and Origins, ### Research Computing Culture and Principled Advocacy, ### GNU Toolchain and the Missing Kernel. The content was factually reasonable. It was also something the user never asked for and could not safely approve, because we had lost all traceability between the original and the proposed version.
The model had, in effect, written its own wiki page from its pre-trained knowledge, using our wiki page as a rough reference. The word "fix" in our prompt was interpreted by the model as license for rewriting.
Failure 2: The Model Kept the Wrong Duplicate
For a page about computing history, the scanner flagged ## Moore's Law appearing three times. Two occurrences were thin summaries of about forty words each. The third was a detailed paragraph with four footnote citations to the source material. Our initial prompt said "keep the first occurrence." The model faithfully kept the first - one of the forty-word summaries - and discarded the version with citations.
Technically correct. Practically, the wrong choice. We had a rule that was precise but not meaningful for the actual goal.
Failure 3: CLI-Provider Artifacts in the Proposed Content
Synthadoc supports multiple LLM backends. One of them is a CLI-based provider - an agentic tool (like opencode or Claude Code) that is itself an agent with its own tool-calling loop. When we routed the content health workflow through this provider, the response text included shell command output: <|func_arg_name|>command<|func_arg_value|>cat /Users/user1/wikis/alan-turing.md. The model had read the actual file from disk, bypassing our carefully constructed context, and its internal tool invocations leaked into the response text we were about to write to the wiki.
Failure 4: Planning Notes Presented as Final Content
In some runs, the proposed content contained lines like:
### LISP (1958) - actually LISP is more third gen but...
### Introduction paragraph (existing)
Expand to 50+ words
These were the model's internal planning notes - draft scaffolding that it had not cleaned up before terminating its response. These notes would have been written to the wiki without detection. It is possible that a user reviewing a diff in a modal at speed might miss them.
Failure 5: Heading Renamed to "Correct" a Date
After tightening the prompt, one run still returned a subtle violation. The original heading was ## The Cathedral and the Bazaar (1999). The proposed content had ## The Cathedral and the Bazaar - the year dropped. The model "knew" the essay was presented in 1997 and silently corrected what it interpreted as an error in the user's own heading. It was an editorial judgment we had not authorized.
The Failure Mode Map
These failures cluster into two categories that are worth naming precisely.
Category A - DRIFT: Model understands task but applies wrong judgment
Category B - HALLUCINATION: Model substitutes its own knowledge for the task
It is important to distinguish. As a result of drift, the right process is being used to make wrong decisions - "keep the first" instead of "keep the best." As a result of hallucination, the right process is being replaced completely - generating a page from weights instead of fixing the one we hand it. Different mitigations are required for each.
How We Hardened the System and Built the Harness
These fixes fall into two distinct layers. Hardening addresses what the model produces: tighter prompts, output parsing, provider-specific guards. Harness engineering addresses what the model can affect: human gates, rollback, scope exclusions. Hardening reduces the probability of a bad output; the harness limits the consequence of one that gets through. Both are required - hardening alone cannot guarantee zero failures, and a harness without hardening just moves the review burden to the human gate.
Hardening the Agent
1. Artifact Extraction as an Input Boundary
The CLI-provider issue required a structural fix, not a prompt fix. We introduced _extract_md_from_response() - a pre-processing step that strips everything before the first # H1 heading in the response. CLI-provider tool call output always appears before the actual content begins. By treating the first H1 as the clean boundary, we recover the markdown content without the artifact noise.
This is an example of boundary enforcement at the output interface: treat the model's raw response as untrusted input to a parser, not as a value ready to write.
2. Draft Detection as a Post-Output Gate
For planning notes, we added _is_draft() - a regex that catches linguistic markers of incomplete thinking: TODO, FIXME, Expand to, Hmm,, Wait,, I'll, Let me, existing content. If any match, the proposed content is flagged and the page is skipped automatically with a clear explanation in the UI.
_DRAFT_RE = re.compile(
r"^\s*(TODO|FIXME|Expand to\b|Hmm[,.\s]|Wait[,.\s]|I'll\b|I will\b|Let me\b"
r"|existing content|combine existing|add more content|placeholder)",
re.IGNORECASE | re.MULTILINE,
)
This is an output validation gate - a deterministic check applied after the model responds and before the result reaches the user approval screen. The model cannot disable it by rephrasing the note.
3. Surgical Prompt Constraints
The original prompt said: "Fix the structural issues listed below." That sentence, to a language model, is nearly unlimited scope. We replaced it with an explicit constraint hierarchy:
ABSOLUTE RULES - violating any one is a failure:
• PRESERVE every existing section heading EXACTLY - same text, same # level,
including any dates or years in parentheses. Do not 'correct' headings.
• PRESERVE the section ORDER exactly as in the original.
• Do NOT add new top-level or second-level sections.
• Do NOT rename, merge, split, or reorder any sections.
• Do NOT alter factual claims in sections you are not fixing.
PERMITTED CHANGES (only these, only where the issue list directs):
1. Duplicate sections: keep the most complete occurrence (most words and/or
citations); delete the others. Place the kept version where the first
occurrence was. Do NOT merge or rewrite - keep it verbatim.
2. Thin/empty sections: add factual prose UNDER the existing heading.
The heading text must remain unchanged.
Changing from "fix issues" to "permitted changes" is intentional. The model cannot rationalize around a list of what it is allowed to do when permissions are enumerated rather than prohibitions.
4. Issue Count Threshold for Nondeterministic Providers
Even with the tighter prompt, pages with more than six issues on a CLI-provider run produced hallucinated new sections. The CLI provider is itself an agentic tool with its own reasoning loop - it does not behave like a simple completion endpoint, and the compound nondeterminism (our model + its internal model) overwhelms a long issue list.
We added a hard skip at six issues for the CLI path:
# CLI providers (opencode, claude-code) are agentic - they read files,
# plan before writing, and hallucinate new sections when the issue list
# is long. API providers follow the prompt reliably and don't need this
# guard. This path is CLI-only, so the cap applies only there.
_MAX_ISSUES_AUTO_FIX = 6
API-key providers (Anthropic, OpenAI) use a separate code path with no cap, since they are stateless completion endpoints that see the prompt exactly as we construct it. Rather than a model capability, this is a property of the architecture.
Building the Harness
5. System Page Exclusion
Thin sections on pages named index and dashboard were correctly identified by the scanner. These are navigation pages, not content pages - thin sections are structurally appropriate. The workflow would propose expanding them without exclusion, which the user would reject correctly, wasting their time.
We use the existing SYSTEM_PAGE_SLUGS constant to exclude them from the fix loop. No duplicate definition, no magic string. The scanner still runs on them - the issues appear in lint reports - but the auto-fix workflow skips them.
6. The Human Gate as a Structural Requirement
Every proposed fix must pass through a BEFORE/AFTER diff review before anything is written. This is not a feature - it is the architectural guarantee that no automated process can corrupt wiki content without explicit consent. The confirm card shows:
- The page slug and issue summary
- BEFORE (first 2000 chars of the original)
- AFTER (first 2000 chars of the proposed content)
- "Approve" / "Skip this page" buttons with a 5-minute timeout (auto-decline)
The scrollability of code blocks in the review modal was a concrete UX issue we had to address - users need to be able to read what they are approving. Gates that are theoretically present but practically unusable are not gates.
7. Snapshot and Rollback as a Safety Net
Before any approved fix is written, snapshot_if_changed() takes a versioned copy of the original content. If the approved result turns out to be wrong, the user can request a rollback by name: "roll back alan-turing." The workflow restores the snapshot, re-runs the scanner, and updates the issue list to reflect the restored state. It is not a rollback to recover from system failure - it is a first-class workflow feature that acknowledges the model may produce something subtly wrong that was not detected during review.
The Broader Principle: Both Layers Are Architectural, Not Textual
The pattern we followed across all of these fixes is consistent: every control - whether it hardens the agent's output or constrains the harness around it - is implemented in code, not in prose.
"Do not hallucinate" in a system prompt is a behavioral suggestion. The model considers it, weighs it against other pressures, and sometimes overrides it. That is not a flaw, it is how language models work. The token prediction process does not have a hard stop for "this would be a hallucination."
Both hardening and harness engineering mean building controls that are not subject to the model's interpretation:
| Layer | Control | What it does | Why it cannot be bypassed |
|---|---|---|---|
| Hardening | _extract_md_from_response() |
Strips pre-H1 content | Runs on raw response bytes, before any model interpretation |
| Hardening |
_is_draft() regex |
Blocks planning-note content | Deterministic regex match - not in the model's loop |
| Hardening | PERMITTED CHANGES prompt | Enumerates allowed scope | Narrows the action space; drift must contradict the list |
| Hardening | Issue count cap | Skips complex pages | Hard code gate; model never sees the page |
| Harness | System page exclusion | Skips non-content pages | Slug lookup against a constant; not in the prompt |
| Harness | BEFORE/AFTER human gate | Requires user approval | No write path bypasses this |
| Harness | Snapshot / rollback | Preserves prior state | Written before the fix is applied |
These controls form layers. A fix that evades the prompt constraints is caught by the draft detector or the human gate. A fix that passes the human gate can be rolled back. No single layer is sufficient. The layers together are.
Why This Matters Beyond Wiki Maintenance
Incorrectly updating a wiki page is a recoverable mistake. A user clicks rollback, the page is restored, and no lasting harm is done. But the failure modes we observed - a model substituting its own knowledge for the task, a model including its internal tool calls in the output, a model silently correcting data it considered wrong - are not domain-specific. The same failure patterns appear in:
- Agentic coding tools that refactor code the user did not ask to touch
- Document automation systems that "improve" a legal clause beyond the approved scope
- Data pipelines where an LLM normalizes values using world knowledge rather than the provided schema
- Financial report generators that extrapolate figures not present in the source data
In each case, the consequence of unhardened agentic behavior scales with the stakes of the domain. A hallucinated section in a personal wiki is recoverable. A hallucinated figure in a financial disclosure is not.
The argument for hardening is not that models are bad. It is that models are nondeterministic, and production systems require deterministic guarantees at the boundaries where AI output meets persistent state.
What to Apply From This
If you are building an agentic system that reads and modifies persistent data, consider these as minimum requirements:
Boundary enforcement. Treat the model's output as untrusted input at every handoff. Parse it. Validate it. Strip what does not belong before it reaches the next step.
Deterministic post-output gates. Add validation logic that runs after the model responds and is not subject to the model's behavior. Draft detection, schema validation, output format checks. These are deterministic; the model cannot reason around them.
Enumerated permissions over enumerated prohibitions. In prompts, tell the model what it is allowed to do, not only what it must avoid. The scope of permitted actions is easier to reason about than an open-ended task with a list of exceptions.
Acknowledge provider heterogeneity. Not all "model" configurations behave the same way. A CLI-based agentic provider and a stateless API completion endpoint have fundamentally different behavioral properties. Design your hardening layer to account for both, or constrain which providers can reach which actions.
Human gates as structural requirements. For any workflow that modifies persistent state, a human review step is not a UX nicety - it is a control. Make it usable. If the review UI cannot display the full diff, the gate is not functioning as intended.
Reversibility as a first-class design constraint. Snapshots and rollback should be designed in from the start, not added after the first production incident. Assume the model will sometimes produce a subtly wrong result that passes review, and design the recovery path accordingly.
Closing
It took multiple iterations for Synthadoc's content health maintenance workflow to behave predictably. Rather than rephrasing the prompt, each iteration addressed a specific failure mode. As a result, the team trusts the workflow - not because the model is perfect, but because the system is designed to catch and contain errors.
Synthadoc is open source. The content health scanner, the resolver workflow, and the draft detection logic are all available at Synthadoc git repo. If you are building AI-native systems that write back to persistent state, the patterns described here are worth examining in the context of real code, not just as abstractions.
The nondeterministic machine can be harnessed. The harness is engineering, not prompting.
Top comments (0)